Commit Graph

7 Commits

Author SHA1 Message Date
Viktor Lofgren
787a20cbaa (crawling-model) Implement a parquet format for crawl data
This is not hooked into anything yet.  The change also makes modifications to the parquet-floor library to support reading and writing of byte[] arrays.  This is desirable since we may in the future want to support inputs that are not text-based, and codifying the assumption that each document is a string will definitely cause us grief down the line.
2023-12-13 16:22:19 +01:00
Viktor Lofgren
dbe9235f3a (*) Upgrade to JDK21 with preview enabled.
... also move some common configuration into the root build.gradle-file.

Support for JDK21 in lombok is a bit sketchy at the moment, but it seems to work.  This upgrade is kind of important as the new index construction really benefits from Arena based lifecycle control over off-heap memory.
2023-09-24 10:38:59 +02:00
Viktor Lofgren
35996d0adb (docs) Update the documentation up-to-date information 2023-09-14 11:33:36 +02:00
Viktor Lofgren
9f672a0cf4 (parquet-floor) Modify the parquet library to permit list-fields. 2023-09-13 15:56:35 +02:00
Viktor Lofgren
a00cabe223 (parquet-floor) Patch in support for writing and reading repeated values 2023-09-11 14:06:43 +02:00
Viktor Lofgren
dbe974f510 (parquet) Use ZSTD compression by default. 2023-09-11 09:02:58 +02:00
Viktor Lofgren
a284682deb (parquet) Add parquet library
This small library, while great, will require some modifications to fit the project's needs, so it goes into third-party directly.
2023-09-05 10:38:51 +02:00