2e536e3141
This update includes the addition of timestamps to the parquet format for crawl data, as extracted from the Warc stream. The parquet format stores the timestamp as a 64 bit long, seconds since unix epoch, without a logical type. This is to avoid having to do format conversions when writing and reading the data. This parquet field populates the timestamp field in CrawledDocument. |
||
|---|---|---|
| .. | ||
| main/java | ||
| test/java/nu/marginalia/crawling/parquet | ||