161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5329968155
CatgirlIntelligenceAgency/code/process-models/crawling-model
History
Viktor Lofgren 5329968155 (crawler) Update CrawlingThenConvertingIntegrationTest 
This commit updates CrawlingThenConvertingIntegrationTest with additional tests for invalid, redirecting, and blocked domains. Improvements have also been made to filter out irrelevant entries in ParquetSerializableCrawlDataStream.
2023-12-15 21:04:06 +01:00
..
src (crawler) Update CrawlingThenConvertingIntegrationTest 2023-12-15 21:04:06 +01:00
build.gradle (crawling-model) Implement a parquet format for crawl data 2023-12-13 16:22:19 +01:00
readme.md (refactor) Remove features-search and update documentation 2023-10-09 15:12:30 +02:00

readme.md

Crawling Models

Contains models shared by the crawling-process and converting-process.

Central Classes

Serialization