161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7617b4cbc2
CatgirlIntelligenceAgency/code/processes/crawling-process
History
Viktor Lofgren 7617b4cbc2 (crawler) Fix NPE in crawler caused by not having fetched the domains list yet
2023-11-06 18:16:38 +01:00
..
src (crawler) Fix NPE in crawler caused by not having fetched the domains list yet 2023-11-06 18:16:38 +01:00
build.gradle (crawler) Integrate atags.parquet with the crawler so that "important" URLs are prioritized 2023-11-06 16:14:58 +01:00
readme.md (refactor) Remove features-search and update documentation 2023-10-09 15:12:30 +02:00

readme.md

Crawling Process

The crawling process downloads HTML and saves them into per-domain snapshots.

Central Classes

See Also