History

Viktor Lofgren eaeb23d41e (refactor) Remove converting-model package completely		2023-09-14 11:21:44 +02:00
..
src	(heartbeat, reverse-index) Better heartbeat mocking, improved heartbeats for reverse index construction.	2023-08-29 13:07:55 +02:00
build.gradle	(refactor) Remove converting-model package completely	2023-09-14 11:21:44 +02:00
readme.md	More restructuring, big bug fixes in keyword extraction.	2023-03-13 17:39:53 +01:00

Crawling Process

The crawling process downloads HTML and saves them into per-domain snapshots.

Central Classes

CrawlerMain orchestrates the crawling.
CrawlerRetreiver visits known addresses from a domain and downloads each document.
HttpFetcher fetches a URL.