History

Viktor 52e2ab45bf Merge branch 'master' into master-control-program		2023-08-07 12:53:43 +02:00
..
src	Merge branch 'master' into master-control-program	2023-08-07 12:53:43 +02:00
build.gradle	(crawler) Reduce long term memory allocation in DomainCrawlFrontier	2023-07-29 19:18:12 +02:00
readme.md	More restructuring, big bug fixes in keyword extraction.	2023-03-13 17:39:53 +01:00

Crawling Process

The crawling process downloads HTML and saves them into per-domain snapshots.

Central Classes

CrawlerMain orchestrates the crawling.
CrawlerRetreiver visits known addresses from a domain and downloads each document.
HttpFetcher fetches a URL.