History

Viktor Lofgren 199c459697 (*) Add node-affinity to services, processes and file storage.		2023-10-10 12:32:22 +02:00
..
src	(*) Add node-affinity to services, processes and file storage.	2023-10-10 12:32:22 +02:00
build.gradle	(build) Move unit test configuration to root build.gradle	2023-10-04 12:46:22 +02:00
readme.md	(refactor) Remove features-search and update documentation	2023-10-09 15:12:30 +02:00

Crawling Process

The crawling process downloads HTML and saves them into per-domain snapshots.

Central Classes

CrawlerMain orchestrates the crawling.
CrawlerRetreiver visits known addresses from a domain and downloads each document.
HttpFetcher fetches a URL.