524 B
524 B
Crawling Process
The crawling process downloads HTML and saves them into per-domain snapshots.
Central Classes
- CrawlerMain orchestrates the crawling.
- CrawlerRetreiver visits known addresses from a domain and downloads each document.
- HttpFetcher fetches a URL.