2023-03-04 16:42:31 +01:00
|
|
|
# Crawling Process
|
|
|
|
|
|
|
|
|
|
The crawling process downloads HTML and saves them
|
|
|
|
|
into per-domain snapshots.
|
|
|
|
|
|
|
|
|
|
## Central Classes
|
|
|
|
|
|
|
|
|
|
* [CrawlerMain](src/main/java/nu/marginalia/crawl/CrawlerMain.java) orchestrates the crawling.
|
|
|
|
|
* [CrawlerRetreiver](src/main/java/nu/marginalia/crawl/retreival/CrawlerRetreiver.java)
|
|
|
|
|
visits known addresses from a domain and downloads each document.
|
|
|
|
|
* [HttpFetcher](src/main/java/nu/marginalia/crawl/retreival/HttpFetcher.java)
|
|
|
|
|
fetches a URL.
|
2023-03-13 17:39:53 +01:00
|
|
|
|
|
|
|
|
## See Also
|
|
|
|
|
|
|
|
|
|
* [features-convert](../../features-convert/)
|
