0889b6d247
This commit further cleans up the warc->parquet conversion. It fixes issues with redirect handling in WarcRecorder, adds support information about redirects and errors due to probe failure. It also refactors the fetch result, body extraction and content type abstractions. |
||
|---|---|---|
| .. | ||
| content-type | ||
| crawl-blocklist | ||
| link-parser | ||
| readme.md | ||
Crawl Features
These are bits of search-engine related code that are relatively isolated pieces of business logic, that benefit from the clarity of being kept separate from the rest of the crawling code.
- crawl-blocklist - IP and URL blocklists
- link-parser - Code for parsing and normalizing links