bf44805e69
This variable had a very confusing name, and was dangerously easy to use in the wrong place with the result of getting something that only works as expected half the time. Ideally this class needs an overhaul, the assumptions it makes about domain names aren't great. |
||
|---|---|---|
| .. | ||
| content-type | ||
| crawl-blocklist | ||
| link-parser | ||
| readme.md | ||
Crawl Features
These are bits of search-engine related code that are relatively isolated pieces of business logic, that benefit from the clarity of being kept separate from the rest of the crawling code.
- crawl-blocklist - IP and URL blocklists
- link-parser - Code for parsing and normalizing links