161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
db0216936e
CatgirlIntelligenceAgency/code/features-crawl
History
Viktor Lofgren 74644d59f3 (crawler) Update URL blocklist 
* Don't crawl MDN mirrors
* More mailing list variants
2023-07-10 18:04:43 +02:00
..
crawl-blocklist (crawler) Update URL blocklist 2023-07-10 18:04:43 +02:00
link-parser The refactoring will continue until morale improves. 2023-03-12 11:42:07 +01:00
readme.md Yet more restructuring. Improved search result ranking. 2023-03-16 21:35:54 +01:00

readme.md

Crawl Features

These are bits of search-engine related code that are relatively isolated pieces of business logic, that benefit from the clarity of being kept separate from the rest of the crawling code.