161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10bad635a8
CatgirlIntelligenceAgency/code/features-crawl
History
Viktor Lofgren 29ddf9e61d (doc) Update docs
2024-02-06 16:29:55 +01:00
..
content-type (crawler) Fix rare exception in content type handling due to improper length checking of a split() array 2024-01-18 11:05:45 +01:00
crawl-blocklist (*) Overhaul settings and properties 2024-01-13 17:12:18 +01:00
link-parser (crawler) Improve meta-tag redirect handling, add tests for redirects. 2024-02-01 20:30:43 +01:00
readme.md (doc) Update docs 2024-02-06 16:29:55 +01:00

readme.md

Crawl Features

These are bits of search-engine related code that are relatively isolated pieces of business logic, that benefit from the clarity of being kept separate from the rest of the crawling code.