161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3bbffd3c22
CatgirlIntelligenceAgency/code/features-crawl
History
Viktor Lofgren 064265b0b9 (crawler) Move content type/charset sniffing to a separate microlibrary 
This functionality needs to be accessed by the WarcSideloader, which is in the converter.  The resultant microlibrary is tiny, but I think in this case it's justifiable.
2023-12-07 15:16:37 +01:00
..
content-type (crawler) Move content type/charset sniffing to a separate microlibrary 2023-12-07 15:16:37 +01:00
crawl-blocklist (build) Move unit test configuration to root build.gradle 2023-10-04 12:46:22 +02:00
link-parser (build) Move unit test configuration to root build.gradle 2023-10-04 12:46:22 +02:00
readme.md Yet more restructuring. Improved search result ranking. 2023-03-16 21:35:54 +01:00

readme.md

Crawl Features

These are bits of search-engine related code that are relatively isolated pieces of business logic, that benefit from the clarity of being kept separate from the rest of the crawling code.