161sh/CatgirlIntelligenceAgency

History

Viktor Lofgren a9f7b4c457 Add synthetic keywords for same-site files linked from a document (e.g. file:png). Also add category keywords, like file:image or file:document.		2023-04-30 19:29:13 +02:00
..
src	Add synthetic keywords for same-site files linked from a document (e.g. file:png). Also add category keywords, like file:image or file:document.	2023-04-30 19:29:13 +02:00
build.gradle	Yet more restructuring. Improved search result ranking.	2023-03-16 21:35:54 +01:00
readme.md	More restructuring, big bug fixes in keyword extraction.	2023-03-13 17:39:53 +01:00

readme.md

Converting Process

The converting process reads crawl data and extracts information to be fed into the index, such as keywords, metadata, urls, descriptions...

Central Classes

ConverterMain orchestrates the conversion process.
DocumentProcessor converts a single document.
- HtmlDocumentProcessorPlugin has HTML-specific logic related to a document, keywords and identifies features such as whether it has javascript.
- PlainTextDocumentProcessorPlugin has plain text-specific logic related to a document...
DomainProcessor converts each document and generates domain-wide metadata such as link graphs.

See Also

features-convert