161sh/CatgirlIntelligenceAgency

History

Viktor Lofgren 5a5430b383 (convert) Wiki specialization that should do a better job at removing junk keywords and providing a useful summary.		2023-11-30 20:04:46 +01:00
..
src	(convert) Wiki specialization that should do a better job at removing junk keywords and providing a useful summary.	2023-11-30 20:04:46 +01:00
build.gradle	Initial Commit Anchor Tags	2023-11-04 14:24:17 +01:00
readme.md	More restructuring, big bug fixes in keyword extraction.	2023-03-13 17:39:53 +01:00

readme.md

Converting Process

The converting process reads crawl data and extracts information to be fed into the index, such as keywords, metadata, urls, descriptions...

Central Classes

ConverterMain orchestrates the conversion process.
DocumentProcessor converts a single document.
- HtmlDocumentProcessorPlugin has HTML-specific logic related to a document, keywords and identifies features such as whether it has javascript.
- PlainTextDocumentProcessorPlugin has plain text-specific logic related to a document...
DomainProcessor converts each document and generates domain-wide metadata such as link graphs.

See Also

features-convert