161sh/CatgirlIntelligenceAgency

Viktor Lofgren 46f81aca2f Break apart reverse index into a separate full index and priority index. It did this before using the same code. This will make the priority index about half as big since it no longer needs to keep metadata.

2023-03-21 16:12:31 +01:00

1.0 KiB

Raw Blame History

Reverse Index

The reverse index contains a mapping from word to document id.

There are two tiers of this index, one priority index which only indexes terms that are flagged with priority flags¹, and a full index that indexes all terms. The full index also provides access to term-level metadata, while the priority index is a binary index.

[1] See WordFlags in common/model and KeywordMetadata in features-convert/keyword-extraction.

Central Classes

ReverseIndexFullConverter constructs the full index.
ReverseIndexFullReader interrogates the full index.
ReverseIndexPriorityConverter constructs the priority index.
ReverseIndexPriorityReader interrogates the priority index.