3101b74580
This is a system-wide change. The index used to have a lexicon, mapping words to wordIds using a large in-memory hash table. This made index-construction easier, but it also added a fairly significant RAM penalty to both the index service and the loader. The new design moves to 64 bit word identifiers calculated using the murmur hash of the keyword, and an index construction based on merging smaller indices. It also became necessary half-way through to upgrade guice as its error reporting wasn't *quite* compatible with JDK20. |
||
|---|---|---|
| .. | ||
| src | ||
| build.gradle | ||
| readme.md | ||
Reverse Index
The reverse index contains a mapping from word to document id.
There are two tiers of this index.
- A priority index which only indexes terms that are flagged with priority flags1.
- A full index that indexes all terms.
The full index also provides access to term-level metadata, while the priority index is a binary index that only offers information about which documents has a specific word.
[1] See WordFlags in common/model and KeywordMetadata in features-convert/keyword-extraction.
Central Classes
- ReverseIndexFullConverter constructs the full index.
- ReverseIndexFullReader interrogates the full index.
- ReverseIndexPriorityConverter constructs the priority index.
- ReverseIndexPriorityReader interrogates the priority index.