161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9f672a0cf4
CatgirlIntelligenceAgency/code/libraries/term-frequency-dict
History
Viktor Lofgren 3101b74580 (index) Move to a lexicon-free index design 
This is a system-wide change.  The index used to have a lexicon, mapping words to wordIds using a large in-memory hash table.   This made index-construction easier, but it
also added a fairly significant RAM penalty to both the index service and the loader.

The new design moves to 64 bit word identifiers calculated using the murmur hash of the keyword, and an index construction based on merging smaller indices.

It also became necessary half-way through to upgrade guice as its error reporting wasn't *quite* compatible with JDK20.
2023-08-28 14:02:23 +02:00
..
src/main/java/nu/marginalia/term_frequency_dict (index) Move to a lexicon-free index design 2023-08-28 14:02:23 +02:00
build.gradle Upgrade antique lombok plugin 2023-08-23 14:34:32 +00:00
readme.md Remove unrelated code, break tools into their own directory. 2023-03-17 16:03:11 +01:00

readme.md

Term Frequency Dictionary

This dictionary is used by various parts of the system to evaluate for example the TF-IDF score of a keyword.

Central Classes

See Also