161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3101b74580
CatgirlIntelligenceAgency/code/features-convert/keyword-extraction
History
Viktor Lofgren 3101b74580 (index) Move to a lexicon-free index design 
This is a system-wide change.  The index used to have a lexicon, mapping words to wordIds using a large in-memory hash table.   This made index-construction easier, but it
also added a fairly significant RAM penalty to both the index service and the loader.

The new design moves to 64 bit word identifiers calculated using the murmur hash of the keyword, and an index construction based on merging smaller indices.

It also became necessary half-way through to upgrade guice as its error reporting wasn't *quite* compatible with JDK20.
2023-08-28 14:02:23 +02:00
..
src (index) Move to a lexicon-free index design 2023-08-28 14:02:23 +02:00
build.gradle Upgrade antique lombok plugin 2023-08-23 14:34:32 +00:00
readme.md Break apart reverse index into a separate full index and priority index. It did this before using the same code. This will make the priority index about half as big since it no longer needs to keep metadata. 2023-03-21 16:12:31 +01:00

readme.md

Keyword Extraction

This code deals with identifying keywords in a document, their positions in the document, their important based on TF-IDF and their grammatical functions based on POS tags.

Central Classes

See Also