161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6f222b9800
CatgirlIntelligenceAgency/code/features-index
History
Viktor Lofgren 704de50a9b (forward-index, valuator) HTML features in valuator 
Put it in the forward index for easy access during index-side valuation.
2023-08-18 11:54:56 +02:00
..
domain-ranking Move database to a separate module 2023-03-25 15:26:17 +01:00
index-forward (forward-index, valuator) HTML features in valuator 2023-08-18 11:54:56 +02:00
index-journal (forward-index, valuator) HTML features in valuator 2023-08-18 11:54:56 +02:00
index-query Use document generator to complement the document selection. 2023-06-22 17:21:33 +02:00
index-reverse (forward-index, valuator) HTML features in valuator 2023-08-18 11:54:56 +02:00
lexicon (lexicon) Optimize lexicon by using Murmur3_128's hash function 2023-08-01 15:02:13 +02:00
readme.md Update readme.md 2023-03-20 16:27:37 +01:00

readme.md

Index

These are components that offer functionality for the index-service.

Indexes

There are two indexes with accompanying tools for constructing them.

  • index-reverse is code for word->document indexes. There are two such indexes, one containing only document-word pairs that are flagged as important, e.g. the word appears in the title or has a high TF-IDF. This allows good results to be discovered quickly without having to sift through ten thousand bad ones first.

  • index-forward is the document->word index containing metadata about each word, such as its position. It is used after identifying candidate search results via the reverse index to fetch metadata and rank the results.

These indices rely heavily on the libraries/btree and libraries/array components.

Algorithms

Libraries

  • index-query contains structures for evaluating search queries.
  • index-journal contains tools for writing and reading index data.
  • lexicon contains a mapping between words' string representation and an unique integer identifier.