edc1acbb7e
The EC_DOMAIN_LINK MariaDB table stores links between domains. This is problematic, as both updating and querying this table is very slow in relation to how small the data is (~10 GB). This slowness is largely caused by the database enforcing ACID guarantees we don't particularly need. This changeset replaces the EC_DOMAIN_LINK table with a file in each index node containing 32 bit integer pairs corresponding to links between two domains. This file is loaded in memory in each node, and can be queried via the Query Service. A migration step is needed before this file is created in each node. Until that happens, the actual data is loaded from the EC_DOMAIN_LINK table, but accessed as though it was a file. The changeset also migrates/renames the links.db file to documents.db to avoid naming confusion between the two. |
||
|---|---|---|
| .. | ||
| domain-ranking | ||
| index-forward | ||
| index-journal | ||
| index-query | ||
| index-reverse | ||
| result-ranking | ||
| readme.md | ||
Index
These are components that offer functionality for the index-service.
Indexes
There are two indexes with accompanying tools for constructing them.
-
index-reverse is code for
word->documentindexes. There are two such indexes, one containing only document-word pairs that are flagged as important, e.g. the word appears in the title or has a high TF-IDF. This allows good results to be discovered quickly without having to sift through ten thousand bad ones first. -
index-forward is the
document->wordindex containing metadata about each word, such as its position. It is used after identifying candidate search results via the reverse index to fetch metadata and rank the results.
These indices rely heavily on the libraries/btree and libraries/array components.
Algorithms
- domain-ranking contains domain ranking algorithms.
- result-ranking contains logic for ranking search results by relevance.
Libraries
- index-query contains structures for evaluating search queries.
- index-journal contains tools for writing and reading index data.