0307c55f9f
To avoid having to either hard-code or manually configure service addresses (possibly several dozen), and to reduce the project's dependency on docker to deal with routing and discovery, the option to use [Zookeeper](https://zookeeper.apache.org/) to manage services and discovery has been added. A service registry interface was added, with a Zookeeper implementation and a basic implementation that only works on docker and hard-codes everything. The last remaining REST service, the assistant-service, has been migrated to gRPC. This also proved a good time to clear out primordial technical debt from the root of the codebase. The 'service-client' library has been taken behind the barn and given a last farewell. It's replaced by a small library for managing gRPC channels. Since it's no longer used by anything, RxJava has been removed as a dependency from the project. Although the current state seems reasonably stable, this is a work-in-progress commit. |
||
|---|---|---|
| .. | ||
| src | ||
| build.gradle | ||
| readme.md | ||
Forward Index
The forward index contains a mapping from document id to various forms of document metadata.
In practice, the forward index consists of two files, an id file and a data file.
The id file contains a list of sorted document ids, and the data file contains
metadata for each document id, in the same order as the id file, with a fixed
size record containing data associated with each document id.
Each record contains a binary encoded DocumentMetadata object, as well as a HtmlFeatures bitmask.
Unlike the reverse index, the forward index is not split into two tiers, and the data is in the same order as it is in the source data, and the cardinality of the document IDs is assumed to fit in memory, so it's relatively easy to construct.
Central Classes
- ForwardIndexConverter constructs the index.
- ForwardIndexReader interrogates the index.