3101b74580
This is a system-wide change. The index used to have a lexicon, mapping words to wordIds using a large in-memory hash table. This made index-construction easier, but it also added a fairly significant RAM penalty to both the index service and the loader. The new design moves to 64 bit word identifiers calculated using the murmur hash of the keyword, and an index construction based on merging smaller indices. It also became necessary half-way through to upgrade guice as its error reporting wasn't *quite* compatible with JDK20. |
||
|---|---|---|
| .. | ||
| src | ||
| build.gradle | ||
| readme.md | ||
DB
This module primarily contains SQL files for the URLs database. The most central tables are EC_DOMAIN, EC_URL and EC_PAGE_DATA.
Flyway
The system uses flyway to track database changes and allow easy migrations, this is accessible via gradle tasks.
flywayMigrateflywayBaselineflywayRepairflywayClean(dangerous as in wipes your entire database)
Refer to the Flyway documentation for guidance. It's well documented and these are probably the only four tasks you'll ever need.
If you are not running the system via docker, you need to provide alternative connection details than the defaults (TODO: how?).
The migration files are in resources/db/migration. The file name convention incorporates the project's cal-ver versioning; and are applied in lexicographical order.
VYY_MM_v_nnn__description.sql
Central Paths
- migrations - Flyway migrations
See Also
- common/service implements DatabaseModule, which is from where the services get database connections.
Relation diagrams for EC_DOMAIN and EC_URL
