f58a9f46be
This behavior is an old vestige from the days of only having a single loader process. We'd truncate the links table because doing inserts/updates was too slow. This was also important because we had 32 bit ID, and there's a lot of links between domains to go around... Instead we delete the rows associated with the current node with a stored procedure PURGE_LINKS_TABLE. We also update the PRIMARY KEY to a BIGINT. We'll need to load the data in excess of billion times to hit an ID rollover, so it'll be fine. |
||
|---|---|---|
| .. | ||
| api | ||
| common | ||
| features-convert | ||
| features-crawl | ||
| features-index | ||
| features-qs | ||
| features-search | ||
| libraries | ||
| process-models | ||
| processes | ||
| services-application | ||
| services-core | ||
| tools | ||
| readme.md | ||
Code
This is a pretty large and diverse project with many moving parts.
You'll find a short description in each module of what it does and how it relates to other modules. The modules each have names like "library" or "process" or "feature". These have specific meanings. See doc/module-taxonomy.md.
Overview
A map of the most important components and how they relate can be found below.
Services
- core services "macroservices", stateful, memory hungry doing heavy lifting.
- application services "microservices", stateless providing additional functionality and making an application out of the search engine.
-
- api - public API
-
- search - marginalia search application
- an internal API
Processes
Processes are batch jobs that deal with data retrieval, processing and loading.
Tools
Features
Features are relatively stand-alone components that serve some part of the domain. They aren't domain-independent, but isolated.
Libraries and primitives
Libraries are stand-alone code that is independent of the domain logic.