2023-03-07 17:32:16 +01:00
|
|
|
# Code
|
|
|
|
|
|
|
|
|
|
This is a pretty large and diverse project with many moving parts.
|
2023-03-21 17:26:59 +01:00
|
|
|
|
2023-03-07 17:32:16 +01:00
|
|
|
You'll find a short description in each module of what it does and how it relates to other modules.
|
2023-03-21 17:26:59 +01:00
|
|
|
The modules each have names like "library" or "process" or "feature". These have specific meanings.
|
|
|
|
|
See [doc/module-taxonomy.md](../doc/module-taxonomy.md).
|
2023-03-07 17:32:16 +01:00
|
|
|
|
|
|
|
|
## Overview
|
|
|
|
|
|
2023-03-21 17:11:28 +01:00
|
|
|
A map of the most important components and how they relate can be found below.
|
|
|
|
|
|
2023-03-25 16:32:10 +01:00
|
|
|
|
2023-03-21 17:11:28 +01:00
|
|
|
|
2023-03-07 17:32:16 +01:00
|
|
|
### Services
|
|
|
|
|
* [core services](services-core/) "macroservices", stateful, memory hungry doing heavy lifting.
|
|
|
|
|
* * [search](services-core/search-service)
|
|
|
|
|
* * [index](services-core/index-service)
|
|
|
|
|
* * [assistant](services-core/assistant-service)
|
|
|
|
|
* [sattelite services](services-satellite/) "microservices", stateless providing additional functionality.
|
|
|
|
|
* * [api](services-satellite/api-service) - public API
|
|
|
|
|
* * [dating](services-satellite/dating-service) - [https://explore.marginalia.nu/](https://explore.marginalia.nu/)
|
|
|
|
|
* * [explorer](services-satellite/explorer-service) - [https://explore2.marginalia.nu/](https://explore2.marginalia.nu/)
|
|
|
|
|
* an [internal API](api/)
|
|
|
|
|
|
2023-03-12 11:42:07 +01:00
|
|
|
### Processes
|
|
|
|
|
|
|
|
|
|
Processes are batch jobs that deal with data retrieval, processing and loading.
|
|
|
|
|
|
2023-03-13 17:39:53 +01:00
|
|
|
* [processes](processes/)
|
|
|
|
|
* * [crawling-process](processes/crawling-process)
|
|
|
|
|
* * [converting-process](processes/converting-process)
|
|
|
|
|
* * [loading-process](processes/loading-process)
|
2023-03-12 11:42:07 +01:00
|
|
|
|
2023-03-17 16:03:11 +01:00
|
|
|
#### Tools
|
|
|
|
|
|
|
|
|
|
* * [crawl-job-extractor](tools/crawl-job-extractor)
|
|
|
|
|
* * [term-frequency-extractor](tools/term-frequency-extractor)
|
|
|
|
|
|
2023-03-12 10:50:31 +01:00
|
|
|
### Features
|
|
|
|
|
|
|
|
|
|
Features are relatively stand-alone components that serve some part of the domain. They aren't domain-independent,
|
|
|
|
|
but isolated.
|
|
|
|
|
|
|
|
|
|
* [features-search](features-search)
|
|
|
|
|
* [features-crawl](features-crawl)
|
2023-03-13 17:39:53 +01:00
|
|
|
* [features-convert](features-convert)
|
2023-03-12 10:50:31 +01:00
|
|
|
* [features-index](features-index)
|
|
|
|
|
|
2023-03-07 17:32:16 +01:00
|
|
|
### Libraries and primitives
|
2023-03-12 10:50:31 +01:00
|
|
|
|
|
|
|
|
Libraries are stand-alone code that is independent of the domain logic.
|
|
|
|
|
|
2023-03-07 17:32:16 +01:00
|
|
|
* [common](common/) elements for creating a service, a client etc.
|
|
|
|
|
* [libraries](libraries/) containing non-search specific code.
|
|
|
|
|
* * [array](libraries/array/) - large memory mapped area library
|
|
|
|
|
* * [btree](libraries/btree/) - static btree library
|
