161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

(docs) Improve architectural documentation

Browse Source
This commit is contained in:
Viktor Lofgren 2023-11-30 21:38:57 +01:00
parent 166a391eae
commit e5d274fe1c
1 changed files with 14 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
code/readme.md
View File

@ -12,14 +12,24 @@ A map of the most important components and how they relate can be found below.
![image](../doc/diagram/conceptual-overview.svg) ![image](../doc/diagram/conceptual-overview.svg)
The core part of the search engine is the index service, which is responsible for storing and retrieving
the document data. The index serive is partitioned, along with the executor service, which is responsible for executing
processes. At least one instance of each service must be run, but more can be run
alongside. Multiple partitions is desirable in production to distribute load across multiple physical drives,
as well as reducing the impact of downtime.
Search queries are delegated via the query service, which is a proxy that fans out the query to all
eligible index services. The control service is responsible for distributing commands to the executor
service, and for monitoring the health of the system. It also offers a web interface for operating the system.
### Services ### Services
* [core services](services-core/) "macroservices", stateful, memory hungry doing heavy lifting. * [core services](services-core/) Most of these services are stateful, memory hungry, and doing heavy lifting.
* * [control](services-core/control-service) * * [control](services-core/control-service)
* * [query](services-core/query-service) * * [query](services-core/query-service)
* * [index](services-core/index-service) * * [index](services-core/index-service)
* * [executor](services-core/executor-service) * * [executor](services-core/executor-service)
* * [assistant](services-core/assistant-service) * * [assistant](services-core/assistant-service)
* [application services](services-application/) "microservices", stateless providing additional functionality and making an application out of the search engine. * [application services](services-application/) Mostly stateless gateways providing access to the core services.
* * [api](services-application/api-service) - public API * * [api](services-application/api-service) - public API
* * [search](services-application/search-service) - marginalia search application * * [search](services-application/search-service) - marginalia search application
* * [dating](services-application/dating-service) - [https://explore.marginalia.nu/](https://explore.marginalia.nu/) * * [dating](services-application/dating-service) - [https://explore.marginalia.nu/](https://explore.marginalia.nu/)
@ -28,7 +38,8 @@ A map of the most important components and how they relate can be found below.
### Processes ### Processes
Processes are batch jobs that deal with data retrieval, processing and loading. Processes are batch jobs that deal with data retrieval, processing and loading. These are spawned and orchestrated by
the executor service, which is controlled by the control service.
* [processes](processes/) * [processes](processes/)
* * [crawling-process](processes/crawling-process) * * [crawling-process](processes/crawling-process)