CatgirlIntelligenceAgency/code/api
Viktor Lofgren edc1acbb7e (*) Replace EC_DOMAIN_LINK table with files and in-memory caching
The EC_DOMAIN_LINK MariaDB table stores links between domains.  This is problematic, as both updating and querying this table is very slow in relation to how small the data is (~10 GB).  This slowness is largely caused by the database enforcing ACID guarantees we don't particularly need.

This changeset replaces the EC_DOMAIN_LINK table with a file in each index node containing 32 bit integer pairs corresponding to links between two domains.  This file is loaded in memory in each node, and can be queried via the Query Service.

A migration step is needed before this file is created in each node.   Until that happens, the actual data is loaded from the EC_DOMAIN_LINK table, but accessed as though it was a file.

The changeset also migrates/renames the links.db file to documents.db to avoid naming confusion between the two.
2024-01-08 15:53:13 +01:00
..
assistant-api (search) Add view for showing mutual links between two websites 2023-12-17 17:50:44 +01:00
executor-api (control) Fix spurious state detection in control-side actors 2023-12-09 12:50:05 +01:00
index-api (*) Replace EC_DOMAIN_LINK table with files and in-memory caching 2024-01-08 15:53:13 +01:00
process-mqapi (convert) Add basic support for Warc file sideloading 2023-12-06 18:43:55 +01:00
query-api (*) Replace EC_DOMAIN_LINK table with files and in-memory caching 2024-01-08 15:53:13 +01:00
readme.md (refactor) Remove features-search and update documentation 2023-10-09 15:12:30 +02:00

Clients

Core Services

These are clients for the core services, along with what models are necessary for speaking to them. They each implement the abstract client classes from service-client.

All that is necessary is to @Inject them into the constructor and then requests can be sent.

Note: If you are looking for the public API, it's handled by the api service in services-application/api-service.

MQ-API Process API

process-mqapi defines requests and inboxes for the message queue based API used for interacting with processes.

See libraries/message-queue and services-application/control-service.