CatgirlIntelligenceAgency/code/readme.md

Code

This is a pretty large and diverse project with many moving parts. You'll find a short description in each module of what it does and how it relates to other modules.

Overview

A map of the most important components and how they relate can be found below.

Services

• core services "macroservices", stateful, memory hungry doing heavy lifting.
• • search
• • index
• • assistant
• sattelite services "microservices", stateless providing additional functionality.
• • api - public API
• • dating - https://explore.marginalia.nu/
• • explorer - https://explore2.marginalia.nu/
• an internal API

Processes

Processes are batch jobs that deal with data retrieval, processing and loading.

• processes
• • crawling-process
• • converting-process
• • loading-process

Tools

• • crawl-job-extractor
• • term-frequency-extractor

Features

Features are relatively stand-alone components that serve some part of the domain. They aren't domain-independent, but isolated.

• features-search
• features-crawl
• features-convert
• features-index

Libraries and primitives

Libraries are stand-alone code that is independent of the domain logic.

• common elements for creating a service, a client etc.
• libraries containing non-search specific code.
• • array - large memory mapped area library
• • btree - static btree library