CatgirlIntelligenceAgency/code
Viktor Lofgren f655ec5a5c (*) Refactor GeoIP-related code
In this commit, GeoIP-related classes are refactored and relocated to a common library as they are shared across multiple services.

The crawler is refactored to enable the GeoIpBlocklist to use the new GeoIpDictionary as the base of its decisions.

The converter is modified ot query this data to add a geoip:-keyword to documents to permit limiting a search to the country of the hosting server.

The commit also adds due BY-SA attribution in the search engine footer for the source of the IP geolocation data.
2023-12-10 17:30:43 +01:00
..
api (search) IP and IP geolocation in site info view 2023-12-09 20:06:55 +01:00
common (search) Move site information out of the search service and into assistant. 2023-12-09 16:30:06 +01:00
features-convert (test) Don't fail test if atags.parquet is not in ~vlofgren 2023-11-15 09:11:38 +01:00
features-crawl (*) Refactor GeoIP-related code 2023-12-10 17:30:43 +01:00
features-index (result-ranking) Nudge up results with ngram matches a tiny bit 2023-11-06 13:14:22 +01:00
features-qs (refactor) Remove features-search and update documentation 2023-10-09 15:12:30 +02:00
features-search (search) Merge similar sites results with the info view. 2023-12-04 22:10:24 +01:00
libraries (*) Refactor GeoIP-related code 2023-12-10 17:30:43 +01:00
process-models (*) Refactor GeoIP-related code 2023-12-10 17:30:43 +01:00
processes (*) Refactor GeoIP-related code 2023-12-10 17:30:43 +01:00
services-application (*) Refactor GeoIP-related code 2023-12-10 17:30:43 +01:00
services-core (*) Refactor GeoIP-related code 2023-12-10 17:30:43 +01:00
tools Initial Commit Anchor Tags 2023-11-04 14:24:17 +01:00
readme.md (docs) Improve architectural documentation 2023-11-30 21:38:57 +01:00

Code

This is a pretty large and diverse project with many moving parts.

You'll find a short description in each module of what it does and how it relates to other modules. The modules each have names like "library" or "process" or "feature". These have specific meanings. See doc/module-taxonomy.md.

Overview

A map of the most important components and how they relate can be found below.

image

The core part of the search engine is the index service, which is responsible for storing and retrieving the document data. The index serive is partitioned, along with the executor service, which is responsible for executing processes. At least one instance of each service must be run, but more can be run alongside. Multiple partitions is desirable in production to distribute load across multiple physical drives, as well as reducing the impact of downtime.

Search queries are delegated via the query service, which is a proxy that fans out the query to all eligible index services. The control service is responsible for distributing commands to the executor service, and for monitoring the health of the system. It also offers a web interface for operating the system.

Services

Processes

Processes are batch jobs that deal with data retrieval, processing and loading. These are spawned and orchestrated by the executor service, which is controlled by the control service.

Tools

Features

Features are relatively stand-alone components that serve some part of the domain. They aren't domain-independent, but isolated.

Libraries and primitives

Libraries are stand-alone code that is independent of the domain logic.

  • common elements for creating a service, a client etc.
  • libraries containing non-search specific code.
    • array - large memory mapped area library
    • btree - static btree library