f655ec5a5c
In this commit, GeoIP-related classes are refactored and relocated to a common library as they are shared across multiple services. The crawler is refactored to enable the GeoIpBlocklist to use the new GeoIpDictionary as the base of its decisions. The converter is modified ot query this data to add a geoip:-keyword to documents to permit limiting a search to the country of the hosting server. The commit also adds due BY-SA attribution in the search engine footer for the source of the IP geolocation data. |
||
|---|---|---|
| .. | ||
| src | ||
| build.gradle | ||
| readme.md | ||
The processed-data package contains models and logic for reading and writing parquet files with the output from the converting-process.
Main models:
Since parquet is a column based format, some of the readable models are projections that only read parts of the input file.