CatgirlIntelligenceAgency/third-party/README.md

# Third Party Code

This is a mix of code from other projects, that has either been aggressively modified to suite the needs of the project,
or lack an artifact, or to override some default that is inappropriate for the type of data Marginalia throws at the library.

## Sources and Licenses

### Modified
* [RDRPosTagger](rdrpostagger/) - GPL3
* [PorterStemmer](porterstemmer/) - LGPL3
* [OpenZIM](openzim/) - GPL-2.0+
* [Commons Codec](commons-codec/) - Apache 2.0
* [encylopedia.marginalia.nu](encyclopedia-marginalia-nu/) - GPL 2.0+

### Repackaged
* [SymSpell](symspell/) - LGPL-3.0
* [Count-Min-Sketch](count-min-sketch/) - Apache 2.0

### Monkey Patched
* [Stanford OpenNLP](monkey-patch-opennlp/) - Apache-2.0
first commit 2022-05-19 17:45:26 +02:00			`# Third Party Code`

			`This is a mix of code from other projects, that has either been aggressively modified to suite the needs of the project,`
Update 3rd party readme 2023-01-30 10:22:28 +01:00			`or lack an artifact, or to override some default that is inappropriate for the type of data Marginalia throws at the library.`
first commit 2022-05-19 17:45:26 +02:00
			`## Sources and Licenses`
Update 3rd party readme 2023-01-30 10:22:28 +01:00
A lot of readmes, some refactoring. 2023-03-06 18:32:13 +01:00			`### Modified`
The refactoring will continue until morale improves. 2023-03-12 10:04:48 +01:00			`* [RDRPosTagger](rdrpostagger/) - GPL3`
			`* [PorterStemmer](porterstemmer/) - LGPL3`
(converter) Integrate zim->db conversion into automatic encyclopedia processing workflow Previously, in order to load encyclopedia data into the search engine, it was necessary to use the encyclopedia.marginalia.nu converter to first create a .db-file. This isn't very ergonomic, so parts of that code-base was lifted in as a 3rd party library, and conversion from .zim to .db is now done automatically. The output file name is based on the original filename, plus a crc32 hash and a .db-ending, to ensure we can recycle the data on repeat loads. 2024-01-19 13:59:03 +01:00			`* [OpenZIM](openzim/) - GPL-2.0+`
(hash) Modified version of common codec's Murmur3 hash 2023-08-01 14:57:40 +02:00			`* [Commons Codec](commons-codec/) - Apache 2.0`
(converter) Integrate zim->db conversion into automatic encyclopedia processing workflow Previously, in order to load encyclopedia data into the search engine, it was necessary to use the encyclopedia.marginalia.nu converter to first create a .db-file. This isn't very ergonomic, so parts of that code-base was lifted in as a 3rd party library, and conversion from .zim to .db is now done automatically. The output file name is based on the original filename, plus a crc32 hash and a .db-ending, to ensure we can recycle the data on repeat loads. 2024-01-19 13:59:03 +01:00			`* [encylopedia.marginalia.nu](encyclopedia-marginalia-nu/) - GPL 2.0+`
(array) Remove unused 'madvise' code and 3rd party dependency on 'uppend' This wasn't actually hooked in anywhere. Removing the dependency and code. If it turns out we need madvise in the future, we'll re-introducde it. 2024-01-22 12:56:45 +01:00
Yet more restructuring. Improved search result ranking. 2023-03-16 21:35:54 +01:00			`### Repackaged`
The refactoring will continue until morale improves. 2023-03-12 10:04:48 +01:00			`* [SymSpell](symspell/) - LGPL-3.0`
Yet more restructuring. Improved search result ranking. 2023-03-16 21:35:54 +01:00			`* [Count-Min-Sketch](count-min-sketch/) - Apache 2.0`
A lot of readmes, some refactoring. 2023-03-06 18:32:13 +01:00
			`### Monkey Patched`
(deps) Remove monkey patched GSON The codebase used to have a monkey patched version of gson that made special optimizations for the unusually large JSON files that used to store e.g. crawl data. Since JSON is no longer used in this fashion, the GSON fork is not needed anymore. 2024-02-06 12:11:39 +01:00			`* [Stanford OpenNLP](monkey-patch-opennlp/) - Apache-2.0`