161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

(language-processing) Add a system property for configuring which language detection model to use

Browse Source 
The flag is `system.languageDetectionModelVersion`.

* If negative, no model is used.
* If 0, both models are used.
* If 1, the old crappy model is used.
* If 2, the new fasttext model is used.
This commit is contained in:
Viktor Lofgren 2024-01-31 13:02:33 +01:00
parent 9ce67029ca
commit d1e02569f4
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
code/libraries/language-processing/src/main/java/nu/marginalia/language/filter/LanguageFilter.java
View File

@ -28,7 +28,7 @@ public class LanguageFilter {
/** Returns the probability the language is in English */
public double dictionaryAgreement(DocumentLanguageData dld) {
if(LANGUAGE_DETECTION_MODEL_VERSION == -1) return 1.0;
if(LANGUAGE_DETECTION_MODEL_VERSION < 0) return 1.0;
if (LANGUAGE_DETECTION_MODEL_VERSION == 1) {
return languagePredictionModel2.predictEnglish(dld);