History

Viktor Lofgren 46d761f34f (language) fasttext based language filter		2023-08-16 15:48:12 +02:00
..
src/main/java/nu/marginalia/tools	(language) fasttext based language filter	2023-08-16 15:48:12 +02:00
build.gradle	Remove unrelated code, break tools into their own directory.	2023-03-17 16:03:11 +01:00
readme.md	Remove unrelated code, break tools into their own directory.	2023-03-17 16:03:11 +01:00

readme.md

Term Frequency Extractor

Generates a term frequency dictionary file from a batch of crawl data.

Usage:

PATH_TO_SAMPLES=run/samples/crawl-s
export JAVA_OPTS=-Dcrawl.rootDirRewrite=/crawl:${PATH_TO_SAMPLES} 

term-frequency-extractor ${PATH_TO_SAMPLES}/plan.yaml out.dat

readme.md

Term Frequency Extractor

See Also