History

Viktor Lofgren a56953c798 (converter, WIP) Refactor converter to not have to load everything into RAM.		2023-07-24 15:25:09 +02:00
..
src/main/java/nu/marginalia/tools	(converter, WIP) Refactor converter to not have to load everything into RAM.	2023-07-24 15:25:09 +02:00
build.gradle	Remove unrelated code, break tools into their own directory.	2023-03-17 16:03:11 +01:00
readme.md	Remove unrelated code, break tools into their own directory.	2023-03-17 16:03:11 +01:00

readme.md

Term Frequency Extractor

Generates a term frequency dictionary file from a batch of crawl data.

Usage:

PATH_TO_SAMPLES=run/samples/crawl-s
export JAVA_OPTS=-Dcrawl.rootDirRewrite=/crawl:${PATH_TO_SAMPLES} 

term-frequency-extractor ${PATH_TO_SAMPLES}/plan.yaml out.dat

readme.md

Term Frequency Extractor

See Also