161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ed373eef61
CatgirlIntelligenceAgency/code/tools/experiment-runner
History
Viktor Lofgren ed373eef61 Refactor crawler and add special logic for some platforms 
* Break apart CrawlerRetreiver
* Break apart HttpFetcher into an interface and impl for testing sanity
* Add special logic for Lemmy, Mediawiki and Discourse to not waste requests on paths that aren't interesting.
2023-06-27 10:57:54 +02:00
..
src/main/java/nu/marginalia/tools Refactor crawler and add special logic for some platforms 2023-06-27 10:57:54 +02:00
build.gradle Add experiment runner tool and got rid of experiments module in processes. 2023-03-28 16:58:46 +02:00
readme.md Add experiment runner tool and got rid of experiments module in processes. 2023-03-28 16:58:46 +02:00

readme.md

Experiment Runner

This tool is a means of launching crawl data processing experiments, for interacting with crawl data.

It's launched with run/experiment.sh. New experiments need to be added to ExperimentRunnerMain in order for the script to be able to run them.