161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6d939175b1
CatgirlIntelligenceAgency/code/crawl/crawl-job-extractor-process
History
Viktor Lofgren 6d939175b1 Additional code restructuring to get rid of util and misc-style packages.
2023-03-11 13:48:40 +01:00
..
src Additional code restructuring to get rid of util and misc-style packages. 2023-03-11 13:48:40 +01:00
build.gradle Gradle shouldn't generate dist zips, zipping jar files is slow and also just ridiculous when you realize jar files are zip files and you can't compress a file twice using the same algo. 2023-03-11 11:34:51 +01:00
readme.md Move all code to a code directory. 2023-03-07 17:14:32 +01:00

readme.md

Crawl Job Extractor

The crawl job extractor creates a file containing a list of domains along with known URLs. This is consumed by the crawling-process.