161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2eb972dea1
CatgirlIntelligenceAgency/code/tools/crawl-job-extractor/readme.md

199 B
Raw Blame History

Crawl Job Extractor

The crawl job extractor creates a file containing a list of domains along with known URLs.

This is consumed by processes/crawling-process.