161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5ef17a2a20
CatgirlIntelligenceAgency/code/processes/crawl-job-extractor-process/readme.md

178 B
Raw Blame History

Crawl Job Extractor

The crawl job extractor creates a file containing a list of domains along with known URLs. This is consumed by the crawling-process.