CatgirlIntelligenceAgency/code/processes/crawl-job-extractor-process/readme.md

4 lines
178 B
Markdown

# Crawl Job Extractor
The crawl job extractor creates a file containing a list of domains
along with known URLs. This is consumed by the [crawling-process](../crawling-process).