161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
bd84c73e05
CatgirlIntelligenceAgency/crawl/crawl-job-extractor-process/readme.md
Viktor Lofgren b945fd7f39 A lot of readmes, some refactoring.
2023-03-06 18:32:13 +01:00

178 B
Raw Blame History

Crawl Job Extractor

The crawl job extractor creates a file containing a list of domains along with known URLs. This is consumed by the crawling-process.