161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

(scripts|docs) Update scripts and documentations for the new operator's gui and file storage workflows.

Browse Source
This commit is contained in:
Viktor Lofgren 2023-08-01 22:50:05 +02:00
parent ba724bc1b2
commit 19402772fc
1 changed files with 2 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
doc/crawling.md
View File

@ -30,22 +30,9 @@ This can be done by editing the file `${WMSA_HOME}/conf/user-agent`.
## Setup
To operate the crawler, you need to set up a filesystem structure.
Ensure that the system is running and go to https://localhost:8081. See the documentation in [run/](../run/) for more information.
By default the system is configured to store data in `run/samples`. (!!!FIXME: How do you change this now?!!!)
You need
* a directory for crawl data
* a directory for processed data
* a crawl specification file
* a crawl plan file
Assuming we want to keep our crawl and processed data in
`/data`, then we would create the following directories:
```bash
$ mkdir /data/crawl
$ mkdir /data/processed
```
### Specifications