161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ce8dca7659
CatgirlIntelligenceAgency/code/tools/stackexchange-converter
History
Viktor Lofgren c51159672e (build) Move unit test configuration to root build.gradle
2023-10-04 12:46:22 +02:00
..
src/main/java/nu/marginalia/tools (stackexchange-converter) Create tool for converting stackexchange 7z-files to digestible sqlite db:s 2023-09-20 15:15:13 +02:00
build.gradle (build) Move unit test configuration to root build.gradle 2023-10-04 12:46:22 +02:00
readme.md (stackexchange-converter) Create tool for converting stackexchange 7z-files to digestible sqlite db:s 2023-09-20 15:15:13 +02:00

readme.md

This tool converts from stackexchange's 7z-compressed XML format to a sqlite database that is digestible by the search engine.

See features-convert/stackexchange-xml for an explanation why this is necessary.

Stackexchange's data dumps can be downloaded from archive.org here: https://archive.org/details/stackexchange

Usage

$ stackexchange-converter domain-name input.7z output.db

Stackexchange is relatively conservative about allowing new questions, so this is a job that doesn't run more than once.

Note: Reading and writing these db files is absurdly slow on a mechanical hard-drive.

See Also