29 lines
794 B
Markdown
29 lines
794 B
Markdown
|
Parquet is used as an intermediate storage format for a lot of processed data.
|
||
|
|
|
||
|
|
See [third-party/parquet-floor](../third-party/parquet-floor).
|
||
|
|
|
||
|
|
## How to query the data?
|
||
|
|
|
||
|
|
[DuckDB](https://duckdb.org/) is probably the best tool for interacting with these files. You can
|
||
|
|
query them with SQL, like
|
||
|
|
|
||
|
|
```sql
|
||
|
|
SELECT foo,bar FROM 'baz.parquet' ...
|
||
|
|
```
|
||
|
|
|
||
|
|
## How to inspect word metadata from `documentNNNN.parquet` ?
|
||
|
|
|
||
|
|
The document keywords records contain repeated values. For debugging these
|
||
|
|
repeated values, they can be unnested in e.g. DuckDB with a query like
|
||
|
|
|
||
|
|
```sql
|
||
|
|
SELECT word, hex(wordMeta) from
|
||
|
|
(
|
||
|
|
SELECT
|
||
|
|
UNNEST(word) AS word,
|
||
|
|
UNNEST(wordMeta) AS wordMeta
|
||
|
|
FROM 'document0000.parquet'
|
||
|
|
WHERE url='...'
|
||
|
|
)
|
||
|
|
WHERE word IN ('foo', 'bar')
|
||
|
|
```
|