Consider columnar storage

[adsharma]

This is about the huggingface datasets.

Many of them are either compressed csv/json dumps which are not viewable/queryable using the huggingface UI. Have you considered using parquet/duckdb file formats?

I have some scripts to process llama3*.zip files to produce parquet/duckdb. They produce a entity -> event -> event graph. Not sure about concepts graph.

[gzone0111]

Thanks for the suggestion! We will definitely consider this. We agree that parquet would be much better for the Hugging Face Viewer, and we will try our best to upload parquet compatible versions of the data soon.

[adsharma]

Also consider this variation of parquet files

https://github.com/adsharma/graph-std

LadybugDB can query such graphs in-place without having to import first.

[adsharma]

@gzone0111 any update on this? Are you still looking to upload parquet to huggingface?