Get quickstart to download straight from HuggingFace

[jerrylin96]

Quickstart assumes data is locally downloaded, which may not be ideal for some users.

[cisaacstern]

Good call.

I've never used it but have heard good things about https://pypi.org/project/pooch/ for this use case.

[jackmiller2003]

When I was downloading the data I used huggingface_hub although this may add too many additional dependencies. I'm not sure if this can be done with pooch but hub allows for really easy download of subsets. For example, here is a script that I used to get just one day's worth of data:

import datasets
import logging
from pathlib import Path
from huggingface_hub import snapshot_download

logging.basicConfig(level=logging.INFO)

custom_cache_dir = ""

datasets.config.DOWNLOADED_DATASETS_PATH = Path(custom_cache_dir)

# Download the dataset from the Hugging Face Hub
download_path = snapshot_download(
    repo_id="LEAP/ClimSim_low-res",
    allow_patterns="train/0001-02/*0001-02-01-*.nc",
    cache_dir=custom_cache_dir,
    repo_type="dataset",
)

print(f"Downloaded the dataset to {download_path}")