feat: huggingface/download: support whole repository downloads

[mwiewior]

vepyr stores its prebuilt annotation caches on Hugging Face as public dataset repositories, for example biodatageeks/vepyr_116_GRCh38_ensembl. Each cache is a directory tree of Parquet shards and manifests. The existing huggingface/download module requires a single filename and emits that file; vepyr needs the cache directory to be downloaded and staged as a whole, optionally restricted to selected chromosomes. See the vepyr cache download documentation for the cache layout and Hugging Face download instructions.

This PR extends huggingface/download to provide that reusable directory-download capability. Its scope is confined to this module. Integration into vcf_annotate_vepyr will follow in #13070 after this dependency is merged.

Changes:

• Accept [] or '' for hf_file to download a repository and emit its directory through the existing output channel. The directory name comes from ext.prefix, defaulting to meta.id.
• Preserve the existing input tuple, output channel and behaviour for single-file downloads. Update stubs to support repository directories and nested file paths.
• Support dataset selection, revision pinning and include/exclude filters through the existing ext.args mechanism. For vepyr caches, use --repo-type dataset; chromosome subsets must include the corresponding Parquet shards and every entity's chrom_manifest.json.
• Refresh Docker/Singularity container metadata and AMD64/ARM64 Conda locks, retaining huggingface_hub=1.18.0. Supply the bundled CA certificates when SSL_CERT_FILE is unset so Xet downloads work in minimal containers, preserving TLS verification and explicit user overrides.
• Add download regression tests and a native_arm64 test profile selecting the published ARM64 Docker image.

Validation:

• All 10 module tests pass with both AMD64 Docker under emulation and native ARM64 Docker, using identical snapshots. Coverage includes single files, complete and filtered repositories, custom directory names, nested paths, stubs and an Xet-backed model file.
• Both downstream llamacpppython/run CPU tests pass with their existing snapshots. Upstream AMD64 Docker, Singularity and Conda test jobs pass.
• Local end-to-end verification downloads the published Ensembl 116 chr22 cache from Hugging Face, then runs bcftools/norm and vepyr/annotate. All 14 cache files match the Hub hashes; all 1,000 variants have CSQ annotations; the tabix index works; and the VCF body matches the existing subworkflow fixture exactly (MD5 1d4a92b815eb6193e6f4546f4ab35978). The cache is pinned to revision 15a048f0585a7a40d2d40cb1c2f7806638ee500b.
• nf-core module lint and repository pre-commit hooks pass.

Generated by Codex

PR checklist

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
• If necessary, include test data in your PR.
• Remove all TODO statements.
• Broadcast software version numbers to topic: versions - See version_topics
• Follow the naming conventions.
• Follow the parameters requirements.
• Follow the input/output options guidelines.
• Add a resource label
• Use BioConda and BioContainers if possible to fulfil software requirements.
• Ensure that the test works with either Docker / Singularity. Conda CI tests can be quite flaky:
  • For modules:
    • nf-core modules test huggingface/download --profile docker (equivalent nf-test test ... --profile=+docker)
    • nf-core modules test huggingface/download --profile singularity (upstream CI)
    • nf-core modules test huggingface/download --profile conda (upstream CI)

[Copilot]

Copilot review overview

🟢 Approval recommended

The focused implementation preserves existing behaviour and has comprehensive regression coverage across the new download modes.

Review effort: Balanced
Findings: None

What changed in this PR

Extends huggingface/download to support complete or filtered repository downloads while preserving single-file behaviour.

Changes:

• Adds directory downloads, nested paths, custom prefixes and CA bundle handling.
• Adds regression and ARM64 profile coverage.
• Refreshes multi-architecture container and Conda metadata.

File	Description
main.nf	Implements repository downloads and updated containers.
meta.yml	Documents the expanded interface and container metadata.
tests/​main.nf.test	Tests file, repository, filtered, Xet and stub modes.
tests/​main.nf.test.snap	Records the expanded test results.
tests/​nextflow.config	Configures arguments, prefixes and ARM64 testing.
.conda-lock/​linux_amd64-…txt	Adds the AMD64 dependency lock.
.conda-lock/​linux_arm64-…txt	Adds the ARM64 dependency lock.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[toniher]

Review follow-ups (pushed in 5f684dc):

1. Removed the native_arm64 profile from tests/nextflow.config. It duplicated the shared arm64 profile in tests/config/nf-test.config, which CI already adds on ARM runners (,arm64,wave). It was also the only module test config in the repo with a profiles {} block, and it hard-coded the ARM64 image a second time. For a local ARM run, use nf-test test modules/nf-core/huggingface/download/tests --profile=+docker,arm64.
2. The test config is now applied per test instead of once at the top. Following the testing spec, only the 5 tests that set params load ./nextflow.config. All 10 tests pass with the existing snapshots.
3. output_path stays type: file in meta.yml. The channel emits either a file or a directory depending on hf_file, and keeping one channel preserves the existing API for downstream users such as llamacpppython/run. The description already covers the directory case.
4. Documented the SSL_CERT_FILE behaviour in the meta.yml description. When the variable is unset, the module uses certifi's CA bundle so that Xet downloads work in minimal containers. Anyone behind a TLS-intercepting proxy, or with a custom CA, should export SSL_CERT_FILE.

Comment added by Claude Opus 5.5

[toniher]

Looks Good! Provided minor modifications per existing recommendations