Skip to content

vepyr: add an unnormalized multiallelic input VCF - #2298

Closed
mwiewior wants to merge 1 commit into
nf-core:modulesfrom
mwiewior:vepyr-multiallelic
Closed

mwiewior wants to merge 1 commit into
nf-core:modulesfrom
mwiewior:vepyr-multiallelic

Conversation

@mwiewior

@mwiewior mwiewior commented Oct 4, 2026

Copy link
Copy Markdown

Adds an unnormalized input VCF for the vcf_annotate_vepyr subworkflow (nf-core/modules#13070). The reviewer there noted that the existing input.vcf.gz has no multiallelic records, so the subworkflow's bcftools norm step changes nothing and is never really tested.

Files, data/genomics/homo_sapiens/vepyr/

File Size Content
input_multiallelic.vcf.gz + .tbi 39 KB The same GIAB HG002 GRCh38 v4.2.1 window as input.vcf.gz (chr22:20572272-21735973), before normalization: 986 records, 14 of them multiallelic

It reuses the existing cache.tar.gz and reference.fa.gz, so nothing else is added.

Checked

  • bcftools norm --fasta-ref reference.fa.gz -m -both --do-not-normalize (the subworkflow's documented configuration) splits 14 records into 28. The record body of the output is byte-identical to input.vcf.gz (md5 ca50096a40b2796376f02df03587516e). Normalizing this file and then annotating it should therefore give the same result as annotating input.vcf.gz.
  • The files downloaded from this branch's raw URLs are byte-identical to the staged ones.
  • The header is the benchmark VCF's own, with the local ##bcftools_*Command lines removed.

Generated by Claude Code

PR checklist

  • I read the test-data specifications
  • I tried testing my module with available data and there is no compatible data in test-datasets. No VCF here covers the vepyr cache window with multiallelic records.
  • If I added data: I described the new files on the root README.md
  • I made sure to submit the smallest dataset possible (e.g. try .tar.gz for larger directories)

input_multiallelic.vcf.gz is the GIAB HG002 chr22:20572272-21735973 window of
input.vcf.gz before bcftools norm: 986 records, 14 multiallelic. Splitting it
with bcftools norm -m -both gives the records of input.vcf.gz, so the
vcf_annotate_vepyr subworkflow can test its normalization step.

Generated by Claude Code
mwiewior added a commit to mwiewior/modules that referenced this pull request Oct 4, 2026
- Prefix val_hf_repo with datasets/, so HUGGINGFACE_DOWNLOAD needs no
  required --repo-type in ext.args; only --revision/--include remain.
- Fail when both ch_cache and val_hf_repo are set, or neither, and test
  both failures.
- Test normalisation on input_multiallelic.vcf.gz (986 records, 14
  multiallelic; nf-core/test-datasets#2298): the output has 1,000
  biallelic records and the same variantsMD5 as annotating the
  pre-split input.vcf.gz without normalisation.
- Drop the basename test (covered by the bcftools/norm module tests)
  and use channel instead of Channel.
- meta.yml: reword the bcftools norm configuration, drop the ext.prefix
  paragraph, keep one "default to true", document that ch_cache and
  ch_fasta must be value channels and that BCF or plain VCF output
  makes VEPYR_ANNOTATE run a single pipeline.

Generated by Claude Code
@mwiewior
mwiewior requested a review from jfy133 October 4, 2026 16:35
@jonasscheid

Copy link
Copy Markdown

I think you dont need more testdata, you can use setup to manipulate right?

setup {
    run("UNTAR") { ... }   // unchanged
    run("BCFTOOLS_NORM", alias: "BCFTOOLS_JOIN_MULTIALLELICS") {
        script "modules/nf-core/bcftools/norm/main.nf"
        process {
            """
            input[0] = channel.of([
                [ id:'test_split' ],
                file(params.modules_testdata_base_path + 'genomics/homo_sapiens/vepyr/input.vcf.gz', checkIfExists: true),
                file(params.modules_testdata_base_path + 'genomics/homo_sapiens/vepyr/input.vcf.gz.tbi', checkIfExists: true)
            ])
            input[1] = channel.value([
                [ id:'GRCh38' ],
                file(params.modules_testdata_base_path + 'genomics/homo_sapiens/vepyr/reference.fa.gz', checkIfExists: true)
            ])
            """
        }
    }
}
// when: input[0] = BCFTOOLS_JOIN_MULTIALLELICS.out.vcf.map { _meta, vcf -> [ [ id:'test' ], vcf, [] ] }

It also needs this in tests/nextflow.config:

withName: 'BCFTOOLS_JOIN_MULTIALLELICS' {
    ext.args = '--multiallelics +both --output-type z'
}

Rest looks good to me

mwiewior added a commit to mwiewior/modules that referenced this pull request Oct 5, 2026
Instead of a new test-datasets file (nf-core/test-datasets#2298), the
normalise tests join input.vcf.gz back into multiallelic records with
an aliased BCFTOOLS_NORM in setup: --multiallelics +both turns 1,000
records into 986, 14 of them multiallelic, and the subworkflow splits
them again. --do-not-normalize is needed because the module always
passes --fasta-ref, and left-aligning fails on chr22 vs 22. Snapshots
are unchanged.

Generated by Claude Code
mwiewior added a commit to biodatageeks/vepyr that referenced this pull request Oct 5, 2026
Sync the vcf_annotate_vepyr tests with nf-core/modules#13070 643d2b157:
the normalise tests join input.vcf.gz back into multiallelic records
with an aliased BCFTOOLS_NORM in setup, so the staged
input_multiallelic.vcf.gz (nf-core/test-datasets#2298) is no longer
needed. Revert its staging in stage_testdata.py and the harness hash,
run the aliased process under amd64 emulation like BCFTOOLS_NORM, and
drop the #2298 references from the README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mwiewior

mwiewior commented Oct 5, 2026

Copy link
Copy Markdown
Author

hey @jonasscheid thanks for the hint! I fixed that in the main sub-workflow PR. Closing this one.

@mwiewior mwiewior closed this Oct 5, 2026
mwiewior added a commit to biodatageeks/vepyr that referenced this pull request Oct 5, 2026
…d badge (#167)

* nf-core: sync vcf_annotate_vepyr with nf-core/modules#13070

Mirror the subworkflow as it stands after the #13070 review round:
optional Hugging Face cache download (val_hf_repo, datasets/ prefix
added by the subworkflow), errors when both or neither of ch_cache and
val_hf_repo are set, normalization tests on a multiallelic input, and
the reworded meta.yml.

The dev harness follows: bcftools/norm pinned past #13076 (default
${meta.id}_norm prefix and collision guard), huggingface/download
fetched at the same commit and run under amd64 emulation, the parity
test passes the new seventh argument, and stage_testdata.py builds
input_multiallelic.vcf.gz from raw_chr22.vcf.gz, checked against the
md5 published in nf-core/test-datasets#2298.

Docs: nextflow.md drops the required ext.prefix, documents the
Hugging Face cache and value-channel inputs, and copies both helper
modules; the subworkflow diagram gains the download branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* nf-core: build the multiallelic test input in setup, as upstream

Sync the vcf_annotate_vepyr tests with nf-core/modules#13070 643d2b157:
the normalise tests join input.vcf.gz back into multiallelic records
with an aliased BCFTOOLS_NORM in setup, so the staged
input_multiallelic.vcf.gz (nf-core/test-datasets#2298) is no longer
needed. Revert its staging in stage_testdata.py and the harness hash,
run the aliased process under amd64 emulation like BCFTOOLS_NORM, and
drop the #2298 references from the README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: advertise the merged nf-core subworkflow

vcf_annotate_vepyr merged as nf-core/modules#13070 (7d5d5ab); the
mirror here is byte-identical to it. Install it with
`nf-core subworkflows install vcf_annotate_vepyr`, which also installs
vepyr/annotate, bcftools/norm and huggingface/download, instead of
copying it from this repository. Add an nf-core subworkflow badge next
to the module badge and mark the submission steps done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
famosab pushed a commit to famosab/modules that referenced this pull request Oct 5, 2026
* Add vcf_annotate_vepyr subworkflow

Compose optional bcftools normalisation with vepyr annotation and add published-fixture tests for indexed, unindexed, bypass and stub paths.

Generated by Codex

* vcf_annotate_vepyr: support Hugging Face cache repositories

Add an optional repository input that downloads one shared cache before annotation, retaining the existing local-cache path. Document revision and contig filters. Exercise local-cache bypass, remote stubs and a real two-sample annotation using the published chr22 cache.

Generated by Codex

* vcf_annotate_vepyr: apply test config per test

Load ./nextflow.config only in the tests that rely on its ext.args
(BCFTOOLS_NORM, HUGGINGFACE_DOWNLOAD), following the nf-core testing
spec and the review of nf-core#13071. The no-normalise test runs without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vcf_annotate_vepyr: run UNTAR setup only where the cache tarball is used

Move the UNTAR setup into the three local-cache tests so the Hugging
Face and stub tests no longer fetch and extract cache.tar.gz. Mention
the optional Hugging Face download in the meta.yml summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* bcftools/norm: refuse to write output over the staged input

With no ext.prefix and meta.id equal to the input basename, the default
output name matches the staged input symlink, so bcftools norm rewrites
the source file through it. Add the input/output collision guard used
by bcftools/annotate, csq, filter, convert and pluginfilltags, in script
and stub, with module and vcf_annotate_vepyr regression tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vcf_annotate_vepyr: move the bcftools/norm guard to nf-core#13076

The input/output collision guard is a module fix, so it now lives in its
own PR. Keep the subworkflow regression test, which passes once nf-core#13076
is merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vcf_annotate_vepyr: drop the collision test until nf-core#13076 lands

The test asserts the bcftools/norm guard from nf-core#13076, which is not on
master yet, so it fails in CI. Restore the meta.yml wording that matches
the current module; both come back once nf-core#13076 is merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vcf_annotate_vepyr: rely on the bcftools/norm default prefix from nf-core#13076

bcftools/norm now defaults to ${meta.id}_norm and refuses colliding
prefixes, so drop the ext.prefix workaround from the documented config.
Add a test where the sample ID matches the input basename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vcf_annotate_vepyr: address review of nf-core#13070

- Prefix val_hf_repo with datasets/, so HUGGINGFACE_DOWNLOAD needs no
  required --repo-type in ext.args; only --revision/--include remain.
- Fail when both ch_cache and val_hf_repo are set, or neither, and test
  both failures.
- Test normalisation on input_multiallelic.vcf.gz (986 records, 14
  multiallelic; nf-core/test-datasets#2298): the output has 1,000
  biallelic records and the same variantsMD5 as annotating the
  pre-split input.vcf.gz without normalisation.
- Drop the basename test (covered by the bcftools/norm module tests)
  and use channel instead of Channel.
- meta.yml: reword the bcftools norm configuration, drop the ext.prefix
  paragraph, keep one "default to true", document that ch_cache and
  ch_fasta must be value channels and that BCF or plain VCF output
  makes VEPYR_ANNOTATE run a single pipeline.

Generated by Claude Code

* vcf_annotate_vepyr: build the multiallelic test input in setup

Instead of a new test-datasets file (nf-core/test-datasets#2298), the
normalise tests join input.vcf.gz back into multiallelic records with
an aliased BCFTOOLS_NORM in setup: --multiallelics +both turns 1,000
records into 986, 14 of them multiallelic, and the subworkflow splits
them again. --do-not-normalize is needed because the module always
passes --fasta-ref, and left-aligning fails on chr22 vs 22. Snapshots
are unchanged.

Generated by Claude Code

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants