Add VibeVoice streaming ASR export - #723
Merged
Merged
Conversation
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
justinchuby
force-pushed
the
justinchuby-add-vibevoice-streaming-asr
branch
2 times, most recently
from
September 3, 2026 22:57
87138b0 to
e6c4950
Compare
justinchuby
force-pushed
the
justinchuby-add-vibevoice-streaming-asr
branch
from
September 4, 2026 16:00
e6c4950 to
bd6a812
Compare
justinchuby
force-pushed
the
justinchuby-add-vibevoice-streaming-asr
branch
from
September 4, 2026 16:00
e6c4950 to
bd6a812
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A few robustness/compatibility issues in the new exception handling import path and pinned-source verification logic should be fixed to avoid avoidable CI/runtime failures.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds first-class support for exporting VibeVoice streaming ASR (microsoft/VibeVoice-ASR-Streaming-7B) as a three-stage ONNX pipeline (audio encoder + embedding + decoder), integrated into the existing Mobius model registry/build flow and onnx-genai export tooling.
Changes:
- Introduces
VibeVoiceASRConfig,VibeVoiceASRForConditionalGeneration, andVibeVoiceASRStreamingTaskto export the staged streaming-ASR pipeline with explicit KV + convolution cache I/O. - Extends Transformers config/module resolution to discriminate VibeVoice TTS vs ASR by
architectures, and updates model/test coverage registration for the new architecture key. - Adds targeted tests (including pinned source-parity CI workflow) and onnx-genai advisory export support for the ASR topology.
File summaries
| File | Description |
|---|---|
| tests/weight_alignment_test.py | Adds ASR checkpoint-index routing audit and marks ASR architecture as filtering preprocess. |
| tests/model_coverage_test.py | Documents why generic L4/L5 runner can’t drive this streaming ASR pipeline yet. |
| tests/build_graph/speech_test.py | Declares expected staged outputs for vibevoice-asr-streaming. |
| tests/arch_validation_test.py | Extends HF config loading/unwrapping to handle ASR’s nested decoder config. |
| tests/_test_configs.py | Adds tiny pinned config entry for VibeVoiceForASRStreamingTraining. |
| src/mobius/tasks/_vibevoice.py | Refactors convolution-cache helpers into shared utility module. |
| src/mobius/tasks/_vibevoice_asr.py | New staged task defining ASR graph contracts and explicit cache I/O. |
| src/mobius/tasks/_streaming_convolution.py | New helper module for causal convolution cache inputs/outputs. |
| src/mobius/tasks/init.py | Registers vibevoice-asr-streaming task and exports it. |
| src/mobius/models/vibevoice.py | Adds ASR provenance constants, final-chunk stride padding, and ASR model modules + weight routing. |
| src/mobius/models/vibevoice_asr_test.py | New comprehensive contract/routing and pinned source-parity tests. |
| src/mobius/models/init.py | Exports VibeVoiceASRForConditionalGeneration. |
| src/mobius/integrations/transformers/_builder.py | Adds architecture-discriminated VibeVoice dispatch and pins ASR revision. |
| src/mobius/integrations/onnx_genai/auto_export.py | Detects ASR 3-stage topology and emits advisory metadata + assets. |
| src/mobius/integrations/onnx_genai/auto_export_test.py | Adds coverage for ASR advisory export path and revision forwarding. |
| src/mobius/_registry.py | Registers VibeVoiceForASRStreamingTraining with pinned model+revision. |
| src/mobius/_configs/vibevoice.py | Expands config extraction to include ASR tokenizer constraints + VibeVoiceASRConfig. |
| src/mobius/_configs/init.py | Exports VibeVoiceASRConfig. |
| requirements/ci/vibevoice-asr.txt | Pins the exact upstream VibeVoice ASR source for synthetic parity tests. |
| docs/model-catalog.md | Lists the new ASR architecture in the catalog. |
| .github/workflows/main.yml | Adds dedicated CI job to run ASR pinned source-parity integration tests. |
Review details
- Files reviewed: 21/21 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
It introduces a new multi-stage, stateful streaming ASR export surface (new config/model/task/CI contract) that warrants final human review despite strong test coverage.
Review details
- Files reviewed: 22/22 changed files
- Comments generated: 0 new
- Review effort level: Lite
rui-ren
approved these changes
Sep 14, 2026
Contributor
|
@copilot resolve the merge conflicts in this pull request |
1 similar comment
Contributor
|
@copilot resolve the merge conflicts in this pull request |
Contributor
Merged
Non-integration suite after the merge: |
|
|
Contributor
Already done — no conflicts remain. The merge commit Details of that merge are in my previous comment: If GitHub is still showing this PR as conflicted, try refreshing the page — the mergeability status can lag behind a push. |
Implement an architecture-discriminated, staged ONNX export for the pinned VibeVoice ASR checkpoint, with explicit audio convolution and decoder cache contracts. Preserve the existing TTS path while recording unsupported ORT GenAI orchestration boundaries and validating source parity, processor behavior, and complete checkpoint routing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Apply the repository import and expression formatting rules after restacking the streaming ASR export on the finalized VibeVoice TTS base. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Record the streaming ASR L4/L5 runner limitation and mark its deliberate checkpoint-name translation as incompatible with the generic identity fixture. The pinned source-index audit remains the exact weight-routing proof. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Install the pinned VibeVoice executable source and its compatible Transformers release in a dedicated L3 job. Verify the installed source commit through pip direct-url metadata so source parity cannot silently skip outside a checkout. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Generalize the pinned ASR checkpoint set, validate both official variants, and preserve explicit tied LM-head tensors during checkpoint routing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4ef52c4-0459-4efa-bc06-d0ebca2275f2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Force-install the Transformers 4.51 reference dependency versions that the source-parity job requires, preventing the public Transformers 5 environment from supplying incompatible Hugging Face Hub and tokenizers releases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4ef52c4-0459-4efa-bc06-d0ebca2275f2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Make config fallback compatible with older Hub releases and validate the pinned reference environment before importing its modules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4ef52c4-0459-4efa-bc06-d0ebca2275f2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Use the composite ASR configuration dtype for the tokenizer and connector stages instead of the nested Qwen checkpoint storage dtype. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4ef52c4-0459-4efa-bc06-d0ebca2275f2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Restore decoder cache imports and declare streaming convolution cache output types and shapes after resolving the native and streaming ASR integration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-add-vibevoice-streaming-asr
branch
from
September 16, 2026 23:12
322427e to
ae7c6a0
Compare
The streaming ASR export remains covered by the standard graph, configuration, weight-routing, and runtime-contract tests without a dedicated external-source dependency environment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
titaiwangms
added a commit
that referenced
this pull request
Sep 17, 2026
## Summary - integrate both supported VibeVoice streaming ASR checkpoints into the shared fail-closed source resolver - pin config, processor, and weight sources to each checkpoint's verified revision - preserve module-specific weight preprocessing: only TTS layouts receive the `checkpoint_layout` argument - retain fail-closed rejection for unsupported BitNet and standalone acoustic-tokenizer checkpoints ## Root cause #723 was tested before #726 landed, although both changed VibeVoice source resolution. The combined `main` state classified the newly supported streaming ASR IDs as unsupported before their architecture-specific exporter could load the pinned config. The fix does not bypass or weaken #726's resolver. It makes streaming ASR an explicit supported provenance route and removes the now-redundant exact-ID pinning branch from the Transformers builder. ## Validation - `python -m pytest src/mobius/integrations/transformers/_builder_test.py src/mobius/models/vibevoice_test.py -q --tb=short` (`60 passed, 1 skipped`) - builder-level `load_weights=True` regression coverage verifies pinned config/weight revisions, the native ASR preprocessing signature, and emitted source metadata for both variants - targeted lintrunner checks: clean - revised lightweight correctness and risk reviews: PASS --------- Signed-off-by: titaiwang <titaiwang@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation
VibeVoiceForASRStreamingTrainingsupport for both official Microsoft streaming-ASR checkpoints, without changing the existing VibeVoice TTS route:microsoft/VibeVoice-ASR-Streaming-1.5B@4262d23d8a539a6530cf64fbd0b1751ef9a30853microsoft/VibeVoice-ASR-Streaming-7B@60d858b518b4e19d404af3737f848fc185b30177audio_encoder(acoustic and semantic causal tokenizers plus connectors),embedding(speech embedding replacement), anddecoder(Qwen2 with explicit KV caches).<speech_start>,<speech_end>, forced<|text_chunk_end|>), chunk loop/sampling, hotword prompt assembly, and speaker JSON parsing remain host-owned.mobius.attention.requires_arbitrary_maskcontract, preventing incorrect GQA fusion for left-padded ASR batches.acoustic_tokenizer.decoder.*training-VAE tensors that are not on the ASR inference path. The 1.5B tied embedding/head checkpoint preserves its explicitlm_head.weight.Provenance and runtime boundary
microsoft/VibeVoice@505653d3873b065a488aea551c6ee3dc51d3062fwithtransformers==4.51.3; ASR-critical blobs were checked unchanged through upstream1541f590c7099820f10ea012f48d2399282df69f.huggingface-hub==0.31.4,tokenizers==0.21.4) after the public Transformers 5 environment.onnxruntime-ep-mlx 0.29.5/ ORT 1.29 cannot execute the complete VibeVoice decoder due toCumSum cannot infer output shapes; this blocks local MLX real-runtime evidence.Validation
4730 passed, 56 skipped, 1 subtests passed.lintrunner f --output oneline --all-filesand diff checks.