Fix accumulated model contract CI regressions - #638
Merged
Conversation
Align specialized decoder model classes with their registered configs, record unsupported golden and L2 coverage honestly, and support both Transformers linear-attention cache layouts in parity tests. Correct the dashboard meta-invariant so an L2 xfail does not erase independent higher-level confidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2fe48e9f-4ebc-43a9-85f3-4e4f99f3ed5d Signed-off-by: titaiwang <titaiwang@microsoft.com>
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes several model-contract and coverage-accounting mismatches that were causing CI regressions across registry dispatch, model config_class declarations, parity tests, and dashboard confidence-level invariants.
Changes:
- Aligns
config_classdeclarations and registry registrations for Jais2, CodeShell, Xverse, and Kimi-K3 to match dispatch expectations (and adds targeted unit tests to lock this in). - Updates Falcon-H1 and GatedDeltaNet parity tests to accept both the older “tensor” and newer “mapping keyed by state index” Transformers cache ABIs.
- Records coverage/xfail limitations for Kimi-K3 and LFM2-MoE without overstating L4/L5 evidence, and adjusts dashboard meta-assertions to keep confidence levels independent when L2 xfails but higher levels pass.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/parity/gated_deltanet_integration_test.py | Adds cache-ABI-compatible extraction helper for recurrent state in parity coverage. |
| tests/falcon_h1_test.py | Adds cache-ABI-compatible extraction helper for Falcon-H1 convolution/recurrent states in parity coverage. |
| tests/model_coverage_test.py | Adds/updates skip reasons documenting Kimi-K3 and LFM2-MoE coverage constraints. |
| tests/arch_validation_test.py | Documents Kimi-K3 validation limitation due to unsupported selective MXFP4 experts. |
| tests/generate_dashboard_test.py | Updates dashboard meta-assertion to avoid forcing confidence below L2 when L2 is an xfail. |
| src/mobius/models/legacy_decoder.py | Introduces specialized model subclasses declaring config_class for Jais2/CodeShell/Xverse. |
| src/mobius/models/legacy_decoder_test.py | Adds assertions that specialized models declare the correct config_class; updates instantiations to use the specialized classes. |
| src/mobius/models/kimi_k3.py | Declares config_class on KimiK3CausalLMModel to match registry dispatch expectations. |
| src/mobius/models/kimi_k3_test.py | Adds a regression test ensuring Kimi-K3 declares its config_class. |
| src/mobius/models/init.py | Exports newly introduced specialized legacy-decoder model classes. |
| src/mobius/_registry.py | Switches registrations to the specialized model classes to keep registry/model contracts aligned. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Apply the repository formatter to the dashboard L2 xfail meta-test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2fe48e9f-4ebc-43a9-85f3-4e4f99f3ed5d Signed-off-by: titaiwang <titaiwang@microsoft.com>
justinchuby
approved these changes
Aug 26, 2026
titaiwangms
enabled auto-merge (squash)
August 26, 2026 00:51
justinchuby
added a commit
that referenced
this pull request
Aug 26, 2026
## Summary - promote `bailingmoe`, original `deepseek`, and `dots1` from deferred to exact GGUF config/tensor-map/graph support while keeping runtime support deferred - preserve authoritative routing contracts: Bailing full-softmax top-k; DeepSeek unnormalized softmax top-k; Dots1 metadata-selected softmax/sigmoid, selection-only correction bias, pinned `2^-14` normalization floor, and route scaling - validate exact dense-prefix/routed/shared-expert schedules, including all-dense DeepSeek/Dots1 files with absent or zero inactive expert metadata - enforce full-head RoPE for all three, full MHA for Dots1, complete supported YaRN metadata, pinned beta/attention defaults, and fail-closed rejection of unsupported scaling - close tensor families and shapes, packed fused-QKV splits, bias alternatives, shared experts, output ownership, auxiliary quantization sidecars, and out-of-range tensors - preserve llama.cpp Q/K inverse permutation for Bailing/DeepSeek while retaining Dots1's Qwen2 checkpoint row order across F32 and Q4 packed weights/scales/zero-points - preserve existing PhiMoE output-head, GLM-MoE-DSA CPU/grouped-routing, LFM2MoE correction-bias, dense/multimodal/specialized/runtime/package, inference-metadata, and Qwen3.5 ORT GenAI contracts ## Exact base and head - base: `c8803afb8251136e850010aba2cc4b8bf9854082` (includes #636, #638, and #631) - head: `6df92b6c80e250fe45d1a6378426fb55220cc045` - the isolated 12-commit MoE series was replayed conflict-free onto this exact live-main SHA and force-updated with a lease against old remote head `a66de72595afdfdb9d94da9fad0b361ffd1803cf` ## Evidence - focused registry/config/tensor/processor/closure/builder/docs/model suite: `1899 passed, 1 skipped` - shared MoE/build regression suite: `204 passed, 8 skipped, 1374 deselected` - broad serial non-integration: `8120 passed, 57 skipped, 12 deselected, 1 subtest passed`, explicit `PYTEST_EXIT=0` - promoted packed fused-QKV value/runtime coverage includes F32 and Q4_0 weights, scales, zero-points, biases, graph construction, and ORT execution - initialized all-files lint: clean - generated GGUF documentation closure: clean - final GPT-5.6 Sol medium architecture review: approved, no findings - final GPT-5.6 Sol medium regression review: approved, no findings An intermittent libc++ `recursive_mutex` teardown diagnostic was previously reproduced after an assertion-clean parallel run. It did not reproduce in the serial GGUF builder probe or final serial broad suites; clean serial process exits are the publication evidence. ## Runtime waiver Runtime remains `DEFERRED` for all three promotions. No practical immutable small real-weight GGUF artifact was available to establish full CLI/ORT GenAI generation parity without downloading production-scale checkpoints. Graph execution, synthetic numerical routing parity, packed tensor value checks, and ORT execution are covered; runtime metadata does not claim support from architecture-name matching. ## Residual deferred MoE Different whole-model contracts remain separate batches: - `ernie4_5-moe`, `hunyuan-moe`, `minimax-m2`, `mellum`: unmatched per-layer schedules or Q/K norm/RoPE ordering - `arctic`, `dbrx`, `grok`, `smallthinker`: incompatible residual topology, norm order/type, fused/clamped attention, scaling/softcaps, router input, expert activation, or per-layer RoPE - `afmoe`, `laguna`, `granite_swa`, conditional `granite`/`minicpm`: unmatched gates, SWA/cache contracts, architecture scales, or value transforms - `grovemoe`: grouped dual-bank routing is unsupported - `gpt-oss`: value-changing MXFP4 packed expert conversion is unsupported - `minimax-m3`: second sparse-index cache ABI is unsupported - `bailingmoe3`, `deepseek4`, Kimi/DeltaNet/Mamba hybrids: heterogeneous recurrent/compressed state ABI - DeepSeek NextN/MTP families: executable MTP/cache ABI is owned by the separate deferred MTP batch - multimodal MoE backbones remain excluded from this text-only cohort --------- Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
config_classdeclarations with registry dispatchRoot causes
Validation