Skip to content

Fix accumulated model contract CI regressions - #638

Merged
titaiwangms merged 2 commits into
mainfrom
fix/model-contract-ci-regressions
Aug 26, 2026
Merged

Fix accumulated model contract CI regressions#638
titaiwangms merged 2 commits into
mainfrom
fix/model-contract-ci-regressions

Conversation

@titaiwangms

Copy link
Copy Markdown
Contributor

Summary

  • align Jais2, CodeShell, Xverse, and Kimi-K3 model config_class declarations with registry dispatch
  • record Kimi-K3 and LFM2-MoE coverage limitations without fabricating golden data
  • support both tensor and mapping Transformers linear-attention cache ABIs in Falcon-H1 and GatedDeltaNet parity tests
  • mark the selective-MXFP4 Kimi-K3 checkpoint as a strict graph-only L2 xfail
  • preserve independent dashboard confidence levels when L2 xfails but L3 passes

Root causes

Validation

  • dashboard plus original CI regression gates: 90 passed
  • affected graph and weight-alignment gates: 25 passed
  • Kimi-K3 L2: 1 passed, 2 strict xfailed

Align specialized decoder model classes with their registered configs, record unsupported golden and L2 coverage honestly, and support both Transformers linear-attention cache layouts in parity tests. Correct the dashboard meta-invariant so an L2 xfail does not erase independent higher-level confidence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2fe48e9f-4ebc-43a9-85f3-4e4f99f3ed5d
Signed-off-by: titaiwang <titaiwang@microsoft.com>
@titaiwangms
titaiwangms requested review from a team and a lite review from Copilot August 26, 2026 00:40
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown

🏗️ Architecture Diff

Comparing 8e3ab92a7bdb08

Model Sub-model Changes Status

No architecture changes detected.


Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed)

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown

Performance Comparison

Comparing 8e3ab92a7bdb08

Model Metric Baseline Current Delta
bert (feature-extraction) model_size_bytes 359 KB 359 KB +0.0%
bert (feature-extraction) num_nodes 68 68 +0.0%
falcon model_size_bytes 364 KB 364 KB +0.0%
falcon num_nodes 66 66 +0.0%
gemma2 model_size_bytes 428 KB 428 KB +0.0%
gemma2 num_nodes 105 105 +0.0%
gpt2 model_size_bytes 324 KB 324 KB +0.0%
gpt2 num_nodes 54 54 +0.0%
llama model_size_bytes 425 KB 425 KB +0.0%
llama num_nodes 60 60 +0.0%
llama (static-cache) model_size_bytes 425 KB 425 KB +0.0%
llama (static-cache) num_nodes 56 56 +0.0%
mamba (ssm-text-generation) model_size_bytes 296 KB 296 KB +0.0%
mamba (ssm-text-generation) num_nodes 94 94 +0.0%
phi3 model_size_bytes 421 KB 421 KB +0.0%
phi3 num_nodes 58 58 +0.0%
phi3 (static-cache) model_size_bytes 421 KB 421 KB +0.0%
phi3 (static-cache) num_nodes 54 54 +0.0%
qwen2 model_size_bytes 425 KB 425 KB +0.0%
qwen2 num_nodes 60 60 +0.0%
qwen2 (static-cache) model_size_bytes 425 KB 425 KB +0.0%
qwen2 (static-cache) num_nodes 56 56 +0.0%
qwen3_5_moe (hybrid-text-generation) model_size_bytes 506 KB 506 KB +0.0%
qwen3_5_moe (hybrid-text-generation) num_nodes 265 265 +0.0%
qwen3_5_text (hybrid-text-generation) model_size_bytes 458 KB 458 KB +0.0%
qwen3_5_text (hybrid-text-generation) num_nodes 127 127 +0.0%
qwen3_5_vl (hybrid-qwen-vl) model_size_bytes 977 KB 977 KB +0.0%
qwen3_5_vl (hybrid-qwen-vl) num_nodes 429 429 +0.0%
t5 (seq2seq) model_size_bytes 836 KB 836 KB +0.0%
t5 (seq2seq) num_nodes 176 176 +0.0%
whisper (speech-to-text) model_size_bytes 1008 KB 1008 KB +0.0%
whisper (speech-to-text) num_nodes 128 128 +0.0%

No performance regressions.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes several model-contract and coverage-accounting mismatches that were causing CI regressions across registry dispatch, model config_class declarations, parity tests, and dashboard confidence-level invariants.

Changes:

  • Aligns config_class declarations and registry registrations for Jais2, CodeShell, Xverse, and Kimi-K3 to match dispatch expectations (and adds targeted unit tests to lock this in).
  • Updates Falcon-H1 and GatedDeltaNet parity tests to accept both the older “tensor” and newer “mapping keyed by state index” Transformers cache ABIs.
  • Records coverage/xfail limitations for Kimi-K3 and LFM2-MoE without overstating L4/L5 evidence, and adjusts dashboard meta-assertions to keep confidence levels independent when L2 xfails but higher levels pass.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/parity/gated_deltanet_integration_test.py Adds cache-ABI-compatible extraction helper for recurrent state in parity coverage.
tests/falcon_h1_test.py Adds cache-ABI-compatible extraction helper for Falcon-H1 convolution/recurrent states in parity coverage.
tests/model_coverage_test.py Adds/updates skip reasons documenting Kimi-K3 and LFM2-MoE coverage constraints.
tests/arch_validation_test.py Documents Kimi-K3 validation limitation due to unsupported selective MXFP4 experts.
tests/generate_dashboard_test.py Updates dashboard meta-assertion to avoid forcing confidence below L2 when L2 is an xfail.
src/mobius/models/legacy_decoder.py Introduces specialized model subclasses declaring config_class for Jais2/CodeShell/Xverse.
src/mobius/models/legacy_decoder_test.py Adds assertions that specialized models declare the correct config_class; updates instantiations to use the specialized classes.
src/mobius/models/kimi_k3.py Declares config_class on KimiK3CausalLMModel to match registry dispatch expectations.
src/mobius/models/kimi_k3_test.py Adds a regression test ensuring Kimi-K3 declares its config_class.
src/mobius/models/init.py Exports newly introduced specialized legacy-decoder model classes.
src/mobius/_registry.py Switches registrations to the specialized model classes to keep registry/model contracts aligned.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Apply the repository formatter to the dashboard L2 xfail meta-test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2fe48e9f-4ebc-43a9-85f3-4e4f99f3ed5d
Signed-off-by: titaiwang <titaiwang@microsoft.com>
@titaiwangms
titaiwangms enabled auto-merge (squash) August 26, 2026 00:51
@titaiwangms
titaiwangms merged commit 133ab10 into main Aug 26, 2026
21 of 25 checks passed
@titaiwangms
titaiwangms deleted the fix/model-contract-ci-regressions branch August 26, 2026 00:55
justinchuby added a commit that referenced this pull request Aug 26, 2026
## Summary

- promote `bailingmoe`, original `deepseek`, and `dots1` from deferred
to exact GGUF config/tensor-map/graph support while keeping runtime
support deferred
- preserve authoritative routing contracts: Bailing full-softmax top-k;
DeepSeek unnormalized softmax top-k; Dots1 metadata-selected
softmax/sigmoid, selection-only correction bias, pinned `2^-14`
normalization floor, and route scaling
- validate exact dense-prefix/routed/shared-expert schedules, including
all-dense DeepSeek/Dots1 files with absent or zero inactive expert
metadata
- enforce full-head RoPE for all three, full MHA for Dots1, complete
supported YaRN metadata, pinned beta/attention defaults, and fail-closed
rejection of unsupported scaling
- close tensor families and shapes, packed fused-QKV splits, bias
alternatives, shared experts, output ownership, auxiliary quantization
sidecars, and out-of-range tensors
- preserve llama.cpp Q/K inverse permutation for Bailing/DeepSeek while
retaining Dots1's Qwen2 checkpoint row order across F32 and Q4 packed
weights/scales/zero-points
- preserve existing PhiMoE output-head, GLM-MoE-DSA CPU/grouped-routing,
LFM2MoE correction-bias, dense/multimodal/specialized/runtime/package,
inference-metadata, and Qwen3.5 ORT GenAI contracts

## Exact base and head

- base: `c8803afb8251136e850010aba2cc4b8bf9854082` (includes #636, #638,
and #631)
- head: `6df92b6c80e250fe45d1a6378426fb55220cc045`
- the isolated 12-commit MoE series was replayed conflict-free onto this
exact live-main SHA and force-updated with a lease against old remote
head `a66de72595afdfdb9d94da9fad0b361ffd1803cf`

## Evidence

- focused registry/config/tensor/processor/closure/builder/docs/model
suite: `1899 passed, 1 skipped`
- shared MoE/build regression suite: `204 passed, 8 skipped, 1374
deselected`
- broad serial non-integration: `8120 passed, 57 skipped, 12 deselected,
1 subtest passed`, explicit `PYTEST_EXIT=0`
- promoted packed fused-QKV value/runtime coverage includes F32 and Q4_0
weights, scales, zero-points, biases, graph construction, and ORT
execution
- initialized all-files lint: clean
- generated GGUF documentation closure: clean
- final GPT-5.6 Sol medium architecture review: approved, no findings
- final GPT-5.6 Sol medium regression review: approved, no findings

An intermittent libc++ `recursive_mutex` teardown diagnostic was
previously reproduced after an assertion-clean parallel run. It did not
reproduce in the serial GGUF builder probe or final serial broad suites;
clean serial process exits are the publication evidence.

## Runtime waiver

Runtime remains `DEFERRED` for all three promotions. No practical
immutable small real-weight GGUF artifact was available to establish
full CLI/ORT GenAI generation parity without downloading
production-scale checkpoints. Graph execution, synthetic numerical
routing parity, packed tensor value checks, and ORT execution are
covered; runtime metadata does not claim support from architecture-name
matching.

## Residual deferred MoE

Different whole-model contracts remain separate batches:

- `ernie4_5-moe`, `hunyuan-moe`, `minimax-m2`, `mellum`: unmatched
per-layer schedules or Q/K norm/RoPE ordering
- `arctic`, `dbrx`, `grok`, `smallthinker`: incompatible residual
topology, norm order/type, fused/clamped attention, scaling/softcaps,
router input, expert activation, or per-layer RoPE
- `afmoe`, `laguna`, `granite_swa`, conditional `granite`/`minicpm`:
unmatched gates, SWA/cache contracts, architecture scales, or value
transforms
- `grovemoe`: grouped dual-bank routing is unsupported
- `gpt-oss`: value-changing MXFP4 packed expert conversion is
unsupported
- `minimax-m3`: second sparse-index cache ABI is unsupported
- `bailingmoe3`, `deepseek4`, Kimi/DeltaNet/Mamba hybrids: heterogeneous
recurrent/compressed state ABI
- DeepSeek NextN/MTP families: executable MTP/cache ABI is owned by the
separate deferred MTP batch
- multimodal MoE backbones remain excluded from this text-only cohort

---------

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants