Promote conventional GGUF MoE architectures - #637
Merged
Conversation
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR promotes three previously-deferred GGUF MoE architectures (bailingmoe, original deepseek, dots1) to exact config extraction, tensor mapping/processing, and graph-build validation in Mobius’ GGUF import pipeline, while explicitly keeping runtime support deferred. It extends the GGUF integration to validate strict tensor closures and routing/metadata contracts for these MoE layouts, and updates DeepSeek routing/model code to support the promoted contracts (including optional correction-bias and pinned normalization-floor behavior).
Changes:
- Add GGUF architecture specs + strict conventional-MoE tensor-closure validation, including fused-QKV alternatives and routing/rope-scaling constraints.
- Extend DeepSeek routing/model logic (bias selection semantics, normalization floor, QMoE encoding selection) and add/expand tests for these behaviors.
- Update registry/config aliases and documentation/support census to reflect the promoted GGUF graph-importable architectures.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/_test_configs.py | Adds small config aliases for deepseek and bailing_moe to exercise graph builds in tests. |
| src/mobius/models/moe.py | Refactors Ernie45MoECausalLMModel init to use the base CausalLMModel initializer and _replace_text_model. |
| src/mobius/models/deepseek.py | Updates DeepSeek MoE gate routing contracts (bias semantics, normalization floor, QMoE router_probs selection) and preprocessing rename behavior. |
| src/mobius/models/deepseek_test.py | Adds/extends unit tests for routing defaults, bias optionality, normalization floor behavior, and GGUF rename behavior. |
| src/mobius/integrations/gguf/_tensor_processors.py | Adds fused qkv_proj splitting before applying optional Llama Q/K inverse permutation. |
| src/mobius/integrations/gguf/_tensor_processors_test.py | Adds tests ensuring fused-QKV split behavior matches separate-projection transforms and preserves Dots1 ordering. |
| src/mobius/integrations/gguf/_tensor_mapping.py | Adds DeepSeek shared-MoE “extras” tensor mappings (router + experts + shared experts + correction bias). |
| src/mobius/integrations/gguf/_conventional_moe_test.py | New focused test suite validating exact tensor closure + metadata constraints for promoted conventional MoE architectures. |
| src/mobius/integrations/gguf/_config_mapping.py | Adds conventional shared-MoE key/post-processing, including strict YaRN-only scaling validation and pinned routing defaults. |
| src/mobius/integrations/gguf/_config_mapping_test.py | Adds extensive tests for conventional shared-MoE config extraction/validation and dense-prefix edge cases. |
| src/mobius/integrations/gguf/_builder.py | Enforces conventional-MoE tensor contract in GGUF validation and applies Q/K permute to split fused quantized projections (weight/scales/zero_points). |
| src/mobius/integrations/gguf/_builder_test.py | Extends GGUF build tests to cover promoted MoE architectures, fused biased QKV, and deepseek tied-quantized head behavior. |
| src/mobius/integrations/gguf/_arch_registry.py | Promotes bailingmoe, deepseek, dots1 to supported (graph) with deferred runtime, including recipes and processors. |
| src/mobius/integrations/gguf/_arch_registry_test.py | Updates expected supported-architecture counts and closure sets for the promotions. |
| src/mobius/_registry.py | Registers fallback deepseek and bailing_moe model_type mappings for graph construction. |
| src/mobius/_configs/_base.py | Adds new config fields (routing_weight_normalization_floor, use_expert_bias) and updates Lfm2MoeConfig typing for the new semantics. |
| docs/api/build_from_gguf.md | Updates census/support matrix entries for bailingmoe, deepseek, and dots1 to reflect supported graph import and deferred runtime. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+161
to
+180
| # Some pinned loaders accept a fused QKV tensor as an alternative to the | ||
| # split projections. Split it before applying the same inverse Q/K RoPE | ||
| # permutation used by the separate layout. | ||
| fused_qkv = [(name, tensor) for name, tensor in state_dict.items() if ".qkv_proj." in name] | ||
| if fused_qkv: | ||
| head_dim = int(config.head_dim) | ||
| q_width = int(num_heads) * head_dim | ||
| kv_width = int(num_kv_heads) * head_dim | ||
| for name, tensor in fused_qkv: | ||
| if tensor.shape[0] != q_width + 2 * kv_width: | ||
| raise ValueError( | ||
| f"Invalid fused QKV width for {name}: expected " | ||
| f"{q_width + 2 * kv_width}, got {tensor.shape[0]}" | ||
| ) | ||
| query, key, value = tensor.split([q_width, kv_width, kv_width], dim=0) | ||
| prefix, suffix = name.rsplit(".qkv_proj.", 1) | ||
| state_dict[f"{prefix}.q_proj.{suffix}"] = query | ||
| state_dict[f"{prefix}.k_proj.{suffix}"] = key | ||
| state_dict[f"{prefix}.v_proj.{suffix}"] = value | ||
| del state_dict[name] |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| fused_weight = torch.from_numpy( | ||
| np.array(source.get_tensor("blk.0.attn_qkv.weight")) | ||
| ) | ||
| fused_scales = fused_zero_points = None |
Add exact config, tensor, routing, shared-expert, fused-QKV, and quantized import contracts for BailingMoE, original DeepSeek-MoE, and Dots1. Keep runtime support deferred pending immutable real-artifact parity evidence and fail closed on contradictory schedules or mixed optional tensors. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Preserve existing HF DeepSeek correction-bias defaults, reuse tied quantized head storage, attach fused-QKV handling only to the promoted architectures, and reject correction tensors outside Dots1. Expand fused packed-QKV and tied-head coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Keep the existing LFM2MoE correction-bias default while allowing base DeepSeek configurations to infer their historical routing behavior, and regenerate the support matrix after the live-main rebase. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Apply llama Q/K row permutation after splitting packed fused projections, including affine scales and zero points. Feed raw logits to CUDA QMoE for the exact ungrouped DeepSeek softmax contract and fail closed for unsupported routing variants. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Account for the multimodal GGUF promotion already present on the exact final base and regenerate the support matrix without dropping its Qwen2-VL entries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Use raw logits for the exact promoted DeepSeek CUDA route while retaining the existing activated-score encoding required by grouped GLM-MoE-DSA CPU graphs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Reject missing, unexpected, malformed, out-of-range, mixed-schedule, and incomplete attention or expert tensor families before config extraction. Cover fused and split QKV layouts, shared experts, Dots1 sidecars, and tied DeepSeek output. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Require Dots1 full-MHA and full-head RoPE geometry while allowing the pinned DeepSeek and Dots1 loaders' all-dense boundary schedule. Add positive and negative boundary coverage and use authoritative Dots1 cache geometry in synthetic builds. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Apply the repository's initialized formatter to the final Dots1 geometry guard and closure regression tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Reject non-finite or non-positive RMS epsilon and effective routing scales while preserving llama.cpp's zero scale sentinel. Accept authoritative single-expert routed and all-dense schedules with explicit boundary coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Allow exact all-dense DeepSeek and Dots1 schedules while keeping routed metadata fail-closed. Enforce authoritative attention and RoPE contracts, including pinned YaRN defaults, and preserve Dots1's Qwen2 Q/K tensor ordering across F32 and Q4 packed routes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Preserve the pinned llama.cpp denominator clamp for normalized Dots1 expert weights while leaving native DeepSeek routing unchanged. Add a low-score ONNX Runtime parity regression for the boundary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-deferred-gguf-moe
branch
from
August 26, 2026 01:55
a66de72 to
6df92b6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bailingmoe, originaldeepseek, anddots1from deferred to exact GGUF config/tensor-map/graph support while keeping runtime support deferred2^-14normalization floor, and route scalingExact base and head
c8803afb8251136e850010aba2cc4b8bf9854082(includes Align inference metadata batching contract #636, Fix accumulated model contract CI regressions #638, and Fix Qwen3.5 standalone ORT GenAI model types #631)6df92b6c80e250fe45d1a6378426fb55220cc045a66de72595afdfdb9d94da9fad0b361ffd1803cfEvidence
1899 passed, 1 skipped204 passed, 8 skipped, 1374 deselected8120 passed, 57 skipped, 12 deselected, 1 subtest passed, explicitPYTEST_EXIT=0An intermittent libc++
recursive_mutexteardown diagnostic was previously reproduced after an assertion-clean parallel run. It did not reproduce in the serial GGUF builder probe or final serial broad suites; clean serial process exits are the publication evidence.Runtime waiver
Runtime remains
DEFERREDfor all three promotions. No practical immutable small real-weight GGUF artifact was available to establish full CLI/ORT GenAI generation parity without downloading production-scale checkpoints. Graph execution, synthetic numerical routing parity, packed tensor value checks, and ORT execution are covered; runtime metadata does not claim support from architecture-name matching.Residual deferred MoE
Different whole-model contracts remain separate batches:
ernie4_5-moe,hunyuan-moe,minimax-m2,mellum: unmatched per-layer schedules or Q/K norm/RoPE orderingarctic,dbrx,grok,smallthinker: incompatible residual topology, norm order/type, fused/clamped attention, scaling/softcaps, router input, expert activation, or per-layer RoPEafmoe,laguna,granite_swa, conditionalgranite/minicpm: unmatched gates, SWA/cache contracts, architecture scales, or value transformsgrovemoe: grouped dual-bank routing is unsupportedgpt-oss: value-changing MXFP4 packed expert conversion is unsupportedminimax-m3: second sparse-index cache ABI is unsupportedbailingmoe3,deepseek4, Kimi/DeltaNet/Mamba hybrids: heterogeneous recurrent/compressed state ABI