Promote conventional GGUF MoE architectures

docs/api/build_from_gguf.md

@@ -15,7 +15,7 @@ from mobius import build_from_gguf
 
 | Census | Total | Closure |
 |---|---:|---|
-| Architectures | 147 | graph verdicts: {'deferred': 73, 'rejected': 2, 'supported': 72}; importable: 71; quantized import: {'rejected': 20, 'supported': 127}; runtime: {'deferred': 144, 'rejected': 2, 'supported': 1} |
+| Architectures | 147 | graph verdicts: {'deferred': 70, 'rejected': 2, 'supported': 75}; importable: 74; quantized import: {'rejected': 20, 'supported': 127}; runtime: {'deferred': 144, 'rejected': 2, 'supported': 1} |
 | Active stored qtypes | 25 | 24 have an import route; 1 are explicitly deferred with no route |
 | Serialized projector strings | 60 | {'graph-importable': 5, 'runtime-supported': 0} |
 | Tokenizer pre identifiers | 87 | 56 semantic groups; all default to deferred and become materializable only from a validated embedded `tokenizer.huggingface.json` or an exact pinned source in runtime evidence |
@@ -349,7 +349,7 @@ before graph construction or durable output.
 | `arctic` | — | none (fails before config extraction) | not claimed | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | The pinned Arctic graph is not a standard pre-norm MoE block: every layer runs a dense parallel SwiGLU branch, then adds a separately normalized routed-expert branch computed from the pre-attention residual. Mobius's generic MoE graph replaces the dense FFN instead, so aliasing the existing Hugging Face 'arctic' registration would change residual topology and normalization. |
 | `arwkv7` | — | none (fails before config extraction) | audited-direct-loader-conditional-union | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | ARWKV7 wraps RWKV7's delta-rule matrix recurrence in a distinct one-shift RMSNorm/Qwen residual topology with optional five-versus-six-way interpolation, optional gate/group norm, and Qwen SwiGLU. Treating it as RWKV7, Qwen, or Mamba would accept the wrong tensor closure and state ABI. |
 | `baichuan` | — | model=`baichuan`; tensor=`llama` | not claimed | config=supported; tensor_map=supported; graph=supported; runtime=deferred; quantized_import=supported | Config extraction, exact tensor-name closure, and a full synthetic GGUF graph build are covered, but no representative real-weight GGUF has yet passed ORT parity or generation validation. Runtime packaging remains deferred until that evidence exists. |
-| `bailingmoe` | — | none (fails before config extraction) | audited-direct-loader-conditional-union | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | BailingMoE requires an untied head, split-and-permuted Q/K/V conversion, and an all-layer softmax routed MoE with mandatory shared experts. Mobius has no exact expert-major tensor transform, preservation contract, or parity evidence. |
+| `bailingmoe` | — | model=`bailing_moe`; tensor=`llama`+`diffusion_fused_qkv`+`moe_extras` | audited-direct-loader-conditional-union | config=supported; tensor_map=supported; graph=supported; runtime=deferred; quantized_import=supported | Config extraction, exact tensor-name closure, and a full synthetic GGUF graph build are covered, but no representative real-weight GGUF has yet passed ORT parity or generation validation. Runtime packaging remains deferred until that evidence exists. |
 | `bailingmoe2` | — | none (fails before config extraction) | audited-direct-loader-conditional-union | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | BailingMoE2 serializes complete dense-or-routed/shared expert trailing blocks plus NextN and layer-output norms, but the pinned loader marks every trailing tensor skipped and exposes no MTP graph. Mobius cannot invent executable head semantics from preserved storage. |
 | `bailingmoe3` | — | none (fails before config extraction) | not claimed | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | BailingMoE3 alternates head-wise KDA recurrent layers with gated MLA layers, so each sequence carries three causal-convolution histories plus a matrix state alongside attention cache. Its routed sigmoid/correction-bias experts, always-on shared experts, and optional single NextN block also require tensor and task contracts Mobius does not implement. Ordinary KV or Mamba state would be wrong. |
 | `bert` | — | model=`bert`; tensor=`bert` | not claimed | config=supported; tensor_map=supported; graph=supported; runtime=deferred; quantized_import=supported | Config extraction, exact pinned tensor closure, encoder-only task dispatch, and synthetic ORT execution are covered, but no pinned real GGUF artifact has passed independent embedding parity. Runtime packaging remains deferred until that evidence exists. |
@@ -365,13 +365,13 @@ before graph construction or durable output.
 | `command-r` | — | model=`command_r`; tensor=`llama`+`command_r_extras` | audited-direct-loader-conditional-union | config=supported; tensor_map=supported; graph=supported; runtime=deferred; quantized_import=supported | Config extraction, exact tensor-name closure, and a full synthetic GGUF graph build are covered, but no representative real-weight GGUF has yet passed ORT parity or generation validation. Runtime packaging remains deferred until that evidence exists. Import requires canonical logit_scale metadata and is restricted to split Q/K/V tensors in the 40-layer Command-R profile; quantization preservation is supported only for that split route. Pinned variants with 64 or more layers require distinct per-head Q/K LayerNorm parameters that the current Attention graph cannot represent. |
 | `dbrx` | — | none (fails before config extraction) | not claimed | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | The pinned DBRX graph requires LayerNorm, a fused QKV projection, Q/K/V projection clamping from attention.clamp_kqv, and a second LayerNorm before its routed experts. Mobius's generic MoE graph uses RMSNorm, separate Q/K/V projections, and no K/Q/V clamp; the existing Hugging Face 'dbrx' registration is therefore not GGUF-compatible. |
 | `deci` | — | model=`llama`; tensor=`llama` | not claimed | config=supported; tensor_map=supported; graph=supported; runtime=deferred; quantized_import=supported | Config extraction, exact tensor-name closure, and a full synthetic GGUF graph build are covered, but no representative real-weight GGUF has yet passed ORT parity or generation validation. Runtime packaging remains deferred until that evidence exists. |
-| `deepseek` | — | none (fails before config extraction) | audited-direct-loader-conditional-union | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | The pinned DeepSeek loader switches from dense SwiGLU to softmax routed experts with a mandatory parallel shared branch at leading_dense_block_count. Existing DeepSeek-family registrations do not establish this GGUF tensor, routing, or quantized expert contract. |
+| `deepseek` | — | model=`deepseek`; tensor=`llama`+`diffusion_fused_qkv`+`deepseek_shared_moe_extras` | audited-direct-loader-conditional-union | config=supported; tensor_map=supported; graph=supported; runtime=deferred; quantized_import=supported | Config extraction, exact tensor-name closure, and a full synthetic GGUF graph build are covered, but no representative real-weight GGUF has yet passed ORT parity or generation validation. Runtime packaging remains deferred until that evidence exists. |
 | `deepseek2` | — | none (fails before config extraction) | audited-direct-loader-conditional-union | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | DeepSeek2 MTP is a complete MLA plus routed/shared MoE block with compressed KV cache, Q/KV LoRA alternatives, target-owned embedding/head fallbacks, and architecture-specific gating. It is not Mobius's dense full-attention MTP head. |
 | `deepseek2-ocr` | — | none (fails before config extraction) | exact-direct-loader-conditional-union | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | DeepSeek-OCR2 is a paired text-plus-vision package, not a generic DeepSeek text model. The text loader, deepseekocr/deepseekocr2 clip sidecars, SAM/projector stages, special token mixing, and cache contract have no single suffix-exact Mobius ownership map. Existing Hugging Face components therefore cannot justify partial GGUF construction. |
 | `deepseek32` | — | none (fails before config extraction) | audited-direct-loader-conditional-union | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | DeepSeek3.2 extends the DeepSeek2 MLA/MoE head with DSA indexer projections, normalization, bias, and sparse-cache metadata. A normal KV-cache NextN task would omit required executed tensors and state. |
 | `deepseek4` | — | none (fails before config extraction) | not claimed | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | The pinned DeepSeek-V4 GGUF runtime uses a dedicated raw sliding-window, CSA, HCA, and indexer compressed-cache ABI with persistent compressor state, rollback snapshots, four-stream hyper-connections, hash/sqrt-softplus routing, and optional MTP storage. Mobius's Hugging Face DeepSeek-V4 graph intentionally exports a dense attention fallback with ordinary KV state, so it is not an exact GGUF runtime graph. |
 | `dflash` | — | model=`DFlashDraftModel`; tensor=`dflash` | not claimed | config=supported; tensor_map=supported; graph=supported; runtime=deferred; quantized_import=supported | This is a target-coupled speculative draft, never a standalone CausalLM. Config extraction, exact tensor closure, target shape/tokenizer validation, and synthetic draft execution are covered, but no pinned real GGUF pair has passed independent target+draft full-logit/proposed-token parity. Runtime packaging remains deferred until that evidence and an acceptance-loop integration exist. |
-| `dots1` | — | none (fails before config extraction) | audited-direct-loader-conditional-union | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | Dots1 adds Q/K norms, a dense prefix, metadata-selected expert gating, optional correction bias, and routed plus shared experts. A fixed Qwen/DeepSeek MoE graph would silently change its routing contract. |
+| `dots1` | — | model=`dots1`; tensor=`llama`+`diffusion_fused_qkv`+`moe_qk_norm_extras`+`deepseek_shared_moe_extras` | audited-direct-loader-conditional-union | config=supported; tensor_map=supported; graph=supported; runtime=deferred; quantized_import=supported | Config extraction, exact tensor-name closure, and a full synthetic GGUF graph build are covered, but no representative real-weight GGUF has yet passed ORT parity or generation validation. Runtime packaging remains deferred until that evidence exists. |
 | `dots3note` | — | none (fails before config extraction) | audited-direct-loader-conditional-union | config=deferred; tensor_map=deferred; graph=deferred; runtime=deferred; quantized_import=supported | Dots3Note preserves an MLA/DSA trunk and a dense sliding-MLA NextN block, but the pinned loader explicitly has no MTP graph and skips the head. Mobius cannot infer runtime semantics from its serialized tensors. |
 | `dream` | — | model=`dream`; tensor=`llama`+`diffusion_fused_qkv` | not claimed | config=supported; tensor_map=supported; graph=supported; runtime=deferred; quantized_import=supported | Config extraction, suffix-exact tensor closure, masked-diffusion task dispatch, and synthetic full-sequence execution are covered, but no pinned real GGUF has passed independent Hugging Face/llama.cpp masked-step logit parity and deterministic multi-step generation parity. Runtime packaging remains deferred until both exist. |
 | `eagle3` | — | model=`Eagle3DraftModel`; tensor=`eagle3` | not claimed | config=supported; tensor_map=supported; graph=supported; runtime=deferred; quantized_import=supported | This is a target-coupled speculative draft, never a standalone CausalLM. Config extraction, exact tensor closure, target shape/tokenizer validation, and synthetic draft execution are covered, but no pinned real GGUF pair has passed independent target+draft full-logit/proposed-token parity. Runtime packaging remains deferred until that evidence and an acceptance-loop integration exist. |

src/mobius/_configs/_base.py

@@ -494,10 +494,12 @@ class ArchitectureConfig(BaseModelConfig):
     n_group: int = 1
     topk_group: int = 1
     routed_scaling_factor: float = 1.0
+    routing_weight_normalization_floor: float | None = None
     scoring_func: str = "softmax"
     topk_method: str = "greedy"
     first_k_dense_replace: int = 0
     n_shared_experts: int | None = None
+    use_expert_bias: bool | None = None
     disable_qmoe: bool = False
 
     # MiniMax-01 hybrid attention and normalized-residual scaling.
@@ -2001,7 +2003,7 @@ class Lfm2MoeConfig(CausalLMConfig):
     """Configuration for LFM2MoE's dense-prefix and routed-expert feed-forwards."""
 
     num_dense_layers: int = 2
-    use_expert_bias: bool = True
+    use_expert_bias: bool | None = True
 
     @classmethod
     def from_transformers(cls, config, parent_config=None) -> Lfm2MoeConfig:

src/mobius/_registry.py

@@ -437,6 +437,7 @@ def _detect_fallback_registration(hf_config) -> ModelRegistration | None:
     "kclgpt": ModelRegistration(CodeShellCausalLMModel, config_class=CodeShellConfig),
     "csm": ModelRegistration(CausalLMModel),
     "dots1": ModelRegistration(DeepSeekV3CausalLMModel),
+    "deepseek": ModelRegistration(DeepSeekV3CausalLMModel),
     "evolla": ModelRegistration(CausalLMModel),
     "exaone": ModelRegistration(CausalLMModel),
     "helium": ModelRegistration(CausalLMModel),
@@ -616,6 +617,7 @@ def _detect_fallback_registration(hf_config) -> ModelRegistration | None:
     "arctic": ModelRegistration(MoECausalLMModel),
     "dbrx": ModelRegistration(MoECausalLMModel),
     "ernie4_5_moe": ModelRegistration(Ernie45MoECausalLMModel),
+    "bailing_moe": ModelRegistration(Ernie45MoECausalLMModel),
     "flex_olmo": ModelRegistration(MoECausalLMModel),
     "glm4_moe": ModelRegistration(Glm4MoECausalLMModel),
     "granitemoe": ModelRegistration(GraniteMoECausalLMModel),

src/mobius/integrations/gguf/_arch_registry.py

@@ -479,22 +479,6 @@
         "interleaved sliding-window attention. Mobius has no graph or cache task owning "
         "that complete topology or its expert sidecars."
     ),
-    "bailingmoe": (
-        "BailingMoE requires an untied head, split-and-permuted Q/K/V conversion, and an "
-        "all-layer softmax routed MoE with mandatory shared experts. Mobius has no exact "
-        "expert-major tensor transform, preservation contract, or parity evidence."
-    ),
-    "deepseek": (
-        "The pinned DeepSeek loader switches from dense SwiGLU to softmax routed experts "
-        "with a mandatory parallel shared branch at leading_dense_block_count. Existing "
-        "DeepSeek-family registrations do not establish this GGUF tensor, routing, or "
-        "quantized expert contract."
-    ),
-    "dots1": (
-        "Dots1 adds Q/K norms, a dense prefix, metadata-selected expert gating, optional "
-        "correction bias, and routed plus shared experts. A fixed Qwen/DeepSeek MoE graph "
-        "would silently change its routing contract."
-    ),
     "ernie4_5": (
         "ERNIE 4.5 requires exact fused-QKV and fused-gate/up converter splits plus an "
         "optional attention-output bias and ERNIE-specific position metadata. Similarity "
@@ -1519,6 +1503,50 @@
         runtime=Support.DEFERRED,
         reason=_RUNTIME_VALIDATION_PENDING,
     ),
+    GGUFArchitectureSpec(
+        gguf_arch="bailingmoe",
+        model_type="bailing_moe",
+        tensor_map_recipe=("llama", "diffusion_fused_qkv", "moe_extras"),
+        config_key_map="conventional_shared_moe",
+        config_postprocessor="conventional_shared_moe",
+        required_metadata=(
+            "attention.layer_norm_rms_epsilon",
+            "expert_feed_forward_length",
+            "expert_shared_count",
+        ),
+        tensor_processor="llama",
+        llama_qk_permute=True,
+        runtime=Support.DEFERRED,
+        reason=_RUNTIME_VALIDATION_PENDING,
+    ),
+    GGUFArchitectureSpec(
+        gguf_arch="deepseek",
+        model_type="deepseek",
+        tensor_map_recipe=("llama", "diffusion_fused_qkv", "deepseek_shared_moe_extras"),
+        config_key_map="conventional_shared_moe",
+        config_postprocessor="conventional_shared_moe",
+        required_metadata=("attention.layer_norm_rms_epsilon",),
+        tensor_processor="llama",
+        llama_qk_permute=True,
+        runtime=Support.DEFERRED,
+        reason=_RUNTIME_VALIDATION_PENDING,
+    ),
+    GGUFArchitectureSpec(
+        gguf_arch="dots1",
+        model_type="dots1",
+        tensor_map_recipe=(
+            "llama",
+            "diffusion_fused_qkv",
+            "moe_qk_norm_extras",
+            "deepseek_shared_moe_extras",
+        ),
+        config_key_map="conventional_shared_moe",
+        config_postprocessor="conventional_shared_moe",
+        required_metadata=("attention.layer_norm_rms_epsilon",),
+        tensor_processor="llama",
+        runtime=Support.DEFERRED,
+        reason=_RUNTIME_VALIDATION_PENDING,
+    ),
     # ------------------------- Remaining conventional-attention MoE (audited/deferred)
     GGUFArchitectureSpec(
         gguf_arch="arctic",

src/mobius/integrations/gguf/_arch_registry_test.py

@@ -57,23 +57,20 @@
 #: Number of importable architectures. Pinned so that adding support is a
 #: deliberate act that also updates the documented support matrix, and so that
 #: accidentally losing an architecture is a failure rather than a silence.
-_EXPECTED_SUPPORTED_COUNT = 71
+_EXPECTED_SUPPORTED_COUNT = 74
 _PROMOTED_CONVENTIONAL_DECODERS = frozenset(
     {"codeshell", "command-r", "jais2", "orion", "qwen", "starcoder", "xverse"}
 )
 _FINAL_CENSUS_CLOSURE = frozenset(
     {
         "afmoe",
-        "bailingmoe",
         "bailingmoe2",
         "bitnet",
         "codeshell",
         "cohere2moe",
         "command-r",
-        "deepseek",
         "deepseek2",
         "deepseek32",
-        "dots1",
         "dots3note",
         "ernie4_5",
         "ernie4_5-moe",
@@ -149,9 +146,12 @@
     {
         "arcee",
         "baichuan",
+        "bailingmoe",
         "bert",
         "cohere2",
         "deci",
+        "deepseek",
+        "dots1",
         "dflash",
         "dream",
         "eagle3",