Add Nemotron-H MoE GGUF support - #614
Merged
Merged
Conversation
Performance Comparison
|
justinchuby
force-pushed
the
justinchuby-complete-jamba-gguf
branch
from
August 25, 2026 18:02
d34bcd4 to
f188ad8
Compare
Implement exact hybrid schedule reconstruction, sigmoid correction-bias routing tensors, stacked and shared experts, optional latent projections, strict tensor closure, and fail-closed quantization/MTP handling. Add synthetic parity, ORT mixed-state, quantized-source, round-trip, and malformed-input coverage, and update the support census. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Adapt the reviewed fail-closed quantization behavior to the centralized architecture policy introduced by the squashed parent, and regenerate the support matrix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Detect quantized stacked expert weights before policy evaluation and reject grouped Nemotron-H routing until its group-selection semantics are implemented. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-complete-nemotron-h-moe-gguf
branch
from
August 25, 2026 18:20
14d9be9 to
41ed446
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds first-class GGUF import support for the nemotron_h_moe backbone in Mobius, including exact hybrid layer scheduling (Mamba2/attention/dense/MoE), strict tensor-closure validation, and expanded synthetic/test coverage. This fits into the existing GGUF integration stack by extending architecture registry, config extraction, tensor mapping, and pre-graph validation for a new hybrid MoE variant while keeping ORT GenAI packaging explicitly deferred.
Changes:
- Add
nemotron_h_moeto the GGUF architecture registry with supported config/tensor-map/graph verdicts, plus deferred runtime and rejected quantized import. - Extend GGUF config and tensor mapping logic to reconstruct Nemotron-H hybrid schedules (including routed/shared/latent MoE semantics) and validate tensor closure + logical shapes before graph construction.
- Add/adjust tests and documentation to cover Nemotron-H MoE behavior, malformed inputs, dequantization requirements, and updated support boundaries.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/synthetic_parity_test.py | Updates HF config translation and adds Nemotron-H MoE-specific parity targeting (correction-bias-driven expert selection). |
| tests/_test_configs.py | Adds a tiny Nemotron-H latent-MoE test config variant. |
| src/mobius/integrations/gguf/_upstream_data/llamacpp_pin.json | Updates pinned upstream tensor inventory for Nemotron-H naming. |
| src/mobius/integrations/gguf/_tensor_mapping.py | Adds Nemotron-H MoE tensor stem mappings and aliases nemotron_h_moe to the Nemotron-H table. |
| src/mobius/integrations/gguf/_tensor_mapping_test.py | Adds mapping assertions for new Nemotron-H MoE stems. |
| src/mobius/integrations/gguf/_mmproj.py | Adds early architecture validation for Gemma4 VLM construction. |
| src/mobius/integrations/gguf/_mmproj_test.py | Aligns tests with new Gemma4 architecture validation behavior. |
| src/mobius/integrations/gguf/_config_mapping.py | Implements nemotron_h_moe schedule derivation and Nemotron-H MoE config postprocessing (incl. latent size, routing constraints). |
| src/mobius/integrations/gguf/_config_mapping_test.py | Adds tests for nemotron_h_moe schedule derivation, defaults, and routing rejection cases. |
| src/mobius/integrations/gguf/_builder.py | Extends pre-graph hybrid tensor contract validation and adds Nemotron-H logical shape validation. |
| src/mobius/integrations/gguf/_builder_test.py | Adds comprehensive synthetic Nemotron-H MoE GGUF build/roundtrip/runtime-threading/malformed-input coverage. |
| src/mobius/integrations/gguf/_arch_registry.py | Switches nemotron_h_moe from rejected to supported (graph) with deferred runtime and explicit dequantization requirements. |
| src/mobius/integrations/gguf/_arch_registry_test.py | Updates supported-architecture census and adds nemotron_h_moe verdict assertions. |
| docs/cli_reference.md | Updates CLI docs to reflect nemotron_h_moe support boundary and dequantization requirement. |
| docs/api/build_from_gguf.md | Updates support matrix and replaces the prior waiver section with a new Nemotron-H MoE support boundary section. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
231
to
234
| # Additional divergences (newly registered models) | ||
| # NemotronH Mamba2 layers diverge (cos=0.65): LinearAttention gated-SSM | ||
| # recurrence on CPU produces different results than HF's naive Mamba2. | ||
| # Attention-only layers match perfectly (cos=0.9999). |
Comment on lines
+2098
to
+2105
| kv_raw = metadata[f"{arch}.attention.head_count_kv"] | ||
| kv_by_layer = [int(value) for value in kv_raw] | ||
| head_dim = int( | ||
| metadata.get( | ||
| f"{arch}.attention.key_length", | ||
| hidden // next(value for value in heads_by_layer if value > 0), | ||
| ) | ||
| ) |
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
## Summary - add exact GraniteMoeHybrid GGUF config extraction for mixed Mamba2/attention schedules, routed experts, optional shared experts, and Granite scaling/norm metadata - map and fuse separate routed/shared gate+up tensors with value-tested gate-then-up ordering while keeping quantized expert preservation fail-closed - enforce strict tensor closure and logical shapes, including optional all-or-none attention biases matching official Granite 4 H GGUFs - cover ORT prefill/decode/reorder/replay, package save/reload, CLI import, Transformers synthetic parity, and malformed sources - update the generated GGUF support census from dense-only to routed-MoE support ## Stack - Depends on #614 ## Validation - `python -m pytest tests/build_graph_test.py tests/cli_test.py src/ -q -k 'not phi4mm and not apply_weights_unknown and not glm_moe_dsa' --tb=short -n auto` — 7141 passed, 54 skipped - `python -m pytest tests/synthetic_parity_test.py -k granitemoehybrid -q --tb=short` — 1 passed - `python -m pytest tests/weight_alignment_test.py -k granitemoehybrid -q --tb=short` — 1 passed - focused GraniteHybrid GGUF closure/build tests — 19 passed - changed-file `lintrunner` — clean The unfiltered suite additionally reached 7164 passed and 54 skipped; its only failures were two reproducible pre-existing `glm_moe_dsa` shape-inference/checker failures unrelated to this diff. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
## Summary - clean up still-valid typing, documentation, test robustness, error-message, maintainability, and performance findings left by the GGUF PR stack - preserve the behavioral fixes merged in #625-#632 - hash reused multi-GB GGUF sources once, at the final pre-publication integrity gate, while retaining cheap identity checks around staging Exact base: `2db9d33debdc254d879a51b14434c9a81c230f4f` Exact head: `4541ed2bc9d2ab4227484510b4a85b2d9113eb25` ## Reconstructed original 17-item low-priority tranche The persisted audit retained only the totals, so this list was reconstructed from the live threads and current source. All but the already-fixed #596 comment are addressed in this PR. | PR | Comment | Disposition | |---|---:|---| | #550 | 3837144656 | Implemented: correct tuple return annotation | | #552 | 3837389183 | Implemented: multichannel waveform shape docs | | #559 | 3837716261 | Implemented: name-based cache assertions | | #573 | 3854308350 | Implemented: one final GGUF hash, with integrity regression coverage | | #574 | 3854345521 | Implemented: `TensorRole \| None` typing | | #574 | 3854345597 | Implemented: removed obsolete verdict filtering | | #577 | 3854472859 | Implemented: documented SSM sequence length | | #578 | 3843112613 | Implemented: documented F64 passthrough | | #579 | 3854518332 | Implemented: generalized fused-projection error | | #580 | 3854576300 | Implemented: removed brittle node counts | | #583 | 3854681386 | Implemented: documented conditional draft outputs | | #587 | 3854816872 | Implemented: corrected MTP output contract docs | | #596 | 3846110059 | Already fixed on base: unambiguous GQA bias comment | | #600 | 3855174281 | Implemented: metadata-count-only MTP error | | #607 | 3855776048 | Implemented: stable route-field assertions | | #607 | 3855776086 | Implemented: public tensor iterator | | #609 | 3856486136 | Implemented: fail-closed LM-head comment | ## Current unresolved-thread disposition This covers all 48 Copilot threads returned by the reproducible #600-#630 query. The one human #623 thread is excluded. | PR | Comment | Current-main disposition and evidence | |---|---:|---| | #600 | 3855174177 | Already fixed by #629: package cycle and reserved-sidecar validation | | #600 | 3855174238 | Already fixed by #629: explicit MTP sidecar naming/loading | | #600 | 3855174281 | Implemented here: error no longer invents an observed block count | | #602 | 3848155961 | Outside exact stack; already fixed: top-level `expert_dtype` is classified before early return | | #602 | 3848155983 | Outside exact stack; still-valid behavioral block-quant validation, unchanged | | #602 | 3848156000 | Outside exact stack; still-valid truncated-read behavioral finding, unchanged | | #602 | 3848156022 | Outside exact stack; still-valid descriptor byte/dtype validation, unchanged | | #602 | 3848156040 | Outside exact stack; still-valid expert-bank payload validation, unchanged | | #603 | 3855249765 | Already fixed by #630: runtime preflight preserves shard sets | | #603 | 3855249840 | Already fixed by #630: success output follows durable runtime publication | | #604 | 3855343082 | Implemented here: graph-only MTP persistence distinguished from runtime rejection | | #604 | 3855343131 | Already fixed by #630: runtime success messages are atomic | | #607 | 3855776001 | Implemented here: missing generation golden skips before provenance read | | #607 | 3855776048 | Implemented here: only stable route fields are asserted | | #607 | 3855776086 | Implemented here: tensor count uses `tensor_items_raw()` | | #608 | 3856079371 | Still-valid behavioral cache-symlink containment finding; unchanged | | #608 | 3856079415 | Still-valid behavioral lowercase-digest validation finding; unchanged | | #609 | 3856486136 | Implemented here: comment matches value-preserving policy | | #610 | 3855541683 | Already fixed by #628: Falcon bias precedence is explicit | | #610 | 3855541761 | Already fixed by #628: CTRL tiny config exercises projection biases | | #611 | 3856595840 | Implemented here: runtime test resolves the distribution providing the module | | #612 | 3855677383 | Already fixed by #625: supported-version endianness detection | | #612 | 3855677427 | Implemented here: shared `INT64_MAX` sentinel | | #612 | 3855677460 | Implemented here: shared PLaMo2 width inference | | #612 | 3855677486 | Implemented here: accepted PLaMo2 activation spellings are explicit | | #613 | 3855845678 | Implemented here: canonical issue URL | | #613 | 3855845757 | Implemented here: Mamba-1 function-registration docs | | #613 | 3855845806 | Implemented here: test expects the canonical issue URL | | #614 | 3855988545 | Implemented here: removed stale Nemotron-H divergence comments | | #614 | 3855988597 | Already fixed by #628: zero-head geometry raises actionable `ValueError` | | #615 | 3856082290 | Already fixed by #626: dense GraniteHybrid bias closure | | #618 | 3856729330 | Implemented here: required routes filter ORT GenAI evidence | | #618 | 3856729409 | Implemented here: env-selected runtime version is authoritative | | #618 | 3856729490 | Stale/N/A: PR-description-only matrix claim; repository workflow claims one pinned version | | #618 | 3856729563 | Implemented here: schema tail restored to normal indentation | | #619 | 3856342484 | Already fixed on base: Kimi Linear uses `/issues/605` | | #619 | 3856342532 | Already fixed by #628: config rejects convolution kernels below 2 | | #619 | 3856342580 | Already fixed by #628: GGUF contract rejects convolution kernels below 2 | | #620 | 3855717041 | Already fixed by #627: tied LM-head-only checkpoints are retained | | #621 | 3856722324 | Already fixed by #628: Kimi-K3 required metadata is complete | | #623 | 3855931210 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939302 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939358 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939394 | N/A to current main: comment belongs to open, unmerged #623 | | #624 | 3856777152 | Implemented here: runtime compatibility reuses the emitted model type | | #625 | 3857049733 | Implemented here: unsupported header reports both endian candidates | | #629 | 3857313182 | Newer post-audit behavioral sidecar-symlink cleanup finding; unchanged | | #629 | 3857313251 | Newer post-audit cross-platform path-safety finding; unchanged | ## Validation - affected GGUF/package/ORT GenAI/model/schema tests: 1,040 passed - broad non-integration suite: 7,851 passed, 56 skipped, 1 subtest passed - generated GGUF docs checks: 7 passed - initialized `lintrunner`; full lint/format passed - GPT-5.6 Sol medium review: one integrity finding fixed; re-review found no significant issues Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nemotron_h_moeGGUF backbone import with mixed Mamba2/attention/dense/MoE schedule reconstructionValidation
python -m pytest src/mobius/integrations/gguf -q --tb=short— 2623 passedpython -m pytest tests/build_graph_test.py tests/synthetic_parity_test.py -k nemotron_h -q --tb=short— 11 passed, 6 skippedThe broad non-integration suite reached 7135 passing tests and exposed two unrelated existing
glm_moe_dsashape-inference/checker failures.Stacked on #613 (
justinchuby-complete-jamba-gguf). ORT GenAI packaging remains deferred to #605 because the released runtime schema does not represent heterogeneous KV/convolution/recurrent state slots. Released 30B-A3B GGUFs also remain rejected before graph construction because they append an unsupported combined attention+MoE MTP block.