Fail closed on lossy GGUF quantization preservation - #609
Merged
Conversation
Performance Comparison
|
justinchuby
force-pushed
the
justinchuby-fix-gguf-q4-parity
branch
from
August 25, 2026 03:12
0806902 to
d673f9d
Compare
justinchuby
changed the base branch from
justinchuby-validate-small-gguf-models
to
justinchuby-materialize-gguf-tokenizers
August 25, 2026 03:12
justinchuby
force-pushed
the
justinchuby-materialize-gguf-tokenizers
branch
from
August 25, 2026 18:31
654abfc to
0e791a5
Compare
Base automatically changed from
justinchuby-materialize-gguf-tokenizers
to
main
August 25, 2026 18:32
Distinguish lossless affine/native preservation from mixed GGUF presets that require value-changing requantization. Reject unsupported projection, embedding, head, multimodal, MTP, and fused-expert routes instead of emitting incorrect MatMulNBits graphs, while retaining explicit float import as the opt-in fallback. Add asymmetric Q4_K and MatMulNBits contract tests plus real SmolLM2 Q4_K_M prefill, decode, generation, save/reload, API, and CLI coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Populate the pinned tokenizer provenance required by the materialized-tokenizer base for the SmolLM2 Q4_K_M runtime case. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Keep exact fused projection repacks byte-preserving, exempt explicitly float graph roles from packed-contract preflight, and strengthen Q4 runtime provenance coverage after the latest GGUF importer additions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Pin the exact float and Q4 dequantized import-route fingerprints produced on the MiniMax-updated GGUF registry. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Use the same explicit float-role classification during packed-target detection so encoder embeddings and Jamba recurrent projections cannot force exact affine projections onto an incompatible target. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Exclude encoder position tables, norms, and Jamba convolutional state weights from packed-target selection so only actual quantized graph roles constrain MatMulNBits. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Kimi Linear's rank-3 MLA projections require reshape and requantization, which changes their dequantized values. Cover the preservation-mode rejection and retain explicit float import as the safe fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Kimi Linear reshapes its rank-3 MLA projections through dequantized values. Reject every quantized source format in preservation mode, including otherwise exact Q4_0 routes, and require the explicit float fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-fix-gguf-q4-parity
branch
from
August 25, 2026 19:20
d673f9d to
62583ae
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR tightens Mobius’s GGUF quantization-preservation policy to fail closed when preserving quantized weights would require lossy dequantize/requantize, addressing correctness divergence for mixed presets like Q4_K_M. It also expands test coverage (unit + real-artifact integration) and updates user-facing docs to reflect the new preservation semantics.
Changes:
- Introduce “value-preserving vs lossy” preservation semantics (e.g.,
losslessaffine repacks) and reject lossy preservation routes underkeep_quantized=True. - Extend preservation gating across builders (text, multimodal, MTP, native blocks, fused tensors) and improve rejection diagnostics.
- Add targeted correctness tests (Q4_K dequant, MatMulNBits value checks, pinned real-artifact integration) and update documentation to match the new behavior.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/gguf_small_model_runtime_integration_test.py | Adds a pinned SmolLM2 Q4_K_M integration test validating fail-closed preservation and parity under explicit dequantization. |
| tests/gguf_dense_cohort_integration_test.py | Simplifies cohort integration to assert fail-closed behavior for incompatible dense GGUF artifacts. |
| src/mobius/integrations/gguf/_spec.py | Adds a lossless flag to affine repack specs and a preserves_values helper for policy decisions. |
| src/mobius/integrations/gguf/_repacker_test.py | Adds a focused Q4_K dequantization/value test to validate layout semantics. |
| src/mobius/integrations/gguf/_quant_registry.py | Marks which affine targets are lossless, refines lm-head preservation classification, and exposes lossless-preservation type names. |
| src/mobius/integrations/gguf/_quant_registry_test.py | Updates expectations for lm-head preservation and adds tests for lossless-preservation advertisement. |
| src/mobius/integrations/gguf/_mtp_test.py | Adjusts MTP tests to require explicit dequantization when preservation would be lossy/incompatible. |
| src/mobius/integrations/gguf/_mmproj.py | Adds fail-closed checks for mixed float/quant projections and packed tensors in multimodal quantized imports. |
| src/mobius/integrations/gguf/_mmproj_test.py | Adds tests covering float-projection fail-closed behavior and packed per-layer table rejection. |
| src/mobius/integrations/gguf/_builder.py | Centralizes preservation rejection logic, improves quant param detection, and enforces lossless-only preservation in quantized state loading. |
| src/mobius/integrations/gguf/_builder_test.py | Updates/extends unit tests to assert fail-closed behavior across mixed qtypes, native blocks, embeddings/heads, fused tensors, and hybrid routes. |
| src/mobius/functions/matmul_nbits_test.py | Adds an accuracy_level=4 test to validate ORT’s MatMulNBits consumption of Mobius packed weights/zero-points. |
| docs/model-catalog.md | Updates GGUF preservation description to explicitly state fail-closed on lossy routes. |
| docs/cli_reference.md | Updates CLI docs to describe fail-closed behavior and direct users to --dequantize for explicit float conversion. |
| docs/api/build_from_gguf.md | Updates API docs to reflect rejection of lossy projection preservation and clarify mixed-preset handling. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
164
to
167
| #: Types an untied ``lm_head`` may stay quantized in. Broader than the union of | ||
| #: the two tables above because the head is also allowed to ride the generic | ||
| #: requantization path. | ||
| _LM_HEAD_PRESERVE: frozenset[str] = frozenset( |
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
## Summary - emit the released architecture-neutral `model.type: "decoder"` contract for compatible single-model decoder-only text graphs - derive semantic inputs, cache templates, outputs, sparse global cache slots, recurrent pairs, and sliding-window metadata from the optimized ONNX graph - preserve runtime-specific LFM2 and Phi-3 LongRoPE behavior, fail closed for Mobius GPT-2's incompatible separate-cache ABI and unsupported state layouts, and record runtime compatibility without emitting main-only `state_groups` - keep GGUF runtime promotion evidence-gated while validating the exact SmolLM F16/CPU/tokenizer route end to end ## Validation - 228 focused ORT GenAI/GGUF config tests - 2,871 affected CLI/ORT GenAI/GGUF tests - exact SmolLM deterministic generation on `onnxruntime-genai` 0.14.1 and 0.15.2 - exact Q4_K_M fail-closed/dequantized parity test from the stacked base - full initialized lintrunner pass - two independent high-confidence reviews plus a final fix verification Stacked on #609. --------- Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
Adapt Kimi-K3 quantized import to the fail-closed policy from #609: reject lossy rank-3 MLA reshaping, retain explicit float import, and split fused Q4_0 KV-B with exact packed-row reordering. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
Adapt Kimi-K3 quantized import to the fail-closed policy from #609: reject lossy rank-3 MLA reshaping, retain explicit float import, and split fused Q4_0 KV-B with exact packed-row reordering. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
Adapt Kimi-K3 quantized import to the fail-closed policy from #609: reject lossy rank-3 MLA reshaping, retain explicit float import, and split fused Q4_0 KV-B with exact packed-row reordering. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
## Summary - add a dedicated Kimi-K3 config, model, and heterogeneous-state task with exact KDA/NoPE gated-MLA scheduling, AttnRes mixing, SiTU latent MoE, shared experts, and an untied output head - import pinned llama.cpp `kimi-k3` GGUF metadata and tensors with strict metadata/tensor/shape/storage closure and malformed-input rejection - preserve lossless quantization routes under #609: separate rank-3 MLA projections fail closed unless explicitly dequantized, while fused Q4_0 KV-B is split by exact packed-row reordering across weights, scales, and zero-points - preserve valid KDA convolution history across padding, validate kernel/state/task contracts, and cover replay, reorder, float/quantized import, roundtrip, CLI, and negative paths - keep generic OGA runtime packaging truthfully deferred under #605 because released cache schemas cannot represent the heterogeneous state ABI ## Reconstruction Reconstructed after #619 was admin squash-merged. The PR contains only the four-commit Kimi-K3 delta and review follow-ups on live-main base `04b7e3f6f2d9de5d5741aafb8fa6375a18eee693`. This preserves #611 graph-driven ORT GenAI decoder configs, #618 ORT GenAI end-to-end CI, #624 generic decoder config migration, and the Kimi Linear, MiniMax, runtime, and fail-closed quantization changes already on main. `git range-diff` reports all four replayed commits as patch-identical (`=`). ## Validation - Kimi-K3 model and GGUF tests: 30 passed - focused builder/build-graph Kimi-K3 checks: 4 passed, 1900 deselected - affected generic ORT config/runtime/E2E tests: 250 passed - affected OGA metadata tests: 94 passed, 1 skipped - broad non-integration suite: 7765 passed, 56 skipped, 1 subtest passed - repository lintrunner passed - final GPT-5.6 Sol medium review reported no findings ## Real-checkpoint note The pinned `yujiepan/kimi-k3-tiny-random@a5c86ee03f07f7b141508b0108304a1447fbb345` checkpoint was assessed, but macOS cannot run its required `fla-core` kernels and its selective compressed-tensors MXFP4 expert representation is unsupported by the generic HF loader. That format fails explicitly rather than producing an incorrectly quantized graph. --------- Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
## Summary - clean up still-valid typing, documentation, test robustness, error-message, maintainability, and performance findings left by the GGUF PR stack - preserve the behavioral fixes merged in #625-#632 - hash reused multi-GB GGUF sources once, at the final pre-publication integrity gate, while retaining cheap identity checks around staging Exact base: `2db9d33debdc254d879a51b14434c9a81c230f4f` Exact head: `4541ed2bc9d2ab4227484510b4a85b2d9113eb25` ## Reconstructed original 17-item low-priority tranche The persisted audit retained only the totals, so this list was reconstructed from the live threads and current source. All but the already-fixed #596 comment are addressed in this PR. | PR | Comment | Disposition | |---|---:|---| | #550 | 3837144656 | Implemented: correct tuple return annotation | | #552 | 3837389183 | Implemented: multichannel waveform shape docs | | #559 | 3837716261 | Implemented: name-based cache assertions | | #573 | 3854308350 | Implemented: one final GGUF hash, with integrity regression coverage | | #574 | 3854345521 | Implemented: `TensorRole \| None` typing | | #574 | 3854345597 | Implemented: removed obsolete verdict filtering | | #577 | 3854472859 | Implemented: documented SSM sequence length | | #578 | 3843112613 | Implemented: documented F64 passthrough | | #579 | 3854518332 | Implemented: generalized fused-projection error | | #580 | 3854576300 | Implemented: removed brittle node counts | | #583 | 3854681386 | Implemented: documented conditional draft outputs | | #587 | 3854816872 | Implemented: corrected MTP output contract docs | | #596 | 3846110059 | Already fixed on base: unambiguous GQA bias comment | | #600 | 3855174281 | Implemented: metadata-count-only MTP error | | #607 | 3855776048 | Implemented: stable route-field assertions | | #607 | 3855776086 | Implemented: public tensor iterator | | #609 | 3856486136 | Implemented: fail-closed LM-head comment | ## Current unresolved-thread disposition This covers all 48 Copilot threads returned by the reproducible #600-#630 query. The one human #623 thread is excluded. | PR | Comment | Current-main disposition and evidence | |---|---:|---| | #600 | 3855174177 | Already fixed by #629: package cycle and reserved-sidecar validation | | #600 | 3855174238 | Already fixed by #629: explicit MTP sidecar naming/loading | | #600 | 3855174281 | Implemented here: error no longer invents an observed block count | | #602 | 3848155961 | Outside exact stack; already fixed: top-level `expert_dtype` is classified before early return | | #602 | 3848155983 | Outside exact stack; still-valid behavioral block-quant validation, unchanged | | #602 | 3848156000 | Outside exact stack; still-valid truncated-read behavioral finding, unchanged | | #602 | 3848156022 | Outside exact stack; still-valid descriptor byte/dtype validation, unchanged | | #602 | 3848156040 | Outside exact stack; still-valid expert-bank payload validation, unchanged | | #603 | 3855249765 | Already fixed by #630: runtime preflight preserves shard sets | | #603 | 3855249840 | Already fixed by #630: success output follows durable runtime publication | | #604 | 3855343082 | Implemented here: graph-only MTP persistence distinguished from runtime rejection | | #604 | 3855343131 | Already fixed by #630: runtime success messages are atomic | | #607 | 3855776001 | Implemented here: missing generation golden skips before provenance read | | #607 | 3855776048 | Implemented here: only stable route fields are asserted | | #607 | 3855776086 | Implemented here: tensor count uses `tensor_items_raw()` | | #608 | 3856079371 | Still-valid behavioral cache-symlink containment finding; unchanged | | #608 | 3856079415 | Still-valid behavioral lowercase-digest validation finding; unchanged | | #609 | 3856486136 | Implemented here: comment matches value-preserving policy | | #610 | 3855541683 | Already fixed by #628: Falcon bias precedence is explicit | | #610 | 3855541761 | Already fixed by #628: CTRL tiny config exercises projection biases | | #611 | 3856595840 | Implemented here: runtime test resolves the distribution providing the module | | #612 | 3855677383 | Already fixed by #625: supported-version endianness detection | | #612 | 3855677427 | Implemented here: shared `INT64_MAX` sentinel | | #612 | 3855677460 | Implemented here: shared PLaMo2 width inference | | #612 | 3855677486 | Implemented here: accepted PLaMo2 activation spellings are explicit | | #613 | 3855845678 | Implemented here: canonical issue URL | | #613 | 3855845757 | Implemented here: Mamba-1 function-registration docs | | #613 | 3855845806 | Implemented here: test expects the canonical issue URL | | #614 | 3855988545 | Implemented here: removed stale Nemotron-H divergence comments | | #614 | 3855988597 | Already fixed by #628: zero-head geometry raises actionable `ValueError` | | #615 | 3856082290 | Already fixed by #626: dense GraniteHybrid bias closure | | #618 | 3856729330 | Implemented here: required routes filter ORT GenAI evidence | | #618 | 3856729409 | Implemented here: env-selected runtime version is authoritative | | #618 | 3856729490 | Stale/N/A: PR-description-only matrix claim; repository workflow claims one pinned version | | #618 | 3856729563 | Implemented here: schema tail restored to normal indentation | | #619 | 3856342484 | Already fixed on base: Kimi Linear uses `/issues/605` | | #619 | 3856342532 | Already fixed by #628: config rejects convolution kernels below 2 | | #619 | 3856342580 | Already fixed by #628: GGUF contract rejects convolution kernels below 2 | | #620 | 3855717041 | Already fixed by #627: tied LM-head-only checkpoints are retained | | #621 | 3856722324 | Already fixed by #628: Kimi-K3 required metadata is complete | | #623 | 3855931210 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939302 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939358 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939394 | N/A to current main: comment belongs to open, unmerged #623 | | #624 | 3856777152 | Implemented here: runtime compatibility reuses the emitted model type | | #625 | 3857049733 | Implemented here: unsupported header reports both endian candidates | | #629 | 3857313182 | Newer post-audit behavioral sidecar-symlink cleanup finding; unchanged | | #629 | 3857313251 | Newer post-audit cross-platform path-safety finding; unchanged | ## Validation - affected GGUF/package/ORT GenAI/model/schema tests: 1,040 passed - broad non-integration suite: 7,851 passed, 56 skipped, 1 subtest passed - generated GGUF docs checks: 7 passed - initialized `lintrunner`; full lint/format passed - GPT-5.6 Sol medium review: one integrity finding fixed; re-review found no significant issues Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the preserved Q4_K_M/MatMulNBits correctness divergence exposed by #607.
Q4_K_M is a model-level mixed preset, not a GGML tensor type. The pinned SmolLM artifact contains Q5_0=166, Q4_K=16, Q8_0=15, Q6_K=14, and F32=61 tensors. Mobius correctly decoded Q4_K's 144-byte/256-value layout, six-bit scale/min fields, nibble order, orientation, and signed minimum subtraction, and ORT correctly consumed the emitted affine packing. The divergence came earlier: the importer normalized the mixed Q5/Q4_K/Q6/Q8 source tensors into one four-bit MatMulNBits contract, adding a second lossy quantization.
This change:
keep_quantized=True;keep_quantized=False/--dequantizeas the opt-in float route;Root-cause evidence
The first divergence is already present in layer 0 before ORT execution:
The Q4_K formula is
value = d * sub_scale * q - dmin * sub_min. Although Q4_K itself can be represented by four-bit MatMulNBits with floating zero points, the full Q4_K_M mixture cannot be represented losslessly by one four-bit graph contract. ORT 1.29 also advertises 8-bit MatMulNBits in schema but the tested CPU unpacked path rejects it withOnly 2b and 4b quantization is supported for unpacked compute for now.Validation
198, 198, 18, 504, 2388, 13685, 284, 5208, 28, 198.glm_moe_dsashape/checker failures reproduce on the base and are unrelated.Verdict
This is an importer preservation-policy bug, not an ORT Q4 packing bug. Preservation now succeeds only for value-equivalent routes and otherwise fails closed rather than emitting wrong logits.