DeepSeek-V4 B2: block-FP8 / packed-FP4 weight-loading + expert-emission primitives (DRAFT) - #602
Merged
Merged
Conversation
…on primitives Real DeepSeek-V4-flash routed experts are FP4-packed in int8 (E2M1, block-32, UE8M0 micro-scales) and its projections are block-FP8 (E4M3 weight + 2D UE8M0 [128,128] block scales). QuantizationConfig.from_transformers treated quant_method='fp8' as ordinary per-tensor fp8 and returned None, so the model built bf16 and rejected the packed [2048,2048] expert against the logical [2048,4096] initializer with a confusing "Weight shape mismatch". Add a clean, breaking quantized-tensor descriptor + load contract (mobius.integrations._block_quant), classified by tensor properties (never by model name): - BlockQuantScheme: property parser over quantization_config + expert_dtype. - QuantizedTensorDescriptor: logical + packed shape, qtype, block geometry, scale name/dtype/shape/layout, byte counts, routed/shared role. - classify_tensor / validate_descriptor: distinguish ORDINARY, BLOCK_FP8, FP4_PACKED, UNSUPPORTED; logical-vs-packed shape, scale pairing, wrong/ missing/orphan scale all fail closed. No dequantization, no copy-to-float. - read_raw_tensor_bytes / LazyRawTensor: byte-exact, header-only, bounded lazy per-shard loading (one tensor resident). - stack_expert_bank / PackedExpertBank: byte-exact expert-major bank packing (reusable lowering primitive; ragged banks are a hard error). - runtime_representation_gap / plan_routed_expert_bank: the emission gate. nxrt's BlockFormat accepts only interleaved llama.cpp block_mxfp4 + iq*; it has no block-FP8 format and no planar-FP4 bank layout, so both families typed-reject with BlockQuantExportError naming the exact ABI gap instead of emitting an unrunnable node. No dense fallback. Route block-scaled fp8 / fp4-expert checkpoints through the typed blocker in QuantizationConfig.from_transformers (property-guarded: per-tensor fp8 without weight_block_size still returns None). This moves the real-checkpoint failure from a shape-mismatch to a precise, actionable BlockQuantExportError. Tests use the real quantization_config + a measured slice of the checkpoint index metadata plus tiny synthetic packed safetensors (real I8 / F8_E4M3 / F8_E8M0 dtypes): classification, logical-vs-packed validation, byte preservation, scale pairing (missing/duplicate/orphan/wrong), shared vs routed, bounded lazy load, byte-exact bank stacking, and the emission typed-reject. An opt-in suite exercises the mounted checkpoint headers directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Performance Comparison
|
Member
Author
APPROVE PRODUCER SLICEIndependent, read-only review at exact HEAD Verified against the real DeepSeek-V4-flash checkpointThe checkpoint is mounted on disk5 and the opt-in
Gates — verifiedProperty-based descriptor (never by model name).
Logical-vs-packed, scale pairing, block geometry, byte preservation.
Bounded lazy load, no float work.
Emission gate: emit only if the runtime ABI really supports the layout, else typed-reject.
Routing change is property-guarded and non-regressing.
Scope honesty.
Independent test/lint runs (isolated worktree,
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds property-based block-FP8 and packed-FP4 loading, validation, expert-bank packing, and typed runtime rejection.
Changes:
- Adds quantization descriptors and validation.
- Adds byte-preserving safetensors loading and expert-bank stacking.
- Updates transformer quantization parsing and adds comprehensive tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Review findings |
|---|---|
src/mobius/integrations/_block_quant.py |
Four moderate findings: validate unsupported descriptors, reject truncated reads, validate dtype and byte counts, and verify expert payload sizes. |
src/mobius/integrations/_block_quant_test.py |
No final comments. |
src/mobius/_configs/_quantization.py |
One moderate finding: classify top-level expert_dtype before the early qc is None return. |
Suppressed comments (1)
src/mobius/integrations/_block_quant_test.py:77
- The opt-in real-checkpoint tests are tied to a developer-specific absolute path, so they silently skip on every runner whose mount differs and cannot provide the advertised real-header coverage. Make the checkpoint root configurable (for example via an environment variable) while retaining a missing-path skip for default test runs.
REAL_CHECKPOINT = pathlib.Path(
"/datadisks/disk5/justinchu/onnx-genai-models/deepseek-v4-flash/checkpoint"
)
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+87
to
+90
| from mobius.integrations._block_quant import BlockQuantScheme | ||
|
|
||
| scheme = BlockQuantScheme.from_quantization_config( | ||
| qc, expert_dtype=getattr(hf_config, "expert_dtype", None) |
Comment on lines
+630
to
+631
| if validate and desc.kind is not QuantKind.UNSUPPORTED: | ||
| validate_descriptor(desc) |
Comment on lines
+175
to
+177
| with open(path, "rb") as f: | ||
| f.seek(start) | ||
| return f.read(end - start) |
Comment on lines
+504
to
+512
| # Both quantized kinds require a paired scale. | ||
| if desc.scale_shape is None or desc.scale_dtype is None: | ||
| raise BlockQuantValidationError( | ||
| f"{desc.name}: {desc.kind.value} tensor has no paired scale" | ||
| ) | ||
| if desc.scale_dtype != "F8_E8M0": | ||
| raise BlockQuantValidationError( | ||
| f"{desc.name}: expected UE8M0 (F8_E8M0) scale, got {desc.scale_dtype}" | ||
| ) |
Comment on lines
+680
to
+685
| n0 = len(per_expert_bytes[0]) | ||
| for i, b in enumerate(per_expert_bytes): | ||
| if len(b) != n0: | ||
| raise BlockQuantValidationError( | ||
| f"ragged expert bank: expert 0 has {n0} bytes but expert {i} has {len(b)}" | ||
| ) |
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
…able export on nxrt runtime capability Rebased #593 (native pkg.nxrt::CompressedSparseAttention export, ratio-4 CSA + ratio-128 HCA) onto Mobius main now that #578 (6992181) and #602 (d16cd4a, mobius.integrations._block_quant block-FP8 / packed-FP4 loading contract) are merged. Consumes the merged _block_quant API directly; the temporary stacked duplicate-parse branch is retired. #602 rejects the block-scaled-FP8 + packed-FP4 DeepSeek-V4-Flash checkpoint at config resolution (the INT4/per-tensor path cannot load it). That is correct for a normal export, but it also blocks a native-CSA export before any graph exists. Split the behavior into the two stages the directive requires: * Non-native (default) export keeps #602's early, loud config-resolution reject -- unchanged, its tests still pass. * native_csa opts into DEFERRING that reject: ArchitectureConfig.from_transformers catches the typed BlockQuantExportError, records the parsed block_quant_scheme on the config, and lets graph construction PROGRESS past the former generic "Weight shape mismatch" (build_from_module emits the CSA nodes + compressed state IO). The runnable FULL EXPORT then fails closed at a runtime-capability gate (assert_native_runtime_supports_block_quant, enforced at the top of DeepSeekV4 preprocess_weights, before apply_weights) while nxrt cannot execute block-FP8 / planar-FP4 weights. The gap string is sourced from _block_quant.runtime_representation_gap, so the gate tracks the real nxrt format strings and opens automatically when the native runtime slice lands -- no change here. Never a silent dense fallback, never partial-native. No BC shims: ArchitectureConfig gains a block_quant_scheme field; deepseek_v4.py adds no FP8/FP4 weight parsing (delegates to _block_quant). No performance claim -- ratio-4 sparse execution and >=16-decode remain blocked on the unmerged native block-FP8 / planar-FP4 runtime. Tests (deepseek_v4_flash_test.py, 45 pass): non-native fail-closed at config resolution; fp4-experts-alone owned; native_csa defers (scheme recorded, quant None); full export typed-rejects at the runtime-capability gate; graph construction progresses past block-quant (CSA node built); preprocess_weights enforces the gate; property-gated not a blanket V4/native_csa refusal; per-tensor fp8 not over-owned. #602 (_block_quant_test) + config (_base_test/_extractors_test) suites unchanged (69 pass). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 25, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
## Summary - clean up still-valid typing, documentation, test robustness, error-message, maintainability, and performance findings left by the GGUF PR stack - preserve the behavioral fixes merged in #625-#632 - hash reused multi-GB GGUF sources once, at the final pre-publication integrity gate, while retaining cheap identity checks around staging Exact base: `2db9d33debdc254d879a51b14434c9a81c230f4f` Exact head: `4541ed2bc9d2ab4227484510b4a85b2d9113eb25` ## Reconstructed original 17-item low-priority tranche The persisted audit retained only the totals, so this list was reconstructed from the live threads and current source. All but the already-fixed #596 comment are addressed in this PR. | PR | Comment | Disposition | |---|---:|---| | #550 | 3837144656 | Implemented: correct tuple return annotation | | #552 | 3837389183 | Implemented: multichannel waveform shape docs | | #559 | 3837716261 | Implemented: name-based cache assertions | | #573 | 3854308350 | Implemented: one final GGUF hash, with integrity regression coverage | | #574 | 3854345521 | Implemented: `TensorRole \| None` typing | | #574 | 3854345597 | Implemented: removed obsolete verdict filtering | | #577 | 3854472859 | Implemented: documented SSM sequence length | | #578 | 3843112613 | Implemented: documented F64 passthrough | | #579 | 3854518332 | Implemented: generalized fused-projection error | | #580 | 3854576300 | Implemented: removed brittle node counts | | #583 | 3854681386 | Implemented: documented conditional draft outputs | | #587 | 3854816872 | Implemented: corrected MTP output contract docs | | #596 | 3846110059 | Already fixed on base: unambiguous GQA bias comment | | #600 | 3855174281 | Implemented: metadata-count-only MTP error | | #607 | 3855776048 | Implemented: stable route-field assertions | | #607 | 3855776086 | Implemented: public tensor iterator | | #609 | 3856486136 | Implemented: fail-closed LM-head comment | ## Current unresolved-thread disposition This covers all 48 Copilot threads returned by the reproducible #600-#630 query. The one human #623 thread is excluded. | PR | Comment | Current-main disposition and evidence | |---|---:|---| | #600 | 3855174177 | Already fixed by #629: package cycle and reserved-sidecar validation | | #600 | 3855174238 | Already fixed by #629: explicit MTP sidecar naming/loading | | #600 | 3855174281 | Implemented here: error no longer invents an observed block count | | #602 | 3848155961 | Outside exact stack; already fixed: top-level `expert_dtype` is classified before early return | | #602 | 3848155983 | Outside exact stack; still-valid behavioral block-quant validation, unchanged | | #602 | 3848156000 | Outside exact stack; still-valid truncated-read behavioral finding, unchanged | | #602 | 3848156022 | Outside exact stack; still-valid descriptor byte/dtype validation, unchanged | | #602 | 3848156040 | Outside exact stack; still-valid expert-bank payload validation, unchanged | | #603 | 3855249765 | Already fixed by #630: runtime preflight preserves shard sets | | #603 | 3855249840 | Already fixed by #630: success output follows durable runtime publication | | #604 | 3855343082 | Implemented here: graph-only MTP persistence distinguished from runtime rejection | | #604 | 3855343131 | Already fixed by #630: runtime success messages are atomic | | #607 | 3855776001 | Implemented here: missing generation golden skips before provenance read | | #607 | 3855776048 | Implemented here: only stable route fields are asserted | | #607 | 3855776086 | Implemented here: tensor count uses `tensor_items_raw()` | | #608 | 3856079371 | Still-valid behavioral cache-symlink containment finding; unchanged | | #608 | 3856079415 | Still-valid behavioral lowercase-digest validation finding; unchanged | | #609 | 3856486136 | Implemented here: comment matches value-preserving policy | | #610 | 3855541683 | Already fixed by #628: Falcon bias precedence is explicit | | #610 | 3855541761 | Already fixed by #628: CTRL tiny config exercises projection biases | | #611 | 3856595840 | Implemented here: runtime test resolves the distribution providing the module | | #612 | 3855677383 | Already fixed by #625: supported-version endianness detection | | #612 | 3855677427 | Implemented here: shared `INT64_MAX` sentinel | | #612 | 3855677460 | Implemented here: shared PLaMo2 width inference | | #612 | 3855677486 | Implemented here: accepted PLaMo2 activation spellings are explicit | | #613 | 3855845678 | Implemented here: canonical issue URL | | #613 | 3855845757 | Implemented here: Mamba-1 function-registration docs | | #613 | 3855845806 | Implemented here: test expects the canonical issue URL | | #614 | 3855988545 | Implemented here: removed stale Nemotron-H divergence comments | | #614 | 3855988597 | Already fixed by #628: zero-head geometry raises actionable `ValueError` | | #615 | 3856082290 | Already fixed by #626: dense GraniteHybrid bias closure | | #618 | 3856729330 | Implemented here: required routes filter ORT GenAI evidence | | #618 | 3856729409 | Implemented here: env-selected runtime version is authoritative | | #618 | 3856729490 | Stale/N/A: PR-description-only matrix claim; repository workflow claims one pinned version | | #618 | 3856729563 | Implemented here: schema tail restored to normal indentation | | #619 | 3856342484 | Already fixed on base: Kimi Linear uses `/issues/605` | | #619 | 3856342532 | Already fixed by #628: config rejects convolution kernels below 2 | | #619 | 3856342580 | Already fixed by #628: GGUF contract rejects convolution kernels below 2 | | #620 | 3855717041 | Already fixed by #627: tied LM-head-only checkpoints are retained | | #621 | 3856722324 | Already fixed by #628: Kimi-K3 required metadata is complete | | #623 | 3855931210 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939302 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939358 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939394 | N/A to current main: comment belongs to open, unmerged #623 | | #624 | 3856777152 | Implemented here: runtime compatibility reuses the emitted model type | | #625 | 3857049733 | Implemented here: unsupported header reports both endian candidates | | #629 | 3857313182 | Newer post-audit behavioral sidecar-symlink cleanup finding; unchanged | | #629 | 3857313251 | Newer post-audit cross-platform path-safety finding; unchanged | ## Validation - affected GGUF/package/ORT GenAI/model/schema tests: 1,040 passed - broad non-integration suite: 7,851 passed, 56 skipped, 1 subtest passed - generated GGUF docs checks: 7 passed - initialized `lintrunner`; full lint/format passed - GPT-5.6 Sol medium review: one integrity finding fixed; re-review found no significant issues Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DeepSeek-V4 real-checkpoint blocker B2: generic block-FP8 / packed-FP4 weight-loading + expert-emission primitives, classified by tensor properties (never by model name). Separate artifact from PR #591 (native BQMoE fusion) — touches none of its files, and does not touch
deepseek_v4.py(Deckard/CSA #593).Root cause
QuantizationConfig.from_transformerstreatedquant_method='fp8'as ordinary per-tensor float8 and returnedNone, so the model built bf16Linearinitializers. The real experts are FP4-packed in int8 (E2M1, block-32, UE8M0 micro-scales), so a packed[2048,2048]expert was compared to its logical[2048,4096]initializer → confusingWeight shape mismatch. Projections are block-FP8 (E4M3 weight + 2D UE8M0[128,128]block scales).What this adds (
mobius.integrations._block_quant)BlockQuantScheme— property parser overquantization_config+ top-levelexpert_dtype. Per-tensor fp8 (noweight_block_size) is not owned.QuantizedTensorDescriptor— clean, breaking contract: logical and packed shape, weight qtype, block geometry, scale name/dtype/shape/layout, exact byte counts, routed/shared role.classify_tensor/validate_descriptor— distinguishORDINARY,BLOCK_FP8,FP4_PACKED,UNSUPPORTED. Logical-vs-packed shape, scale pairing, and wrong/missing/orphan scales fail closed. No dequantization, no copy-to-float.read_raw_tensor_bytes/LazyRawTensor— byte-exact, header-only, bounded lazy per-shard loading (one tensor resident).stack_expert_bank/PackedExpertBank— byte-exact expert-major bank packing (reusable lowering primitive; ragged banks are a hard error).runtime_representation_gap/plan_routed_expert_bank— the emission gate.ABI verdict (why it typed-rejects, no fake fusion)
onnx-genai
nxrtCPU kernels (block_quantized_{matmul,moe}.rs) accept only the interleaved llama.cppblock_mxfp4layout (QK=32, 17 bytes/block) and theiq*GGUF formats:block_fp8BlockFormat + no E4M3×2D-UE8M0 dequant path → typed reject.plan_routed_expert_bankraisesBlockQuantExportErrornaming the exact gap rather than emitting an unrunnable node. No dense fallback, no env-flag emission.from_transformerschange (property-guarded)Block-scaled fp8 / fp4-expert checkpoints now surface a precise typed blocker instead of
None → shape-mismatch. Ordinary per-tensor fp8 still returnsNone; GPTQ/AWQ/GGUF/ModelOpt paths unchanged.Preflight (B2 moved)
ArchitectureConfig.from_file(<real checkpoint>)now raisesBlockQuantExportErrorwith the exact layout + ABI gap — the failure moved from a confusing shape-mismatch to an actionable typed reject.Tests
Real
quantization_config+ a measured slice of the checkpoint index metadata + tiny synthetic packed safetensors (realI8/F8_E4M3/F8_E8M0dtypes): classification, logical-vs-packed validation, byte preservation, scale pairing (missing/duplicate/orphan/wrong), shared-vs-routed, bounded lazy load, byte-exact bank stacking, emission typed-reject. An opt-in suite exercises the mounted checkpoint headers directly.ruff check+ruff formatclean.Deckard #593 handoff (stable public API)
from mobius.integrations._block_quant import (BlockQuantScheme, QuantizedTensorDescriptor, QuantKind, classify_tensor, validate_descriptor, build_descriptors, pair_weight_scales, read_raw_tensor_bytes, LazyRawTensor, stack_expert_bank, PackedExpertBank, plan_routed_expert_bank, runtime_representation_gap, BlockQuantError, BlockQuantValidationError, BlockQuantExportError). No edits todeepseek_v4.py.Remaining typed blockers (not this PR)
No A100 benchmark (no runnable shape-faithful path yet). Draft — do not merge.
🤖 Flagged for squad review (needs review): touches the shared
QuantizationConfig.from_transformersconfig seam.