diff --git a/docs/api/build_from_gguf.md b/docs/api/build_from_gguf.md index a8b8a66e2..bdd4f79a7 100644 --- a/docs/api/build_from_gguf.md +++ b/docs/api/build_from_gguf.md @@ -181,7 +181,7 @@ def build_from_gguf( | `gguf_path` | `str \| Path` | (required) | Local `.gguf` path or `owner/repo:filename.gguf` Hub reference. | | `task` | `str \| None` | `None` | Override the model task (e.g. `"text-generation"`). When `None`, the task is auto-detected from the model type. | | `dtype` | `str \| None` | `None` | Override model dtype (e.g. `"f16"`). When `None`, defaults to float32. | -| `keep_quantized` | `bool` | `True` | Preserve quantization when present. Supported affine blocks are repacked as `MatMulNBits`; in text-only builds, supported native IQ/MXFP4 projection blocks retain their bytes. Multimodal and mixed presets may require dequantization/requantization. Set to `False` to dequantize all weights. | +| `keep_quantized` | `bool` | `True` | Preserve quantization when present. Supported affine blocks are repacked as `MatMulNBits`; in text-only builds, supported native IQ/MXFP4 projection blocks retain their bytes. Projection tensors that would require lossy requantization are rejected. Set to `False` to dequantize all weights. | | `execution_provider` | `str` | `"default"` | Target EP for EP-aware graph optimization. | | `mmproj` | `str \| Path \| None` | `None` | Optional companion multimodal-projector GGUF. | | `static_cache` | `bool` | `False` | Build a fixed-width KV cache when the architecture supports it. | @@ -267,9 +267,10 @@ mobius build-gguf draft.gguf --target-config target/ --output output/draft/ 1. Reads GGUF metadata to detect architecture and config 2. Maps GGUF tensor names to HuggingFace weight names -3. Preserves supported quantized tensors by default, using repacking, - text-only native-block retention, or dequantize/requantize according to the - source qtype and build path; `keep_quantized=False` dequantizes every tensor +3. Preserves supported quantized tensors by default using value-preserving + repacking or text-only native-block retention; any projection requiring + lossy requantization fails closed, while `keep_quantized=False` explicitly + dequantizes every tensor 4. Applies architecture-specific tensor processors (e.g. Q/K permute) 5. Builds the ONNX graph using the same pipeline as `build()` 6. Runs `preprocess_weights()` (HF → ONNX name mapping) @@ -277,10 +278,12 @@ mobius build-gguf draft.gguf --target-config target/ --output output/draft/ F32-, F16-, and BF16-only GGUFs use the normal float import path even though `keep_quantized=True` is the default: there is no quantization to preserve. -Quantized GGUFs whose qtypes have no trustworthy decoder or compatible runtime -kernel (currently `Q2_0`) fail with an actionable error. Decoder-backed formats -such as `Q5_K` use the explicit dequantize/requantize route; they are not -silently treated as preserved source quantization. +Quantized GGUFs containing only qtypes with no supported preservation target +(for example, pure Q5_K weights) fail with an actionable error rather +than silently becoming float. Pass `keep_quantized=False` to request that float +conversion explicitly. +Mixed presets such as Q4_K_M also fail when their projection inventory includes +Q5/Q6/Q8 tensors that cannot share one lossless MatMulNBits contract. ### Native blocks, conversion, and source-file reuse diff --git a/docs/cli_reference.md b/docs/cli_reference.md index 7561c4bbc..c93de71fa 100644 --- a/docs/cli_reference.md +++ b/docs/cli_reference.md @@ -323,9 +323,10 @@ mobius build --model Qwen/Qwen2.5-0.5B --output output_dir/ \ Build an ONNX model from a GGUF file (e.g. from llama.cpp). This is an explicit opt-in import path; `mobius build` does not auto-discover or select GGUF files. -Supported GGUF quantization is preserved by default. This can involve -byte-preserving native blocks in text-only builds, affine repacking, or -dequantize/requantize for multimodal and mixed source qtypes. +Supported GGUF quantization is preserved by default through byte-preserving +native blocks or value-preserving affine repacking. Mixed source qtypes that +would require lossy dequantization/requantization, including Q4_K_M presets, +fail closed and require `--dequantize`. > **Note**: Requires the optional `gguf` package: `pip install mobius-onnx[gguf]` @@ -377,6 +378,8 @@ contain no quantization to preserve. Quantized files containing only qtypes with no supported preservation target (for example, pure Q5_K weights) fail instead of silently becoming float. Re-run with `--dequantize` to request explicit float conversion. +The same rule applies when only some projection tensors are incompatible with +the selected affine target; Mobius does not silently requantize those tensors. Encoder-only BERT and ModernBERT GGUF backbones auto-select `feature-extraction` and output `last_hidden_state`; they do not produce logits diff --git a/docs/model-catalog.md b/docs/model-catalog.md index d3c29cfd1..23afd97a2 100644 --- a/docs/model-catalog.md +++ b/docs/model-catalog.md @@ -267,11 +267,10 @@ All decoder-only LLMs and MoE models support quantized weight loading: | **AWQ** | HuggingFace (e.g. `-AWQ` suffix models) | `build("TheBloke/Llama-2-7B-AWQ")` | | **GGUF** | Local `.gguf` files | `build_from_gguf("model.gguf")` | -GGUF import preserves supported quantization by default using affine repacking -or, for supported text-only inputs, native quantized ops. Multimodal and mixed -source qtypes can require dequantization/requantization, so this does not imply -byte preservation for every tensor. Use `--dequantize` for an explicitly float -model. +GGUF import preserves supported quantization by default using value-preserving +affine repacking or, for supported text-only inputs, native quantized ops. +Projection tensors that would require lossy dequantization/requantization fail +closed. Use `--dequantize` for an explicitly float model. Encoder GGUF imports for `bert` and `modern-bert` select `feature-extraction` and expose token-level `last_hidden_state` only. Pooling, diff --git a/src/mobius/functions/matmul_nbits_test.py b/src/mobius/functions/matmul_nbits_test.py index 83bc23e3a..bd8ac3d4c 100644 --- a/src/mobius/functions/matmul_nbits_test.py +++ b/src/mobius/functions/matmul_nbits_test.py @@ -46,7 +46,12 @@ def _const(name: str, arr: np.ndarray) -> ir.Value: def _build_matmulnbits_model( - n_out: int, k_in: int, block: int, seed: int = 0 + n_out: int, + k_in: int, + block: int, + seed: int = 0, + *, + accuracy_level: int | None = None, ) -> tuple[ir.Model, dict, np.ndarray, np.ndarray, np.ndarray]: """Build a single-node MatMulNBits model with random 4-bit weights. @@ -67,14 +72,15 @@ def _build_matmulnbits_model( sc_c = _const("scales", scales) zp_c = _const("zero_points", zero_points) y = ir.Value(name="Y") + attributes = {"K": k_in, "N": n_out, "bits": 4, "block_size": block} + if accuracy_level is not None: + attributes["accuracy_level"] = accuracy_level node = ir.Node( "com.microsoft", "MatMulNBits", inputs=[a_val, b_c, sc_c, zp_c], outputs=[y], - attributes=ir.convenience.convert_attributes( - {"K": k_in, "N": n_out, "bits": 4, "block_size": block} - ), + attributes=ir.convenience.convert_attributes(attributes), ) graph = ir.Graph( inputs=[a_val], @@ -118,6 +124,30 @@ def test_inline_matches_native_op(self): np.testing.assert_allclose(got, reference, rtol=0, atol=0) + def test_accuracy_four_matches_explicit_asymmetric_dequantization(self): + """The CPU kernel consumes Mobius's packed weights and zero points correctly.""" + model, feeds, packed, scales, packed_zero_points = _build_matmulnbits_model( + n_out=8, + k_in=64, + block=32, + seed=11, + accuracy_level=4, + ) + got = _run(model, feeds) + + quants = np.empty((8, 2, 32), dtype=np.float32) + quants[..., 0::2] = packed & 0x0F + quants[..., 1::2] = packed >> 4 + zero_points = np.empty((8, 2), dtype=np.float32) + zero_points[..., 0::2] = packed_zero_points & 0x0F + zero_points[..., 1::2] = packed_zero_points >> 4 + weights = ((quants - zero_points[..., None]) * scales[..., None]).reshape(8, 64) + expected = feeds["A"] @ weights.T + + error = got - expected + assert float(np.max(np.abs(error))) < 0.1 + assert float(np.linalg.norm(error) / np.linalg.norm(expected)) < 0.02 + def test_inline_matches_native_op_odd_blocks(self): """Odd n_blocks exercises the zero-point nibble slice (ceil(nb/2)).""" # K=96, block=32 → nb=3 (odd) diff --git a/src/mobius/integrations/gguf/_builder.py b/src/mobius/integrations/gguf/_builder.py index 91ef6c3a4..ca335dd18 100644 --- a/src/mobius/integrations/gguf/_builder.py +++ b/src/mobius/integrations/gguf/_builder.py @@ -6,12 +6,11 @@ Converts ``.gguf`` model files to ONNX using the standard build pipeline. Quantized preservation is the default: affine linear-layer weights are repacked into MatMulNBits format and compatible token embeddings into -GatherBlockQuantized format. For text-only builds, operator-native -IQ/MXFP4 projection blocks are preserved for BlockQuantizedMatMul. Multimodal -text backbones and mixed presets such as Q4_K_M are normalized to one affine -layout, so not every source tensor is byte-preserved. Other tensors are -dequantized. Set ``keep_quantized=False`` to dequantize all weights to float -explicitly. +GatherBlockQuantized format. For text-only builds, runtime-supported native +IQ/MXFP4 projection blocks are preserved for BlockQuantizedMatMul. A requested +preservation route fails closed when a projection would need lossy +dequantization/requantization, including mixed Q4_K_M presets. Set +``keep_quantized=False`` to request a fully float import explicitly. """ from __future__ import annotations @@ -3082,7 +3081,7 @@ def build_from_gguf( By default, supported affine tensors are repacked into MatMulNBits format. For text-only builds, operator-native IQ/MXFP4 projection blocks are retained byte-for-byte for BlockQuantizedMatMul. Multimodal text - backbones normalize quantized projections to a common affine layout. GGUFs + backbones require a lossless route for every quantized projection. GGUFs containing only F32, F16, or BF16 weights use the float path because there is no quantization to preserve. Quantized files with no supported preservation target raise an actionable @@ -3102,10 +3101,10 @@ def build_from_gguf( defaults to float32. keep_quantized: Preserve quantization when quantized tensors are present. This is the default. Supported affine blocks are repacked, - text-only operator-native IQ/MXFP4 projection blocks - retain their bytes, and mixed or multimodal source types can be - normalized to a common affine layout. Set to ``False`` to - dequantize all weights. + text-only runtime-supported native IQ/MXFP4 projection blocks + retain their bytes, and any projection requiring lossy + requantization is rejected. Set to ``False`` to dequantize all + weights. execution_provider: Target execution provider for EP-aware optimisations (e.g. ``"cpu"`` to apply the GroupQueryAttention rewrite). Defaults to ``"default"`` @@ -3261,12 +3260,6 @@ def build_from_gguf( ) if keep_quantized and not preserve_quantization: logger.info("GGUF contains no mapped quantized weights; using the float import path") - _reject_unsupported_quantization_preservation( - gguf_model, - gguf_arch, - preserve_quantization=preserve_quantization, - ) - # 2. Extract config from GGUF metadata config = gguf_to_config(gguf_model) spec = get_arch_spec(gguf_arch) @@ -3540,7 +3533,17 @@ def build_from_gguf( # 5. Build ONNX graph module = module_class(config) + float_linear_dequantization_types = _float_linear_dequantization_types( + module, + gguf_arch, + ) if preserve_quantization: + _reject_unsupported_quantization_preservation( + gguf_model, + gguf_arch, + preserve_quantization=True, + dequantize_float_linear_types=float_linear_dequantization_types, + ) _replace_native_block_linears(module, gguf_model, gguf_arch) # The sparse-MoE honesty gate runs post-export on the final graph state # (see step 9b): routed native-block experts are first collapsed into a @@ -3566,10 +3569,7 @@ def build_from_gguf( module, config, reuse_candidates=reuse_candidates_by_id, - dequantize_float_linear_types=_float_linear_dequantization_types( - module, - gguf_arch, - ), + dequantize_float_linear_types=float_linear_dequantization_types, ) else: state_dict = _load_dequantized_state_dict( @@ -4336,25 +4336,52 @@ def _expand(perm, stride): def _has_quantized_weights(gguf_model, gguf_arch: str) -> bool: - """Return whether a GGUF has mapped weights with a quantized tensor type.""" + """Return whether a GGUF has stored weights with a quantized tensor type.""" from mobius.integrations.gguf._quant_registry import float_storage_type_ids - from mobius.integrations.gguf._tensor_mapping import map_gguf_to_hf_names float_type_ids = float_storage_type_ids() for name, _raw, qtype, _shape in gguf_model.tensor_items_raw(): - hf_name = map_gguf_to_hf_names(name, gguf_arch) type_id = getattr(qtype, "value", qtype) - if hf_name is not None and name.endswith(".weight") and type_id not in float_type_ids: + if name.endswith(".weight") and type_id not in float_type_ids: return True return False +def _uses_explicit_float_route( + gguf_arch: str, + tensor_name: str, +) -> bool: + """Return whether a quantized source weight is intentionally loaded as float.""" + if gguf_arch in {"bert", "modern-bert"} and tensor_name in { + "token_embd.weight", + "token_embd_norm.weight", + "token_types.weight", + "position_embd.weight", + }: + return True + if tensor_name.endswith(("_norm.weight", ".norm.weight")): + return True + return gguf_arch == "jamba" and tensor_name.endswith( + ( + "ssm_in.weight", + "ssm_out.weight", + "ssm_x.weight", + "ssm_dt.weight", + "ssm_conv1d.weight", + ) + ) + + def _reject_unsupported_quantization_preservation( gguf_model, gguf_arch: str, *, preserve_quantization: bool, + allow_native_blocks: bool = True, + allow_quantized_embeddings: bool = True, + allow_quantized_lm_head: bool = True, + dequantize_float_linear_types: Mapping[str, Collection[str]] | None = None, ) -> None: """Reject architectures that cannot preserve all compatible quantized weights. @@ -4383,6 +4410,117 @@ def _reject_unsupported_quantization_preservation( f"Quantization-preserving {gguf_arch} import is unsupported because {reason}. " "Use keep_quantized=False (or --dequantize) for a float import." ) + + from mobius.integrations.gguf._quant_registry import ( + float_storage_type_ids, + get_quant_spec, + lossless_preservation_type_names, + ) + from mobius.integrations.gguf._tensor_mapping import map_gguf_to_hf_names + + float_type_ids = float_storage_type_ids() + lossless_types = lossless_preservation_type_names() + affine_targets: dict[tuple[int, int], tuple[str, str]] = {} + metadata = getattr(gguf_model, "metadata", {}) + block_count = int(metadata.get(f"{gguf_arch}.block_count", 0)) + mtp_count = int(metadata.get(f"{gguf_arch}.nextn_predict_layers", 0)) + mtp_blocks = set(range(max(0, block_count - mtp_count), block_count)) + for tensor_name, _raw, qtype, _shape in gguf_model.tensor_items_raw(): + type_id = getattr(qtype, "value", qtype) + type_name = getattr(qtype, "name", str(qtype)) + if not tensor_name.endswith(".weight") or type_id in float_type_ids: + continue + hf_name = map_gguf_to_hf_names(tensor_name, gguf_arch) + module_stem = ( + hf_name[: -len(".weight")] + if hf_name is not None and hf_name.endswith(".weight") + else None + ) + if gguf_arch == "bert" and module_stem is not None and module_stem.startswith("bert."): + module_stem = module_stem[len("bert.") :] + explicitly_dequantized = ( + dequantize_float_linear_types is not None + and module_stem in dequantize_float_linear_types + and type_name in dequantize_float_linear_types[module_stem] + ) + if explicitly_dequantized or _uses_explicit_float_route(gguf_arch, tensor_name): + continue + if tensor_name.endswith(".ffn_gate_up_exps.weight"): + raise ValueError( + "Quantization-preserving GGUF import cannot split packed fused expert " + f"tensor {tensor_name} ({type_name}) into separate gate/up graph " + "targets without changing its stored representation. Use " + "keep_quantized=False (API) or --dequantize (CLI) for explicit " + "float import." + ) + if not allow_quantized_embeddings and tensor_name in { + "token_embd.weight", + "shared.weight", + }: + raise ValueError( + "Quantization-preserving GGUF import cannot retain packed embedding " + f"{tensor_name} ({type_name}) in this graph. Use keep_quantized=False " + "(API) or --dequantize (CLI) for explicit float import." + ) + if not allow_quantized_lm_head and tensor_name == "output.weight": + raise ValueError( + "Quantization-preserving GGUF import cannot retain packed LM head " + f"{tensor_name} ({type_name}) in this graph. Use keep_quantized=False " + "(API) or --dequantize (CLI) for explicit float import." + ) + if type_name not in lossless_types: + raise ValueError( + "Quantization-preserving GGUF import would change the dequantized " + f"values of {tensor_name} ({type_name}). The current ORT " + "MatMulNBits route cannot represent this source block format " + "losslessly; mixed presets such as Q4_K_M must not be normalized " + "to a common 4-bit affine layout. Use keep_quantized=False (API) " + "or --dequantize (CLI) for explicit float import." + ) + spec = get_quant_spec(qtype) + block_match = re.match(r"blk\.(\d+)\.", tensor_name) + is_mtp_block = block_match is not None and int(block_match.group(1)) in mtp_blocks + if ( + spec is not None + and spec.native_preserve is not None + and (not allow_native_blocks or ".nextn." in tensor_name or is_mtp_block) + ): + raise ValueError( + "Quantization-preserving GGUF import cannot retain native block " + f"format {type_name} for {tensor_name} in this graph. Use " + "keep_quantized=False (API) or --dequantize (CLI) for explicit " + "float import." + ) + if ( + spec is not None + and spec.native_preserve is not None + and tensor_name + in { + "token_embd.weight", + "shared.weight", + } + ): + raise ValueError( + "Quantization-preserving GGUF import cannot retain native block " + f"format {type_name} for embedding tensor {tensor_name}; " + "GatherBlockQuantized does not consume that layout. Use " + "keep_quantized=False (API) or --dequantize (CLI) for explicit " + "float import." + ) + if spec is not None and spec.affine_repack is not None: + affine_targets.setdefault(spec.affine_repack.as_params(), (tensor_name, type_name)) + + if len(affine_targets) > 1: + details = ", ".join( + f"{name} ({qtype}: {bits}-bit/block-{block_size})" + for (bits, block_size), (name, qtype) in sorted(affine_targets.items()) + ) + raise ValueError( + "Quantization-preserving GGUF import requires one affine MatMulNBits " + f"contract across projection modules, but found incompatible targets: {details}. " + "Use keep_quantized=False (API) or --dequantize (CLI) for explicit float import." + ) + if gguf_arch not in {"dream", "llada-moe", "rnd1"}: return @@ -4415,7 +4553,6 @@ def _detect_quant_params(gguf_model, gguf_arch: str) -> tuple[int, int, bool]: from gguf import GGMLQuantizationType from mobius.integrations.gguf._quant_registry import ( - explicit_zero_point_type_names, float_storage_type_ids, get_quant_spec, ) @@ -4432,6 +4569,8 @@ def _detect_quant_params(gguf_model, gguf_arch: str) -> tuple[int, int, bool]: hf_name = map_gguf_to_hf_names(name, gguf_arch) if hf_name is None or not hf_name.endswith(".weight"): continue + if _uses_explicit_float_route(gguf_arch, name): + continue type_id = getattr(qtype, "value", qtype) if type_id not in float_type_ids: counts[qtype] += 1 @@ -4468,17 +4607,30 @@ def _detect_quant_params(gguf_model, gguf_arch: str) -> tuple[int, int, bool]: {qtype: count for qtype, count in counts.items() if _native_block_format(qtype)} ) if native_counts: - explicit_zero_point_types = explicit_zero_point_type_names() - can_omit_zero_points = not any( - getattr(qtype, "name", None) in explicit_zero_point_types + affine_specs = { + spec.affine_repack for qtype in counts if qtype not in native_counts - ) + if (spec := get_quant_spec(qtype)) is not None and spec.affine_repack is not None + } + if len(affine_specs) > 1: + raise ValueError( + "Native-block GGUF contains incompatible affine projection targets; " + "use keep_quantized=False for explicit float import." + ) + if affine_specs: + target = next(iter(affine_specs)) + bits, block_size = target.as_params() + can_omit_zero_points = target.omit_zero_points + else: + bits, block_size, can_omit_zero_points = 4, 32, True logger.info( - "Native GGUF quant types present; using 4-bit/block-32 module " - "scaffolding for non-native quantized tensors", + "Native GGUF quant types present; using %d-bit/block-%d module " + "scaffolding for affine tensors", + bits, + block_size, ) - return 4, 32, can_omit_zero_points + return bits, block_size, can_omit_zero_points if any( spec is not None and spec.import_route is QuantImportRoute.DEQUANTIZE_REQUANTIZE @@ -4821,6 +4973,7 @@ def _load_quantized_state_dict( QuantizedLinear, ) from mobius.integrations.gguf._quant_registry import ( + float_storage_type_ids, get_quant_spec, quant_import_decision, ) @@ -4830,7 +4983,7 @@ def _load_quantized_state_dict( repack_dequantized_tensor, repack_gguf_tensor, ) - from mobius.integrations.gguf._spec import QuantImportRoute, TensorRole + from mobius.integrations.gguf._spec import QuantImportRoute, RepackExactness, TensorRole from mobius.integrations.gguf._tencent_q1_0 import ( is_tencent_q1_0_layout, parse_tencent_q1_0_tensor, @@ -4890,6 +5043,7 @@ def _load_quantized_state_dict( target_bits = config.quantization.bits target_block_size = config.quantization.group_size target_symmetric = config.quantization.sym + float_type_ids = float_storage_type_ids() for gguf_name, raw, qtype, np_shape in tqdm.tqdm( gguf_model.tensor_items_raw(), @@ -4973,11 +5127,14 @@ def _load_quantized_state_dict( else TensorRole.OUTPUT if hf_name == "lm_head.weight" else TensorRole.PROJECTION - if module_stem is not None - and ( - module_stem in quantized_stems - or module_stem in native_block_stems - or module_stem in float_linear_stems + if fused_projection_targets + or ( + module_stem is not None + and ( + module_stem in quantized_stems + or module_stem in native_block_stems + or module_stem in float_linear_stems + ) ) else TensorRole.NON_MATMUL ) @@ -4996,18 +5153,30 @@ def _load_quantized_state_dict( ) if explicitly_dequantized and quant_spec.dequantize is Support.SUPPORTED: route = QuantImportRoute.DEQUANTIZE_FLOAT - if is_kimi_reshaped_projection and route is QuantImportRoute.NATIVE_BYTES: + if is_kimi_reshaped_projection: if quant_spec.dequantize is not Support.SUPPORTED: raise ValueError( - f"Cannot reshape native {quant_spec.name} tensor {hf_name}: " + f"Cannot reshape quantized {quant_spec.name} tensor {hf_name}: " "the stored format has no supported dequantization route." ) route = QuantImportRoute.DEQUANTIZE_REQUANTIZE + if is_encoder_embedding and quant_spec.dequantize is Support.SUPPORTED: + route = QuantImportRoute.DEQUANTIZE_FLOAT if route is QuantImportRoute.REJECTED: raise ValueError( f"Cannot import GGUF tensor {gguf_name} mapped to {hf_name} " f"({quant_spec.name}, role={tensor_role.value}): {reason}" ) + if route is QuantImportRoute.DEQUANTIZE_REQUANTIZE or ( + _exactness is RepackExactness.LOSSY + and route is not QuantImportRoute.DEQUANTIZE_FLOAT + ): + raise ValueError( + "Quantization-preserving GGUF import would change the dequantized " + f"values of {gguf_name} ({quant_spec.name}). Use " + "keep_quantized=False (API) or --dequantize (CLI) for explicit " + "float import." + ) if route is QuantImportRoute.NATIVE_BYTES and not native_targets: raise ValueError( f"Cannot preserve native {quant_spec.name} bytes for {hf_name}: " @@ -5045,32 +5214,46 @@ def _load_quantized_state_dict( "quantized: the model graph does not expose MatMulNBits. Use " "keep_quantized=False for explicit float import." ) + if qtype_val in float_type_ids and ( + fused_projection_targets or affine_targets or should_repack + ): + raise ValueError( + "Quantization-preserving GGUF import would quantize float projection " + f"{gguf_name} ({getattr(qtype, 'name', qtype)}) to the graph's " + f"{target_bits}-bit/block-{target_block_size} MatMulNBits contract. " + "Use keep_quantized=False (API) or --dequantize (CLI) for an " + "explicit float import." + ) if fused_projection_targets: - values = gguf_model.dequantize_raw_tensor(raw, qtype, np_shape) + if route is not QuantImportRoute.AFFINE_REPACK or not can_repack(qtype_val): + raise ValueError( + "Quantization-preserving GGUF import cannot split fused projection " + f"{gguf_name} ({getattr(qtype, 'name', qtype)}) without changing " + "its dequantized values. Use keep_quantized=False (API) or " + "--dequantize (CLI) for explicit float import." + ) + repacked = repack_gguf_tensor(raw, qtype_val, tuple(int(dim) for dim in np_shape)) offset = 0 for target_stem in fused_projection_targets: n_out = quantized_output_sizes[target_stem] - target_values = values[offset : offset + n_out] - offset += n_out - repacked = repack_dequantized_tensor( - target_values, - bits=target_bits, - block_size=target_block_size, - symmetric=target_symmetric, + end = offset + n_out + state_dict[f"{target_stem}.weight"] = torch.from_numpy( + np.array(repacked.weight[offset:end], copy=True) + ) + state_dict[f"{target_stem}.scales"] = torch.from_numpy( + np.array(repacked.scales[offset:end], copy=True) ) - state_dict[f"{target_stem}.weight"] = torch.from_numpy(repacked.weight) - state_dict[f"{target_stem}.scales"] = torch.from_numpy(repacked.scales) if repacked.zero_points is not None: state_dict[f"{target_stem}.zero_points"] = torch.from_numpy( - repacked.zero_points + np.array(repacked.zero_points[offset:end], copy=True) ) + offset = end if offset != int(np_shape[0]): raise ValueError( f"Fused QKV tensor {hf_name!r} has {np_shape[0]} rows, " f"but Q/K/V targets require {offset}" ) n_repacked += len(fused_projection_targets) - n_requantized += 1 elif native_targets and native_spec is not None: n_out = int(np_shape[-2]) k_in = int(np_shape[-1]) @@ -5287,7 +5470,7 @@ def _load_quantized_state_dict( state_dict[f"{stem}.zero_points"] = zp n_repacked += 1 else: - # Dequantize to float + # Dequantize weights whose graph target is intentionally float. if qtype in ( GGMLQuantizationType.F32, GGMLQuantizationType.F16, diff --git a/src/mobius/integrations/gguf/_builder_test.py b/src/mobius/integrations/gguf/_builder_test.py index d2c1e8325..490863dae 100644 --- a/src/mobius/integrations/gguf/_builder_test.py +++ b/src/mobius/integrations/gguf/_builder_test.py @@ -2967,22 +2967,24 @@ def test_default_quantized_package_save_reload(self, q4_0_gguf: Path, tmp_path: is not None ) - def test_decoder_backed_qtype_build_save_reload( + def test_decoder_backed_qtype_requires_explicit_dequantization( self, q5_1_gguf: Path, tmp_path: Path ) -> None: - """A pure Q5_1 file uses the declared 4-bit requantization route.""" + """A pure Q5_1 file fails closed, while explicit float import round-trips.""" from mobius._model_package import ModelPackage from mobius.integrations.gguf import build_from_gguf + with pytest.raises( + ValueError, match="cannot represent this source block format losslessly" + ): + build_from_gguf(q5_1_gguf) + output_dir = tmp_path / "saved_q5_1" - package = build_from_gguf(q5_1_gguf) + package = build_from_gguf(q5_1_gguf, keep_quantized=False) package.save(str(output_dir), progress_bar=False) model = ModelPackage.load(str(output_dir))["model"] - quantized_nodes = [node for node in model.graph if node.op_type == "MatMulNBits"] - assert quantized_nodes - assert all(node.attributes["bits"].value == 4 for node in quantized_nodes) - assert all(node.attributes["block_size"].value == 32 for node in quantized_nodes) + assert all(node.op_type != "MatMulNBits" for node in model.graph) def test_float_only_default_uses_float_path(self, float_only_gguf: Path): """F32/BF16-only GGUFs do not fail when preservation is the default.""" @@ -3381,6 +3383,34 @@ def test_phimoe_fused_qkv_is_split_without_loss( ) assert any(f"model.layers.0.self_attn.{projection}.bias" in name for name in names) assert not any("qkv_proj" in name for name in names) + if projection_quantization == "q4_0": + from mobius.integrations.gguf._reader import GGUFModel + from mobius.integrations.gguf._repacker import repack_gguf_tensor + + gguf_model = GGUFModel(path) + raw, qtype, shape = next( + (raw, qtype, shape) + for name, raw, qtype, shape in gguf_model.tensor_items_raw() + if name == "blk.0.attn_qkv.weight" + ) + repacked = repack_gguf_tensor(raw, qtype.value, shape) + offset = 0 + for projection, rows in (("q_proj", 64), ("k_proj", 32), ("v_proj", 32)): + stem = f"model.layers.0.self_attn.{projection}" + end = offset + rows + np.testing.assert_array_equal( + model.graph.initializers[f"{stem}.weight"].const_value.numpy(), + repacked.weight[offset:end], + ) + np.testing.assert_array_equal( + model.graph.initializers[f"{stem}.scales"].const_value.numpy(), + repacked.scales[offset:end], + ) + np.testing.assert_array_equal( + model.graph.initializers[f"{stem}.zero_points"].const_value.numpy(), + repacked.zero_points[offset:end], + ) + offset = end @pytest.mark.parametrize("architecture", ["qwen3moe", "granitemoe"]) def test_tied_quantized_embedding_is_shared_with_output_head( @@ -3514,28 +3544,14 @@ def test_native_blocks_emit_block_quantized_matmul_and_preserve_bytes( imports = {opset.domain: opset.version for opset in proto.opset_import} assert imports["pkg.nxrt"] == 1 - def test_mixed_native_quantization_uses_q4_scaffold(self, mixed_native_q5_q8_gguf: Path): - """Native IQ tensors force Q5_1 fallback weights onto a Q4 scaffold.""" + def test_mixed_native_quantization_rejects_lossy_q5_fallback( + self, mixed_native_q5_q8_gguf: Path + ): + """Native IQ blocks do not make lossy Q5_1-to-Q4 conversion acceptable.""" from mobius.integrations.gguf import build_from_gguf - model = build_from_gguf(mixed_native_q5_q8_gguf, keep_quantized=True)["model"] - native_nodes = [node for node in model.graph if node.op_type == "BlockQuantizedMatMul"] - assert len(native_nodes) == 6 - - value_nodes = [ - node - for node in model.graph - if node.op_type == "MatMulNBits" - and node.inputs[1].name == "model.layers.0.self_attn.v_proj.weight" - ] - assert len(value_nodes) == 1 - assert value_nodes[0].attributes["bits"].value == 4 - assert value_nodes[0].attributes["block_size"].value == 32 - assert "model.layers.0.self_attn.v_proj.zero_points" in model.graph.initializers - - native_weight = model.graph.initializers["model.layers.0.self_attn.o_proj.weight"] - expected = np.arange(256 * 136, dtype=np.uint8).reshape(256, 1, 136) - np.testing.assert_array_equal(native_weight.const_value.numpy(), expected) + with pytest.raises(ValueError, match=r"attn_v\.weight \(Q5_1\)"): + build_from_gguf(mixed_native_q5_q8_gguf, keep_quantized=True) def test_quantized_embedding_uses_gatherblockquantized(self, q4_0_embedding_gguf: Path): """A quantized GGUF embedding remains packed in the ONNX graph.""" @@ -3575,17 +3591,17 @@ def test_gatherblockquantized_zero_point_dequantizes_q4_0(self, tmp_path: Path): wrong = _run_gather_block_quantized(tmp_path, zero_point=0x00).astype(np.float32) assert not np.allclose(wrong, expected) - def test_native_projection_abi_embedding_is_converted_to_gather( + def test_native_projection_abi_embedding_requires_explicit_dequantization( self, iq4_nl_embedding_gguf: Path ) -> None: from mobius.integrations.gguf import build_from_gguf - model = build_from_gguf(iq4_nl_embedding_gguf)["model"] - gather_nodes = [node for node in model.graph if node.op_type == "GatherBlockQuantized"] + with pytest.raises(ValueError, match="cannot retain native block format IQ4_NL"): + build_from_gguf(iq4_nl_embedding_gguf) - assert len(gather_nodes) == 1 - assert "model.embed_tokens.qweight" in model.graph.initializers - assert "model.embed_tokens.weight" not in model.graph.initializers + model = build_from_gguf(iq4_nl_embedding_gguf, keep_quantized=False)["model"] + assert all(node.op_type != "GatherBlockQuantized" for node in model.graph) + assert "model.embed_tokens.weight" in model.graph.initializers def test_tied_quantized_embedding_drives_matmulnbits_head( self, q4_0_tied_embedding_gguf: Path @@ -3601,41 +3617,29 @@ def test_tied_quantized_embedding_drives_matmulnbits_head( assert "model.embed_tokens.zero_points" in model.graph.initializers assert not any(name.startswith("lm_head.") for name in model.graph.initializers) - def test_untied_quantized_head_uses_q4_matmulnbits( + def test_untied_head_with_incompatible_affine_target_fails_closed( self, q4_0_embedding_q8_head_gguf: Path ): - """An untied quantized output is requantized to the graph's Q4 layout.""" - import onnx_ir as ir - + """An untied Q8 head is not silently requantized to the graph's Q4 layout.""" from mobius.integrations.gguf import build_from_gguf - model = build_from_gguf(q4_0_embedding_q8_head_gguf, keep_quantized=True)["model"] - head_nodes = [ - node - for node in model.graph - if node.op_type == "MatMulNBits" and node.outputs[0].name == "logits" - ] - assert len(head_nodes) == 1 - assert head_nodes[0].attributes["bits"].value == 4 - assert head_nodes[0].attributes["block_size"].value == 32 - - qweight = model.graph.initializers["lm_head.weight"] - assert qweight.dtype == ir.DataType.UINT8 - assert list(qweight.shape) == [256, 2, 16] - assert list(model.graph.initializers["lm_head.scales"].shape) == [256, 2] - assert "lm_head.weight_t" not in model.graph.initializers + with pytest.raises( + ValueError, match=r"token_embd\.weight \(Q4_0.*output\.weight \(Q8_0" + ): + build_from_gguf(q4_0_embedding_q8_head_gguf, keep_quantized=True) def test_unsupported_requantization_target_has_clear_error( self, q8_0_projection_q4_head_gguf: Path ): - """Mixed targets outside 4-bit/block-32 fail before the Q4 repacker.""" + """Mixed affine targets fail before any head requantization.""" from mobius.integrations.gguf import build_from_gguf with pytest.raises( ValueError, match=( - "keep_quantized MatMulNBits requantization currently supports only " - r"4-bit/block-32 targets; got bits=8 block=32 for tensor lm_head\.weight" + r"incompatible targets: output\.weight " + r"\(Q4_0: 4-bit/block-32\), blk\.0\.attn_q\.weight " + r"\(Q8_0: 8-bit/block-32\)" ), ): build_from_gguf(q8_0_projection_q4_head_gguf, keep_quantized=True) @@ -3733,7 +3737,7 @@ def test_decoder_backed_embedding_uses_affine_gather_target(self, monkeypatch): assert _can_quantize_embedding(model, "llama", bits=4, block_size=32) - def test_decoder_backed_output_head_stays_quantized(self): + def test_decoder_backed_output_head_is_not_claimed_as_preserved(self): from mobius.integrations.gguf._builder import _can_quantize_lm_head class _OutputModel: @@ -3745,13 +3749,15 @@ def tensor_items_raw(self): (256, 64), ) - assert _can_quantize_lm_head(_OutputModel(), "llama") + assert not _can_quantize_lm_head(_OutputModel(), "llama") - def test_detect_q4_k_m_mixed_profile(self): - """Q4_K presence selects a 4-bit target despite more Q5_0 tensors.""" + def test_q4_k_m_mixed_profile_fails_closed(self): + """Mixed Q4_K_M projections cannot be losslessly normalized to INT4.""" from gguf import GGMLQuantizationType - from mobius.integrations.gguf._builder import _detect_quant_params + from mobius.integrations.gguf._builder import ( + _reject_unsupported_quantization_preservation, + ) class _MixedModel: def tensor_items_raw(self): @@ -3769,8 +3775,16 @@ def tensor_items_raw(self): (64, 128), ) - bits, block_size, is_sym = _detect_quant_params(_MixedModel(), "llama") - assert (bits, block_size, is_sym) == (4, 32, False) + with pytest.raises( + ValueError, + match=( + r"blk\.0\.attn_q\.weight \(Q5_0\).*cannot represent this " + r"source block format losslessly" + ), + ): + _reject_unsupported_quantization_preservation( + _MixedModel(), "llama", preserve_quantization=True + ) def test_decoder_backed_qtype_selects_explicit_requantization_target(self): """A decoder-backed qtype takes the declared 4-bit requantization route.""" @@ -3846,17 +3860,13 @@ def test_architecture_without_quantized_modules_rejects_preservation( ): build_from_gguf(path) - def test_q6_k_selects_the_asymmetric_four_bit_repack_target(self): - """Q6_K repacks to the same 4-bit/32 target as Q4_K, with zero points. - - Q6_K's source form is symmetric around 32, but it reaches MatMulNBits - through the asymmetric affine requantizer, so zero points are required. - A missing entry in `type_can_omit_zero_points` raises `KeyError` here - rather than producing a wrong model. - """ + def test_q6_k_projection_fails_closed(self): + """A stacked 6-bit expert projection must not be silently requantized.""" from gguf import GGMLQuantizationType - from mobius.integrations.gguf._builder import _detect_quant_params + from mobius.integrations.gguf._builder import ( + _reject_unsupported_quantization_preservation, + ) class _Q6KModel: def tensor_items_raw(self): @@ -3864,13 +3874,296 @@ def tensor_items_raw(self): "blk.0.ffn_down.weight", np.empty(0, dtype=np.uint8), GGMLQuantizationType.Q6_K, + (4, 64, 128), + ) + + with pytest.raises(ValueError, match=r"ffn_down\.weight \(Q6_K\)"): + _reject_unsupported_quantization_preservation( + _Q6KModel(), "llama", preserve_quantization=True + ) + + def test_native_blocks_fail_closed_when_builder_cannot_install_native_modules(self): + """A builder without native module replacement must not requantize native blocks.""" + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._builder import ( + _reject_unsupported_quantization_preservation, + ) + + class _NativeModel: + def tensor_items_raw(self): + yield ( + "blk.0.attn_q.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.MXFP4, + (64, 64), + ) + + with pytest.raises(ValueError, match=r"native block format MXFP4"): + _reject_unsupported_quantization_preservation( + _NativeModel(), + "gemma4", + preserve_quantization=True, + allow_native_blocks=False, + ) + + def test_native_mtp_blocks_fail_closed_without_sidecar_module_replacement(self): + """Native MTP blocks must not fall through to affine requantization.""" + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._builder import ( + _reject_unsupported_quantization_preservation, + ) + + class _NativeMtpModel: + metadata: ClassVar[dict[str, int]] = { + "qwen35.block_count": 25, + "qwen35.nextn_predict_layers": 1, + } + + def tensor_items_raw(self): + yield ( + "blk.24.attn_q.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.MXFP4, (64, 128), ) - bits, block_size, is_sym = _detect_quant_params(_Q6KModel(), "llama") + with pytest.raises(ValueError, match=r"native block format MXFP4.*blk\.24"): + _reject_unsupported_quantization_preservation( + _NativeMtpModel(), "qwen35", preserve_quantization=True + ) - assert (bits, block_size) == (4, 32) - assert is_sym is False + @pytest.mark.parametrize( + ("tensor_name", "options", "message"), + [ + ( + "token_embd.weight", + {"allow_quantized_embeddings": False}, + "packed embedding", + ), + ( + "output.weight", + {"allow_quantized_lm_head": False}, + "packed LM head", + ), + ], + ) + def test_builder_specific_float_roles_fail_closed( + self, tensor_name: str, options: dict[str, bool], message: str + ): + """A multimodal builder cannot silently dequantize packed tables.""" + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._builder import ( + _reject_unsupported_quantization_preservation, + ) + + class _PackedTableModel: + def tensor_items_raw(self): + yield ( + tensor_name, + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q4_0, + (256, 64), + ) + + with pytest.raises(ValueError, match=message): + _reject_unsupported_quantization_preservation( + _PackedTableModel(), + "muse_glimmer", + preserve_quantization=True, + **options, + ) + + def test_quantized_fused_expert_gate_up_fails_closed(self): + """Packed fused expert tensors cannot be split through the float normalizer.""" + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._builder import ( + _reject_unsupported_quantization_preservation, + ) + + class _FusedExpertModel: + def tensor_items_raw(self): + yield ( + "blk.0.ffn_gate_up_exps.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q4_0, + (4, 128, 64), + ) + + with pytest.raises(ValueError, match=r"cannot split packed fused expert tensor"): + _reject_unsupported_quantization_preservation( + _FusedExpertModel(), "qwen35moe", preserve_quantization=True + ) + + def test_unmapped_mtp_projection_fails_closed(self): + """Sidecar-specific mappings cannot bypass value-preservation policy.""" + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._builder import ( + _has_quantized_weights, + _reject_unsupported_quantization_preservation, + ) + + class _MTPModel: + def tensor_items_raw(self): + yield ( + "blk.24.nextn.ffn_down.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q4_K, + (64, 128), + ) + + model = _MTPModel() + assert _has_quantized_weights(model, "qwen3") + with pytest.raises(ValueError, match=r"nextn\.ffn_down\.weight \(Q4_K\)"): + _reject_unsupported_quantization_preservation( + model, "qwen3", preserve_quantization=True + ) + + def test_lossy_tied_embedding_source_fails_closed(self): + """A tied Q4_K table cannot silently dequantize for Gather and the head.""" + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._builder import ( + _reject_unsupported_quantization_preservation, + ) + + class _EmbeddingModel: + def tensor_items_raw(self): + yield ( + "token_embd.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q4_K, + (256, 128), + ) + + with pytest.raises(ValueError, match=r"token_embd\.weight \(Q4_K\)"): + _reject_unsupported_quantization_preservation( + _EmbeddingModel(), "llama", preserve_quantization=True + ) + + def test_mixed_lossless_affine_targets_still_fail_closed(self): + """Q4_0 and Q8_0 are exact alone but cannot share one graph contract.""" + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._builder import ( + _reject_unsupported_quantization_preservation, + ) + + class _MixedExactModel: + def tensor_items_raw(self): + yield ( + "blk.0.attn_q.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q4_0, + (64, 64), + ) + yield ( + "blk.0.attn_v.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q8_0, + (64, 64), + ) + + with pytest.raises(ValueError, match=r"incompatible targets:.*Q4_0.*Q8_0"): + _reject_unsupported_quantization_preservation( + _MixedExactModel(), "llama", preserve_quantization=True + ) + + def test_encoder_float_embedding_does_not_constrain_projection_target(self): + """An encoder embedding's float route is independent of packed projections.""" + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._builder import ( + _detect_quant_params, + _reject_unsupported_quantization_preservation, + ) + + class _EncoderModel: + def tensor_items_raw(self): + yield ( + "position_embd.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q8_0, + (256, 64), + ) + yield ( + "blk.0.attn_norm.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q8_0, + (64,), + ) + yield ( + "blk.0.attn_q.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q4_0, + (64, 64), + ) + + _reject_unsupported_quantization_preservation( + _EncoderModel(), "bert", preserve_quantization=True + ) + assert _detect_quant_params(_EncoderModel(), "bert") == (4, 32, False) + + def test_explicit_float_linear_does_not_constrain_projection_target(self): + """A hybrid float-only linear may use a different source qtype.""" + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._builder import ( + _detect_quant_params, + _reject_unsupported_quantization_preservation, + ) + + class _HybridModel: + def tensor_items_raw(self): + yield ( + "blk.0.ssm_in.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q8_0, + (64, 64), + ) + yield ( + "blk.1.attn_q.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q4_0, + (64, 64), + ) + + _reject_unsupported_quantization_preservation( + _HybridModel(), + "jamba", + preserve_quantization=True, + dequantize_float_linear_types={ + "model.layers.0.mamba.in_proj": {"Q8_0"}, + }, + ) + assert _detect_quant_params(_HybridModel(), "jamba") == (4, 32, False) + + def test_native_blocks_use_the_exact_affine_companion_target(self): + """Native projections do not force exact Q8_0 companions through INT4.""" + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._builder import _detect_quant_params + + class _NativeAndQ8Model: + def tensor_items_raw(self): + yield ( + "blk.0.attn_q.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.IQ4_XS, + (64, 64), + ) + yield ( + "blk.0.attn_v.weight", + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q8_0, + (64, 64), + ) + + assert _detect_quant_params(_NativeAndQ8Model(), "llama") == (8, 32, False) def test_runtime_unsupported_format_does_not_select_native_op(self): """A GGUF type outside the runtime contract remains on the fallback.""" @@ -4160,7 +4453,7 @@ def test_float_fused_bert_qkv_splits_losslessly_and_runs(self, tmp_path: Path) - assert output.shape == (1, 5, 64) assert np.isfinite(output).all() - def test_float_fused_bert_qkv_repacks_in_mixed_quantized_file( + def test_float_fused_bert_qkv_fails_closed_in_mixed_quantized_file( self, tmp_path: Path ) -> None: from mobius.integrations.gguf import build_from_gguf @@ -4173,13 +4466,11 @@ def test_float_fused_bert_qkv_repacks_in_mixed_quantized_file( fused_qkv=True, fused_qkv_float=True, ) - preserved = build_from_gguf(path, keep_quantized=True)["model"] - explicit_float = build_from_gguf(path, keep_quantized=False)["model"] + with pytest.raises(ValueError, match=r"would quantize float projection"): + build_from_gguf(path, keep_quantized=True) - assert "MatMulNBits" in {node.op_type for node in preserved.graph} - actual = self._run(preserved, 5, masked=True) - expected = self._run(explicit_float, 5, masked=True) - np.testing.assert_allclose(actual, expected, rtol=0, atol=1e-2) + explicit_float = build_from_gguf(path, keep_quantized=False)["model"] + assert np.isfinite(self._run(explicit_float, 5, masked=True)).all() @pytest.mark.parametrize( ("quantized", "width_delta", "message"), @@ -5901,38 +6192,32 @@ def test_float_import_runtime_state_replay_reorder_and_roundtrip( == model.metadata_props["mobius.cache_abi"] ) - def test_quantized_source_preserves_only_matmul_roles(self, tmp_path: Path) -> None: + @pytest.mark.parametrize( + ("native_quantized", "qtype_name"), + [(False, "Q4_0"), (True, "IQ4_NL")], + ) + def test_quantized_mla_reshape_requires_explicit_dequantization( + self, + tmp_path: Path, + native_quantized: bool, + qtype_name: str, + ) -> None: from mobius.integrations.gguf import build_from_gguf - path = tmp_path / "kimi-linear-q4.gguf" - _write_kimi_linear_gguf(path, quantized=True) - model = build_from_gguf(path, keep_quantized=True)["model"] - quantized_inputs = [ - node.inputs[1].name - for node in model.graph - if node.op_type == "MatMulNBits" and len(node.inputs) > 1 - ] - assert quantized_inputs - assert all( - not any(term in name for term in ("norm", "conv1d", "A_log", "dt_bias")) - for name in quantized_inputs + path = tmp_path / f"kimi-linear-{qtype_name.lower()}.gguf" + _write_kimi_linear_gguf( + path, + quantized=True, + native_quantized=native_quantized, ) + with pytest.raises( + ValueError, + match=rf"change the dequantized values.*attn_k_b\.weight \({qtype_name}\)", + ): + build_from_gguf(path, keep_quantized=True) - def test_native_quantized_mla_projections_requantize_after_reshape( - self, tmp_path: Path - ) -> None: - from mobius.integrations.gguf import build_from_gguf - - path = tmp_path / "kimi-linear-iq4-nl.gguf" - _write_kimi_linear_gguf(path, quantized=True, native_quantized=True) - model = build_from_gguf(path, keep_quantized=True)["model"] - quantized_inputs = { - node.inputs[1].name - for node in model.graph - if node.op_type == "MatMulNBits" and len(node.inputs) > 1 - } - assert any(name.endswith("k_b_proj.weight") for name in quantized_inputs) - assert any(name.endswith("v_b_proj.weight") for name in quantized_inputs) + model = build_from_gguf(path, keep_quantized=False)["model"] + assert all(node.op_type != "MatMulNBits" for node in model.graph) def test_left_padding_does_not_change_valid_logits_or_states(self, tmp_path: Path) -> None: from mobius._testing.ort_inference import OnnxModelSession diff --git a/src/mobius/integrations/gguf/_mmproj.py b/src/mobius/integrations/gguf/_mmproj.py index cc0b917a2..1ec8dd64e 100644 --- a/src/mobius/integrations/gguf/_mmproj.py +++ b/src/mobius/integrations/gguf/_mmproj.py @@ -929,11 +929,13 @@ def _text_gguf_to_hf_multimodal_quantized( import torch from mobius.integrations.gguf._builder import repack_gguf_weight_to_target + from mobius.integrations.gguf._quant_registry import float_storage_type_ids from mobius.integrations.gguf._spec import TensorRole quantize_embeddings = bool(getattr(config.quantization, "quantize_embeddings", False)) quantize_lm_head = bool(getattr(config.quantization, "quantize_lm_head", False)) tie_word_embeddings = bool(config.tie_word_embeddings) + float_type_ids = float_storage_type_ids() def _emit_linear(state_dict: dict, stem: str, repacked) -> None: state_dict[f"{stem}.weight"] = torch.from_numpy(repacked.weight) @@ -960,6 +962,14 @@ def _emit_embedding(state_dict: dict, stem: str, repacked) -> None: is_quant_embedding = quantize_embeddings and hf_name in _QUANTIZED_EMBEDDING_NAMES if (is_quant_linear or is_quant_embedding) and len(np_shape) == 2: + qtype_id = getattr(qtype, "value", qtype) + if qtype_id in float_type_ids: + raise ValueError( + "Quantization-preserving Gemma4 GGUF import would quantize float " + f"projection {gguf_name} ({getattr(qtype, 'name', qtype)}) to " + f"the graph's {bits}-bit/block-{block_size} MatMulNBits contract. " + "Use keep_quantized=False (API) for explicit float import." + ) repacked = repack_gguf_weight_to_target( text_gguf, raw, @@ -992,6 +1002,13 @@ def _emit_embedding(state_dict: dict, stem: str, repacked) -> None: continue # Everything else (norms, float per-layer projections) stays float. + qtype_id = getattr(qtype, "value", qtype) + if qtype_id not in float_type_ids: + raise ValueError( + "Quantization-preserving Gemma4 GGUF import cannot retain packed tensor " + f"{gguf_name} ({getattr(qtype, 'name', qtype)}) because its graph " + "target is float. Use keep_quantized=False (API) for explicit float import." + ) values = np.array(text_gguf.dequantize_raw_tensor(raw, qtype, np_shape)).astype( np.float32 ) @@ -1080,6 +1097,7 @@ def build_gemma4_vlm_from_gguf( from mobius._builder import resolve_dtype from mobius.integrations.gguf._builder import ( _has_quantized_weights, + _reject_unsupported_quantization_preservation, _validate_gguf_model, ) from mobius.integrations.gguf._config_mapping import gguf_to_config @@ -1099,6 +1117,12 @@ def build_gemma4_vlm_from_gguf( _validate_gguf_model(text_gguf, source=str(text_gguf_path)) preserve_quantization = keep_quantized and _has_quantized_weights(text_gguf, "gemma4") + _reject_unsupported_quantization_preservation( + text_gguf, + "gemma4", + preserve_quantization=preserve_quantization, + allow_native_blocks=False, + ) if keep_quantized and not preserve_quantization: logger.info( "Text GGUF contains no mapped quantized weights; using the float import path" @@ -1312,6 +1336,7 @@ def build_muse_glimmer_vlm_from_gguf( _load_dequantized_state_dict, _load_quantized_state_dict, _normalize_gguf_weights, + _reject_unsupported_quantization_preservation, _replace_native_block_linears, _validate_gguf_model, ) @@ -1364,6 +1389,13 @@ def build_muse_glimmer_vlm_from_gguf( _validate_projector_output_and_media_tokens(config, text_gguf, mmproj_gguf) preserve_quantization = keep_quantized and _has_quantized_weights(text_gguf, text_arch) + _reject_unsupported_quantization_preservation( + text_gguf, + text_arch, + preserve_quantization=preserve_quantization, + allow_quantized_embeddings=False, + allow_quantized_lm_head=False, + ) if keep_quantized and not preserve_quantization: logger.info( "Text GGUF contains no mapped quantized weights; using the float import path" diff --git a/src/mobius/integrations/gguf/_mmproj_test.py b/src/mobius/integrations/gguf/_mmproj_test.py index 4a8f9f52a..c33d9542b 100644 --- a/src/mobius/integrations/gguf/_mmproj_test.py +++ b/src/mobius/integrations/gguf/_mmproj_test.py @@ -206,7 +206,7 @@ def _write_minimal_gguf( writer.close() -def _write_quantized_gemma4_text_gguf(path: Path) -> None: +def _write_quantized_gemma4_text_gguf(path: Path, *, float_projection: bool = False) -> None: """Write a tiny Gemma4 text GGUF with Q4 projections and a float embedding.""" from gguf import GGMLQuantizationType, GGUFWriter @@ -274,7 +274,10 @@ def _q4_0(name: str, n_out: int, k_in: int) -> None: for layer in range(num_layers): prefix = f"blk.{layer}" - _q4_0(f"{prefix}.attn_q.weight", num_heads * head_dim, hidden_size) + if float_projection and layer == 0: + _f32(f"{prefix}.attn_q.weight", (num_heads * head_dim, hidden_size)) + else: + _q4_0(f"{prefix}.attn_q.weight", num_heads * head_dim, hidden_size) _q4_0(f"{prefix}.attn_k.weight", num_kv_heads * head_dim, hidden_size) _q4_0(f"{prefix}.attn_v.weight", num_kv_heads * head_dim, hidden_size) _q4_0(f"{prefix}.attn_output.weight", hidden_size, num_heads * head_dim) @@ -1030,6 +1033,63 @@ def test_incompatible_embedding_stays_float_and_package_round_trips( for initializer in model.graph.initializers.values() ) + def test_float_projection_in_quantized_text_fails_closed( + self, clip_mmproj_gguf: Path, tmp_path: Path + ): + from mobius.integrations.gguf import build_gemma4_vlm_from_gguf + + text_gguf = tmp_path / "gemma4-q4-f32-projection.gguf" + _write_quantized_gemma4_text_gguf(text_gguf, float_projection=True) + + with pytest.raises(ValueError, match=r"would quantize float projection"): + build_gemma4_vlm_from_gguf(text_gguf, clip_mmproj_gguf, image_token_id=63) + + package = build_gemma4_vlm_from_gguf( + text_gguf, + clip_mmproj_gguf, + image_token_id=63, + keep_quantized=False, + ) + assert "MatMulNBits" not in _component_op_types(package["decoder"]) + + @pytest.mark.parametrize( + "tensor_name", + ["per_layer_token_embd.weight", "per_layer_model_proj.weight"], + ) + def test_quantized_per_layer_table_fails_closed(self, tensor_name: str): + from types import SimpleNamespace + + from gguf import GGMLQuantizationType + + from mobius.integrations.gguf._mmproj import ( + _text_gguf_to_hf_multimodal_quantized, + ) + + class _PackedPerLayerModel: + def tensor_items_raw(self): + yield ( + tensor_name, + np.empty(0, dtype=np.uint8), + GGMLQuantizationType.Q4_0, + (2, 32, 32), + ) + + config = SimpleNamespace( + quantization=SimpleNamespace( + quantize_embeddings=False, + quantize_lm_head=False, + ), + tie_word_embeddings=False, + ) + with pytest.raises(ValueError, match=r"cannot retain packed tensor"): + _text_gguf_to_hf_multimodal_quantized( + _PackedPerLayerModel(), + config, + bits=4, + block_size=32, + symmetric=True, + ) + def test_quantized_decoder_loads_in_onnxruntime( self, clip_mmproj_gguf: Path, tmp_path: Path ): diff --git a/src/mobius/integrations/gguf/_mtp_test.py b/src/mobius/integrations/gguf/_mtp_test.py index d6ab0ef3e..eec5f66d2 100644 --- a/src/mobius/integrations/gguf/_mtp_test.py +++ b/src/mobius/integrations/gguf/_mtp_test.py @@ -594,9 +594,7 @@ def test_target_and_sidecar_save_with_weight_checks_and_reload( assert "embed_tokens.weight" not in sidecar.graph.initializers assert not any("nextn" in name for name in target.graph.initializers) - def test_asymmetric_dedicated_head_full_logits_match_explicit_dequantization( - self, tmp_path: Path - ): + def test_asymmetric_dedicated_head_requires_explicit_dequantization(self, tmp_path: Path): import onnxruntime as ort from mobius.integrations.gguf import build_from_gguf @@ -614,7 +612,12 @@ def test_asymmetric_dedicated_head_full_logits_match_explicit_dequantization( asymmetric_dedicated_head=True, ) shared = build_from_gguf(shared_path, keep_quantized=True).mtp_head - dedicated = build_from_gguf(dedicated_path, keep_quantized=True).mtp_head + with pytest.raises( + ValueError, + match=r"nextn\.shared_head_head\.weight \(Q4_1\).*cannot represent", + ): + build_from_gguf(dedicated_path, keep_quantized=True) + dedicated = build_from_gguf(dedicated_path, keep_quantized=False).mtp_head shared_dir = tmp_path / "shared" dedicated_dir = tmp_path / "dedicated" shared.save(shared_dir, progress_bar=False, check_weights=True) diff --git a/src/mobius/integrations/gguf/_quant_registry.py b/src/mobius/integrations/gguf/_quant_registry.py index 2ef46dfad..d8c45c8a4 100644 --- a/src/mobius/integrations/gguf/_quant_registry.py +++ b/src/mobius/integrations/gguf/_quant_registry.py @@ -34,6 +34,7 @@ "get_quant_spec", "iter_quant_specs", "lm_head_preserve_type_names", + "lossless_preservation_type_names", "native_block_format", "quant_import_decision", "quant_spec_by_name", @@ -89,16 +90,18 @@ #: affine path. So every target here emits zero points explicitly. _AFFINE_REPACK_TARGETS: MappingProxyType[str, AffineRepackSpec] = MappingProxyType( { - "Q4_0": AffineRepackSpec(bits=4, block_size=32, omit_zero_points=False), + "Q4_0": AffineRepackSpec(bits=4, block_size=32, omit_zero_points=False, lossless=True), "Q4_1": AffineRepackSpec(bits=4, block_size=32, omit_zero_points=False), - "Q8_0": AffineRepackSpec(bits=8, block_size=32, omit_zero_points=False), + "Q8_0": AffineRepackSpec(bits=8, block_size=32, omit_zero_points=False, lossless=True), "Q4_K": AffineRepackSpec(bits=4, block_size=32, omit_zero_points=False), "Q6_K": AffineRepackSpec(bits=4, block_size=32, omit_zero_points=False), # Mainline Q1_0 is 1-bit binary over 128-element blocks, repacked into # 2-bit MatMulNBits with zp=1. Tencent's custom Q1_0 reuses the same # type id with a different on-disk layout and is handled separately in # ``_tencent_q1_0``. - "Q1_0": AffineRepackSpec(bits=2, block_size=128, omit_zero_points=False), + "Q1_0": AffineRepackSpec( + bits=2, block_size=128, omit_zero_points=False, lossless=True + ), } ) @@ -162,9 +165,21 @@ #: the two tables above because the head is also allowed to ride the generic #: requantization path. _LM_HEAD_PRESERVE: frozenset[str] = frozenset( - name - for name, (route, _) in _STORED_ROUTE_POLICY.items() - if route is not QuantImportRoute.REJECTED + { + "Q1_0", + "Q4_0", + "Q8_0", + "MXFP4", + "IQ4_NL", + "IQ4_XS", + "IQ3_S", + "IQ3_XXS", + "IQ2_XXS", + "IQ2_XS", + "IQ2_S", + "IQ1_S", + "IQ1_M", + } ) #: Upstream ``gguf_role`` strings → :class:`StorageRole`. The census stores the @@ -491,3 +506,9 @@ def render_quant_support_matrix() -> str: f"{spec.runtime.value} |" ) return "\n".join(rows) + + +@functools.lru_cache(maxsize=1) +def lossless_preservation_type_names() -> frozenset[str]: + """Return types whose quantized route preserves source dequantized values.""" + return frozenset(spec.name for spec in iter_quant_specs() if spec.preserves_values) diff --git a/src/mobius/integrations/gguf/_quant_registry_test.py b/src/mobius/integrations/gguf/_quant_registry_test.py index b23010182..e8acb7d4c 100644 --- a/src/mobius/integrations/gguf/_quant_registry_test.py +++ b/src/mobius/integrations/gguf/_quant_registry_test.py @@ -27,6 +27,7 @@ get_quant_spec, iter_quant_specs, lm_head_preserve_type_names, + lossless_preservation_type_names, quant_import_decision, quant_spec_by_name, render_quant_support_matrix, @@ -87,19 +88,8 @@ _EXPECTED_LM_HEAD_PRESERVE = frozenset( { "Q1_0", - "Q2_K", - "Q3_K", "Q4_0", - "Q4_1", - "Q4_K", - "Q5_0", - "Q5_1", - "Q5_K", - "Q6_K", "Q8_0", - "TQ1_0", - "TQ2_0", - "NVFP4", "MXFP4", "IQ4_NL", "IQ4_XS", @@ -186,6 +176,11 @@ def test_explicit_zero_point_types(self) -> None: def test_lm_head_preserve_types(self) -> None: assert lm_head_preserve_type_names() == _EXPECTED_LM_HEAD_PRESERVE + def test_only_value_preserving_affine_types_are_advertised(self) -> None: + preserved = lossless_preservation_type_names() + assert {"Q1_0", "Q4_0", "Q8_0"} <= preserved + assert {"Q4_1", "Q4_K", "Q6_K"}.isdisjoint(preserved) + def test_float_storage_types(self) -> None: assert float_storage_type_ids() == _LEGACY_FLOAT_TYPE_IDS diff --git a/src/mobius/integrations/gguf/_repacker_test.py b/src/mobius/integrations/gguf/_repacker_test.py index ad184835f..471e8a3a2 100644 --- a/src/mobius/integrations/gguf/_repacker_test.py +++ b/src/mobius/integrations/gguf/_repacker_test.py @@ -550,6 +550,31 @@ def test_batch(self): class TestRepackQ4K: + def test_asymmetric_block_matches_explicit_llama_dequantization(self): + """Exercise scale high bits, min subtraction, and paired nibble ordering.""" + block = _make_q4_k_block( + d=1.0, + dmin=1.0, + sub_scales=[1, 2, 3, 4, 17, 18, 19, 20], + sub_mins=[5, 6, 7, 8, 33, 34, 35, 36], + nibbles=[ + value + for pair in ((0, 1), (2, 3), (4, 5), (6, 7)) + for value in pair + for _ in range(32) + ], + ) + + actual = quants.dequantize( + block.reshape(1, -1), quants.GGMLQuantizationType.Q4_K + ).ravel() + expected = np.repeat( + np.array([-5.0, -4.0, -1.0, 4.0, 35.0, 56.0, 79.0, 104.0]), + 32, + ) + + np.testing.assert_array_equal(actual, expected) + def test_single_super_block_shapes(self): """Single Q4_K super-block -> 8 MatMulNBits sub-blocks.""" sc = [10] * 8 diff --git a/src/mobius/integrations/gguf/_spec.py b/src/mobius/integrations/gguf/_spec.py index 34dc4a792..d2b3227d3 100644 --- a/src/mobius/integrations/gguf/_spec.py +++ b/src/mobius/integrations/gguf/_spec.py @@ -166,11 +166,14 @@ class AffineRepackSpec: input. This is a property of the *target*, not of the source: Q6_K is symmetric on disk but requantizes through the asymmetric affine path, so it still needs zero points. + lossless: Whether repacking preserves the source dequantized values + without requantization. """ bits: int block_size: int omit_zero_points: bool = False + lossless: bool = False def __post_init__(self) -> None: if self.bits <= 0 or self.block_size <= 0: @@ -392,6 +395,7 @@ def __post_init__(self) -> None: f"{label}: unreadable slots cannot be dequantized; the GGUF parse " "layer rejects them first" ) + if self.native_preserve is not None and self.affine_repack is not None: raise ValueError( f"{label}: a type is either preserved natively or repacked into an " @@ -436,6 +440,13 @@ def __post_init__(self) -> None: "storage types" ) + @property + def preserves_values(self) -> bool: + """Whether a quantized graph can consume this type without requantization.""" + return self.native_preserve is not None or ( + self.affine_repack is not None and self.affine_repack.lossless + ) + @property def readable(self) -> bool: """Whether the GGUF parse layer accepts a tensor of this type.""" diff --git a/tests/gguf_dense_cohort_integration_test.py b/tests/gguf_dense_cohort_integration_test.py index f25129a0a..01f17a3ad 100644 --- a/tests/gguf_dense_cohort_integration_test.py +++ b/tests/gguf_dense_cohort_integration_test.py @@ -8,8 +8,6 @@ from dataclasses import dataclass from pathlib import Path -import numpy as np -import onnxruntime as ort import pytest from huggingface_hub import hf_hub_download @@ -26,7 +24,6 @@ class _Artifact: size: int sha256: str qtypes: dict[str, int] - greedy_token: int _ARTIFACTS = ( @@ -38,7 +35,6 @@ class _Artifact: size=733_520_128, sha256="2a848051ef7a3edfd829ce915835794e789e6ed7f425066c242759b8dbc645b4", qtypes={"Q4_K": 96, "Q6_K": 17}, - greedy_token=187, ), _Artifact( architecture="olmo2", @@ -48,7 +44,6 @@ class _Artifact: size=935_515_296, sha256="abd8187934a438fbf7cfff0a1de5b9d2793ce913f158794df1951dcba6c93cc6", qtypes={"F32": 65, "Q4_K": 97, "Q6_K": 17}, - greedy_token=16, ), _Artifact( architecture="smollm3", @@ -58,7 +53,6 @@ class _Artifact: size=1_915_305_312, sha256="8334b850b7bd46238c16b0c550df2138f0889bf433809008cc17a8b05761863e", qtypes={"F32": 73, "Q4_K": 216, "Q6_K": 37}, - greedy_token=12_286, ), ) @@ -73,8 +67,8 @@ def _sha256(path: Path) -> str: @pytest.mark.integration @pytest.mark.parametrize("artifact", _ARTIFACTS, ids=lambda artifact: artifact.architecture) -def test_real_dense_gguf_artifact(artifact: _Artifact, tmp_path: Path) -> None: - """Pinned mixed-qtype GGUFs import through the asserted affine-requantization route.""" +def test_real_dense_gguf_artifact_fails_closed(artifact: _Artifact) -> None: + """Pinned Q4_K_M artifacts do not silently requantize incompatible projections.""" path = Path( hf_hub_download( repo_id=artifact.repo_id, @@ -89,36 +83,8 @@ def test_real_dense_gguf_artifact(artifact: _Artifact, tmp_path: Path) -> None: qtypes = Counter(qtype.name for _, _, qtype, _ in gguf_model.tensor_items_raw()) assert dict(sorted(qtypes.items())) == artifact.qtypes - package = build_from_gguf(path) - op_types = {node.op_type for node in package["model"].graph} - # Q4_K/Q6_K cannot be preserved by MatMulNBits. They are dequantized and - # affine-requantized to explicit-zero-point 4-bit/block-32 weights. - assert "MatMulNBits" in op_types - assert "GatherBlockQuantized" in op_types - assert "BlockQuantizedMatMul" not in op_types - - output_dir = tmp_path / artifact.architecture - package.save(output_dir, progress_bar=False) - session = ort.InferenceSession( - str(output_dir / "model.onnx"), - providers=["CPUExecutionProvider"], - ) - feeds: dict[str, np.ndarray] = {} - for model_input in session.get_inputs(): - if model_input.name == "input_ids": - feeds[model_input.name] = np.array([[1, 2]], dtype=np.int64) - elif model_input.name == "attention_mask": - feeds[model_input.name] = np.ones((1, 2), dtype=np.int64) - elif model_input.name == "position_ids": - feeds[model_input.name] = np.array([[0, 1]], dtype=np.int64) - elif model_input.name.endswith((".key", ".value")): - feeds[model_input.name] = np.empty( - [1, model_input.shape[1], 0, model_input.shape[3]], - dtype=np.float32, - ) - - first = session.run(["logits"], feeds)[0] - second = session.run(["logits"], feeds)[0] - assert np.isfinite(first).all() - np.testing.assert_array_equal(first, second) - assert int(first[0, -1].argmax()) == artifact.greedy_token + with pytest.raises( + ValueError, + match=r"Quantization-preserving GGUF import would change the dequantized values", + ): + build_from_gguf(path) diff --git a/tests/gguf_small_model_runtime_integration_test.py b/tests/gguf_small_model_runtime_integration_test.py index 43d81e2c3..93cd2f887 100644 --- a/tests/gguf_small_model_runtime_integration_test.py +++ b/tests/gguf_small_model_runtime_integration_test.py @@ -27,7 +27,7 @@ from huggingface_hub import hf_hub_download from transformers import AutoModelForCausalLM, AutoTokenizer -from mobius import ModelPackage +from mobius import ModelPackage, build_from_gguf from mobius.__main__ import main from mobius.integrations.gguf import ( GGUFTokenizerAsset, @@ -130,7 +130,7 @@ class _RuntimeCase: reference_revision="12fd25f77366fa6b3b4b768ec3050bf629380bac", prompt="Here is my poem:", tensor_qtypes={"F16": 211, "F32": 61}, - config_sha256="c62123baf4e95656cdc9f5b798c14319bbaafec594526c462b10555f561969f9", + config_sha256="e8ca654a7c9180821b588ad167eaccf45a14869883789911e4481b755dc97164", generated_tokens=( 198, 198, @@ -179,6 +179,44 @@ class _RuntimeCase: ), ) +_Q4_K_M_CASE = _RuntimeCase( + name="smollm2-135m-instruct-q4-k-m", + gguf_repository="unsloth/SmolLM2-135M-Instruct-GGUF", + gguf_revision="9e6855bc4be717fca1ef21360a1db4b29d5c559a", + gguf_filename="SmolLM2-135M-Instruct-Q4_K_M.gguf", + gguf_size=105_454_144, + gguf_sha256="ed5fa30c487b282ec156c29062f1222e5c20875a944ac98289dbd242e947f747", + reference_repository="HuggingFaceTB/SmolLM2-135M-Instruct", + reference_revision="12fd25f77366fa6b3b4b768ec3050bf629380bac", + prompt="Here is my poem:", + tensor_qtypes={"F32": 61, "Q4_K": 16, "Q5_0": 166, "Q6_K": 14, "Q8_0": 15}, + config_sha256="e8ca654a7c9180821b588ad167eaccf45a14869883789911e4481b755dc97164", + generated_tokens=(198, 198, 18, 504, 2388, 13685, 284, 5208, 28, 198), + tokenizer_repository="HuggingFaceTB/SmolLM2-135M-Instruct", + tokenizer_revision="12fd25f77366fa6b3b4b768ec3050bf629380bac", + tokenizer_metadata_sha256=( + "cb0b637d59effdc3ab02f063039e597157fa4996663848cc2178510af5880ace" + ), + tokenizer_assets=( + ( + "special_tokens_map.json", + 655, + "2b7379f3ae813529281a5c602bc5a11c1d4e0a99107aaa597fe936c1e813ca52", + ), + ( + "tokenizer.json", + 2_104_556, + "9ca9acddb6525a194ec8ac7a87f24fbba7232a9a15ffa1af0c1224fcd888e47c", + ), + ( + "tokenizer_config.json", + 3_764, + "4ec77d44f62efeb38d7e044a1db318f6a939438425312dfa333b8382dbad98df", + ), + ), + tokenizer_identity_exact=False, +) + def _sha256(path: Path) -> str: digest = hashlib.sha256() @@ -420,3 +458,174 @@ def capture_save(package: ModelPackage, *args: object, **kwargs: object) -> None ort_logits = output["logits"] assert len(repeated) == len(expected) np.testing.assert_array_equal(repeated, expected) + + +@pytest.mark.integration +@pytest.mark.integration_fast +def test_smollm_q4_k_m_fails_closed_or_matches_same_artifact_when_dequantized( + tmp_path: Path, +) -> None: + """Q4_K_M preservation is rejected; explicit dequantization retains full parity.""" + case = _Q4_K_M_CASE + gguf_path = Path( + hf_hub_download( + repo_id=case.gguf_repository, + revision=case.gguf_revision, + filename=case.gguf_filename, + ) + ) + assert gguf_path.stat().st_size == case.gguf_size + assert _sha256(gguf_path) == case.gguf_sha256 + gguf_model = GGUFModel(gguf_path) + qtypes = Counter(qtype.name for _, _, qtype, _ in gguf_model.tensor_items_raw()) + assert dict(sorted(qtypes.items())) == case.tensor_qtypes + for filename, size, sha256 in case.tokenizer_assets: + asset_path = Path( + hf_hub_download( + repo_id=case.tokenizer_repository, + revision=case.tokenizer_revision, + filename=filename, + ) + ) + assert asset_path.stat().st_size == size + assert _sha256(asset_path) == sha256 + + with pytest.raises( + ValueError, + match=r"blk\.0\.attn_k\.weight \(Q5_0\).*cannot represent.*losslessly", + ): + build_from_gguf(gguf_path) + + with pytest.raises(ValueError, match=r"cannot represent.*losslessly"): + main(["build-gguf", str(gguf_path), "--output", str(tmp_path / "rejected")]) + + output_dir = tmp_path / case.name + captured: list[ModelPackage] = [] + original_save = ModelPackage.save + + def capture_save(package: ModelPackage, *args: object, **kwargs: object) -> None: + captured.append(package) + original_save(package, *args, **kwargs) + + with mock.patch.object(ModelPackage, "save", capture_save): + main( + [ + "build-gguf", + str(gguf_path), + "--output", + str(output_dir), + "--dequantize", + "--dtype", + "f32", + "--execution-provider", + "cpu", + ] + ) + + assert len(captured) == 1 + package = captured[0] + route = json.loads(package.gguf_import_route) + assert route == { + "architecture": "llama", + "config_sha256": case.config_sha256, + "execution_provider": "cpu", + "model_type": "llama", + "module_type": "llama", + "preserve_quantization": False, + "registry_import": { + "config_key_map": None, + "config_postprocessor": None, + "llama_qk_permute": True, + "offset_norm": False, + "required_metadata": [], + "rope_interleave": False, + "tensor_processor": "llama", + "v_head_reorder": False, + "vlm_builder": None, + }, + "route_schema": 1, + "static_cache": False, + "task": {"class": "builtins.str", "state": "text-generation"}, + "tensor_map_recipe": ["llama"], + } + assert all(node.op_type != "MatMulNBits" for node in package["model"].graph) + + reloaded = ModelPackage.load(output_dir) + assert tuple(reloaded) == ("model",) + rejected_package = tmp_path / f"{case.name}-runtime" + with pytest.raises(ValueError, match="No unique GGUF runtime evidence"): + write_gguf_runtime_package( + package, + gguf_path, + rejected_package, + runtime="onnx-genai", + runtime_version="1.29.0", + tokenizer_repository=case.tokenizer_repository, + tokenizer_revision=case.tokenizer_revision, + local_files_only=True, + ) + assert not rejected_package.exists() + source = GGUFTokenizerSource( + repository=case.tokenizer_repository, + revision=case.tokenizer_revision, + metadata_sha256=case.tokenizer_metadata_sha256, + assets=tuple(GGUFTokenizerAsset(*asset) for asset in case.tokenizer_assets), + ) + rejected_output = tmp_path / f"{case.name}-tokenizer" + assert ( + inspect_gguf_tokenizer(gguf_model.metadata, require_complete=True).metadata_sha256 + == case.tokenizer_metadata_sha256 + ) + with pytest.raises(ValueError, match="pad_token id differs from GGUF"): + materialize_gguf_tokenizer( + gguf_path, + rejected_output, + source=source, + metadata=gguf_model.metadata, + local_files_only=True, + ) + assert not rejected_output.exists() + session = ort.InferenceSession( + str(output_dir / "model.onnx"), providers=["CPUExecutionProvider"] + ) + + tokenizer = AutoTokenizer.from_pretrained( + case.reference_repository, revision=case.reference_revision + ) + reference = AutoModelForCausalLM.from_pretrained( + case.gguf_repository, + revision=case.gguf_revision, + gguf_file=case.gguf_filename, + dtype=torch.float32, + ).eval() + prompt_ids = tokenizer(case.prompt, return_tensors="pt").input_ids + with torch.no_grad(): + reference_output = reference(prompt_ids, use_cache=True) + + input_ids = prompt_ids.numpy() + ort_output = _run_ort(session, input_ids, _empty_cache(session), 0) + ort_logits = ort_output["logits"] + reference_logits = reference_output.logits.numpy() + np.testing.assert_allclose(ort_logits, reference_logits, rtol=1e-4, atol=2e-4) + + cache = _next_cache(ort_output) + reference_cache = reference_output.past_key_values + generated: list[int] = [] + for step in range(len(case.generated_tokens)): + token = int(ort_logits[0, -1].argmax()) + generated.append(token) + token_ids = np.array([[token]], dtype=np.int64) + ort_output = _run_ort(session, token_ids, cache, input_ids.shape[1] + step) + cache = _next_cache(ort_output) + with torch.no_grad(): + reference_output = reference( + torch.from_numpy(token_ids), + past_key_values=reference_cache, + use_cache=True, + ) + reference_cache = reference_output.past_key_values + ort_logits = ort_output["logits"] + reference_logits = reference_output.logits.numpy() + np.testing.assert_allclose(ort_logits, reference_logits, rtol=1e-4, atol=2e-4) + + np.testing.assert_array_equal(generated, case.generated_tokens)