Materialize exact GGUF tokenizers - #608
Merged
Merged
Conversation
Performance Comparison
|
justinchuby
force-pushed
the
justinchuby-validate-small-gguf-models
branch
from
August 25, 2026 17:54
efef079 to
bc8afd5
Compare
Base automatically changed from
justinchuby-validate-small-gguf-models
to
main
August 25, 2026 17:55
Bind runtime packaging to immutable tokenizer repositories, revisions, asset hashes, and GGUF semantic metadata. Publish complete packages atomically and reject any identity that cannot be proven. Record real SmolLM packaged-tokenizer L4/L5 evidence while retaining a fail-closed SmolLM2 rejection for its padding-token contradiction. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Update the SmolLM route and runtime package identities for the latest main tree, and keep newly inherited runtime guard tests pinned to their intended preconditions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Validate the complete SmolLM tokenizer pipeline rather than only its presence and publish runtime package directories with platform no-replace atomics to prevent concurrent destination clobbering. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-materialize-gguf-tokenizers
branch
from
August 25, 2026 18:31
654abfc to
0e791a5
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR tightens GGUF runtime packaging by requiring exact, immutable tokenizer provenance (Hub repo+commit plus per-asset size/SHA-256), validating tokenizer semantics against GGUF metadata, and only enabling runtime packaging when a structured evidence record matches exactly (currently SmolLM F16 on CPU with ONNX Runtime 1.29.0).
Changes:
- Add “pinned-source” tokenizer materialization: download/cache-read exact tokenizer assets by pinned Hub revision and validate vocab/merges/special IDs/pipeline/chat templates against GGUF metadata.
- Extend runtime evidence to include tokenizer metadata SHA-256 + tokenizer asset identities, and bind runtime packaging eligibility to exact evidence matches (enabling
llamaruntime only for the SmolLM F16 evidenced route). - Update CLI, schema, and golden/case testdata to require explicit pinned tokenizer inputs and to validate provenance consistently.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/gguf_test.py | Updates CLI tests to require explicit pinned tokenizer args for runtime packaging. |
| tests/gguf_small_model_runtime_integration_test.py | Adds real-artifact checks for tokenizer asset size/SHA-256 and validates exact vs rejected runtime/tokenizer flows. |
| tests/e2e_golden_test.py | Asserts GGUF golden provenance is present and matches the pinned case source. |
| tests/cli_test.py | Adds/updates CLI behavior tests for required pinned tokenizer inputs and immutable revision enforcement. |
| testdata/golden/causal-lm/smollm2-135m-gguf-f16.json | Records pinned tokenizer evidence and rejection reason in GGUF provenance. |
| testdata/golden/causal-lm/smollm2-135m-gguf-f16_generation.json | Records pinned tokenizer evidence and rejection reason in GGUF provenance (generation). |
| testdata/golden/causal-lm/smollm-135m-gguf-f16.json | Records exact pinned tokenizer evidence in GGUF provenance. |
| testdata/golden/causal-lm/smollm-135m-gguf-f16_generation.json | Records exact pinned tokenizer evidence in GGUF provenance (generation). |
| testdata/cases/schema.json | Extends GGUF case schema to require a structured pinned tokenizer block with assets + identity status. |
| testdata/cases/causal-lm/smollm2-135m-gguf-f16.yaml | Adds pinned tokenizer evidence with “rejected” status and explicit rejection reason. |
| testdata/cases/causal-lm/smollm-135m-gguf-f16.yaml | Adds pinned tokenizer evidence with “exact” status and asset identities. |
| src/mobius/integrations/gguf/_tokenizer.py | Implements pinned-source tokenizer materialization and semantic validation against GGUF metadata. |
| src/mobius/integrations/gguf/_tokenizer_test.py | Adds unit tests for pinned-source invariants, offline cache hardening, and fail-closed behavior. |
| src/mobius/integrations/gguf/_runtime_package.py | Requires pinned tokenizer inputs, materializes tokenizer from evidence, and switches publication to “no-replace” atomic directory publish. |
| src/mobius/integrations/gguf/_runtime_package_test.py | Updates runtime package tests for pinned tokenizer requirements and no-replace publication behavior. |
| src/mobius/integrations/gguf/_runtime_evidence.py | Extends evidence records with tokenizer metadata digest + asset identities and adds a concrete SmolLM F16 runtime evidence record. |
| src/mobius/integrations/gguf/_runtime_evidence_test.py | Updates evidence tests for new tokenizer evidence fields and matching behavior. |
| src/mobius/integrations/gguf/_docs.py | Updates generated closure summary text to reflect pinned-source tokenizer support. |
| src/mobius/integrations/gguf/_docs_test.py | Updates docs tests to assert only the expected architecture/evidence is runtime-supported. |
| src/mobius/integrations/gguf/_arch_registry.py | Enables runtime support for llama only via the single SmolLM F16 evidence ID. |
| src/mobius/integrations/gguf/init.py | Exposes pinned-source tokenizer types/helpers in the GGUF integration public API. |
| src/mobius/main.py | Adds CLI flags for pinned tokenizer repo/revision and enforces immutability/required pairing with --runtime. |
| docs/api/build_from_gguf.md | Updates public docs to describe pinned-source tokenizer route and no-replace atomic publication semantics. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+642
to
+650
| if path.is_symlink(): | ||
| try: | ||
| path.absolute().relative_to(Path(HF_HUB_CACHE).absolute()) | ||
| except ValueError as error: | ||
| raise ValueError( | ||
| f"Cached tokenizer asset is an untrusted symlink: {path}" | ||
| ) from error | ||
| path = path.resolve(strict=True) | ||
| else: |
Comment on lines
+143
to
+149
| or any( | ||
| filename != Path(filename).name | ||
| or size <= 0 | ||
| or len(sha256) != 64 | ||
| or not _is_hex(sha256) | ||
| for filename, size, sha256 in self.tokenizer_assets | ||
| ) |
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
## Summary - clean up still-valid typing, documentation, test robustness, error-message, maintainability, and performance findings left by the GGUF PR stack - preserve the behavioral fixes merged in #625-#632 - hash reused multi-GB GGUF sources once, at the final pre-publication integrity gate, while retaining cheap identity checks around staging Exact base: `2db9d33debdc254d879a51b14434c9a81c230f4f` Exact head: `4541ed2bc9d2ab4227484510b4a85b2d9113eb25` ## Reconstructed original 17-item low-priority tranche The persisted audit retained only the totals, so this list was reconstructed from the live threads and current source. All but the already-fixed #596 comment are addressed in this PR. | PR | Comment | Disposition | |---|---:|---| | #550 | 3837144656 | Implemented: correct tuple return annotation | | #552 | 3837389183 | Implemented: multichannel waveform shape docs | | #559 | 3837716261 | Implemented: name-based cache assertions | | #573 | 3854308350 | Implemented: one final GGUF hash, with integrity regression coverage | | #574 | 3854345521 | Implemented: `TensorRole \| None` typing | | #574 | 3854345597 | Implemented: removed obsolete verdict filtering | | #577 | 3854472859 | Implemented: documented SSM sequence length | | #578 | 3843112613 | Implemented: documented F64 passthrough | | #579 | 3854518332 | Implemented: generalized fused-projection error | | #580 | 3854576300 | Implemented: removed brittle node counts | | #583 | 3854681386 | Implemented: documented conditional draft outputs | | #587 | 3854816872 | Implemented: corrected MTP output contract docs | | #596 | 3846110059 | Already fixed on base: unambiguous GQA bias comment | | #600 | 3855174281 | Implemented: metadata-count-only MTP error | | #607 | 3855776048 | Implemented: stable route-field assertions | | #607 | 3855776086 | Implemented: public tensor iterator | | #609 | 3856486136 | Implemented: fail-closed LM-head comment | ## Current unresolved-thread disposition This covers all 48 Copilot threads returned by the reproducible #600-#630 query. The one human #623 thread is excluded. | PR | Comment | Current-main disposition and evidence | |---|---:|---| | #600 | 3855174177 | Already fixed by #629: package cycle and reserved-sidecar validation | | #600 | 3855174238 | Already fixed by #629: explicit MTP sidecar naming/loading | | #600 | 3855174281 | Implemented here: error no longer invents an observed block count | | #602 | 3848155961 | Outside exact stack; already fixed: top-level `expert_dtype` is classified before early return | | #602 | 3848155983 | Outside exact stack; still-valid behavioral block-quant validation, unchanged | | #602 | 3848156000 | Outside exact stack; still-valid truncated-read behavioral finding, unchanged | | #602 | 3848156022 | Outside exact stack; still-valid descriptor byte/dtype validation, unchanged | | #602 | 3848156040 | Outside exact stack; still-valid expert-bank payload validation, unchanged | | #603 | 3855249765 | Already fixed by #630: runtime preflight preserves shard sets | | #603 | 3855249840 | Already fixed by #630: success output follows durable runtime publication | | #604 | 3855343082 | Implemented here: graph-only MTP persistence distinguished from runtime rejection | | #604 | 3855343131 | Already fixed by #630: runtime success messages are atomic | | #607 | 3855776001 | Implemented here: missing generation golden skips before provenance read | | #607 | 3855776048 | Implemented here: only stable route fields are asserted | | #607 | 3855776086 | Implemented here: tensor count uses `tensor_items_raw()` | | #608 | 3856079371 | Still-valid behavioral cache-symlink containment finding; unchanged | | #608 | 3856079415 | Still-valid behavioral lowercase-digest validation finding; unchanged | | #609 | 3856486136 | Implemented here: comment matches value-preserving policy | | #610 | 3855541683 | Already fixed by #628: Falcon bias precedence is explicit | | #610 | 3855541761 | Already fixed by #628: CTRL tiny config exercises projection biases | | #611 | 3856595840 | Implemented here: runtime test resolves the distribution providing the module | | #612 | 3855677383 | Already fixed by #625: supported-version endianness detection | | #612 | 3855677427 | Implemented here: shared `INT64_MAX` sentinel | | #612 | 3855677460 | Implemented here: shared PLaMo2 width inference | | #612 | 3855677486 | Implemented here: accepted PLaMo2 activation spellings are explicit | | #613 | 3855845678 | Implemented here: canonical issue URL | | #613 | 3855845757 | Implemented here: Mamba-1 function-registration docs | | #613 | 3855845806 | Implemented here: test expects the canonical issue URL | | #614 | 3855988545 | Implemented here: removed stale Nemotron-H divergence comments | | #614 | 3855988597 | Already fixed by #628: zero-head geometry raises actionable `ValueError` | | #615 | 3856082290 | Already fixed by #626: dense GraniteHybrid bias closure | | #618 | 3856729330 | Implemented here: required routes filter ORT GenAI evidence | | #618 | 3856729409 | Implemented here: env-selected runtime version is authoritative | | #618 | 3856729490 | Stale/N/A: PR-description-only matrix claim; repository workflow claims one pinned version | | #618 | 3856729563 | Implemented here: schema tail restored to normal indentation | | #619 | 3856342484 | Already fixed on base: Kimi Linear uses `/issues/605` | | #619 | 3856342532 | Already fixed by #628: config rejects convolution kernels below 2 | | #619 | 3856342580 | Already fixed by #628: GGUF contract rejects convolution kernels below 2 | | #620 | 3855717041 | Already fixed by #627: tied LM-head-only checkpoints are retained | | #621 | 3856722324 | Already fixed by #628: Kimi-K3 required metadata is complete | | #623 | 3855931210 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939302 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939358 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939394 | N/A to current main: comment belongs to open, unmerged #623 | | #624 | 3856777152 | Implemented here: runtime compatibility reuses the emitted model type | | #625 | 3857049733 | Implemented here: unsupported header reports both endian candidates | | #629 | 3857313182 | Newer post-audit behavioral sidecar-symlink cleanup finding; unchanged | | #629 | 3857313251 | Newer post-audit cross-platform path-safety finding; unchanged | ## Validation - affected GGUF/package/ORT GenAI/model/schema tests: 1,040 passed - broad non-integration suite: 7,851 passed, 56 skipped, 1 subtest passed - generated GGUF docs checks: 7 passed - initialized `lintrunner`; full lint/format passed - GPT-5.6 Sol medium review: one integrity finding fixed; re-review found no significant issues Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
386 passedfocused GGUF/tokenizer/runtime/CLI/schema suite2658 passedfull GGUF + CLI suite7062 passed, 52 skippedbroad affected suite; only the two pre-existingglm_moe_dsashape-inference failures remain2 passedreal SmolLM/SmolLM2 runtime integration6 passedtargeted L4/L5 golden testsStacked on #607 via
justinchuby-validate-small-gguf-models.