Finalize generated GGUF support matrix - #604
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Comparison
|
justinchuby
force-pushed
the
justinchuby-close-gguf-tokenizer-census
branch
from
August 25, 2026 16:46
5a4757e to
bf14467
Compare
Base automatically changed from
justinchuby-close-gguf-tokenizer-census
to
main
August 25, 2026 16:48
Generate an exhaustive support matrix from the pinned llama.cpp census and lock it to the architecture, qtype, projector, and tokenizer registries. Fail runtime publication closed unless immutable source, import-route, graph, tokenizer, runtime configuration, and package evidence all match. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Apply the repository formatters to the generated support documentation and runtime-evidence tests after transplanting the PR onto the squash-merged parent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-finalize-gguf-support-matrix
branch
from
August 25, 2026 16:56
7326da4 to
2e0df5a
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR “finalizes” the GGUF support matrix by generating docs from pinned llama.cpp census data and by making GGUF runtime packaging explicitly fail-closed unless structured, immutable runtime evidence exists. It strengthens provenance/identity tracking across GGUF parsing, graph import-route capture, and package emission so the documentation and the runtime gate cannot drift.
Changes:
- Add structured runtime-evidence plumbing and enforce fail-closed runtime packaging (architecture verdict + immutable source identity + import-route + package hash checks).
- Introduce a generator-backed documentation pipeline for the GGUF architecture / qtype / projector / tokenizer matrices (plus CI-style staleness checks).
- Harden GGUF source identity handling (reader open-time identity check, logical source filename preservation, upstream pin date surfaced in docs).
Reviewed changes
Copilot reviewed 26 out of 27 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/gguf_test.py | Updates CLI/runtime packaging tests to account for new runtime gate inputs (arch + registry verdicts). |
| tests/cli_test.py | Adds coverage ensuring deferred runtime architectures fail before creating output directories. |
| src/mobius/integrations/gguf/_upstream.py | Exposes pinned upstream commit date for doc generation. |
| src/mobius/integrations/gguf/_upstream_data/llamacpp_pin.json | Records upstream commit_date alongside the pinned commit SHA. |
| src/mobius/integrations/gguf/_tokenizer.py | Adds source_identity override so tokenizer manifests don’t have to record machine-local paths. |
| src/mobius/integrations/gguf/_tokenizer_test.py | Switches doc-table validation to the centralized GGUF docs generator check. |
| src/mobius/integrations/gguf/_spec.py | Makes runtime default to DEFERRED and adds structured runtime evidence IDs validation when runtime is SUPPORTED. |
| src/mobius/integrations/gguf/_spec_test.py | Updates spec tests for the new runtime default + evidence ID requirements. |
| src/mobius/integrations/gguf/_runtime_package.py | Adds runtime-evidence gating, identity checks, and package hashing to fail closed on any mismatch. |
| src/mobius/integrations/gguf/_runtime_package_test.py | Adds tests for early rejection paths (deferred arch, MTP, save_model=False) and evidence-bound packaging. |
| src/mobius/integrations/gguf/_runtime_evidence.py | Introduces the structured runtime evidence schema + hashing/identity utilities. |
| src/mobius/integrations/gguf/_runtime_evidence_test.py | Tests evidence binding behavior and graph-package identity hashing (including symlink rejection). |
| src/mobius/integrations/gguf/_reader.py | Adds open-time stat-identity capture and a source_matches_path() check for later provenance binding. |
| src/mobius/integrations/gguf/_mmproj.py | Separates “graph-importable” vs “runtime-supported” projector gating. |
| src/mobius/integrations/gguf/_mmproj_registry.py | Adds is_importable and updates supported-projector listing to reflect graph importability. |
| src/mobius/integrations/gguf/_mmproj_registry_test.py | Moves projector-matrix doc drift checks to the centralized generator check. |
| src/mobius/integrations/gguf/_docs.py | Adds the authoritative GGUF docs generator and a strict staleness/pin guard. |
| src/mobius/integrations/gguf/_docs_test.py | Adds closure tests for doc generation counts, pin closure, and runtime-evidence fail-closed policy. |
| src/mobius/integrations/gguf/_builder.py | Captures import-route identity, logical source filename, and (when enabled) artifact identity for evidence binding. |
| src/mobius/integrations/gguf/_builder_test.py | Adds unit coverage for logical Hub-relative filename preservation. |
| src/mobius/integrations/gguf/_arch_registry.py | Downgrades runtime verdicts to DEFERRED with a consistent “runtime evidence pending” reason across specs. |
| src/mobius/integrations/gguf/_arch_registry_test.py | Switches architecture doc drift checks to the centralized generator check and updates guidance. |
| src/mobius/integrations/gguf/init.py | Updates public docs to reflect fail-closed runtime packaging status. |
| src/mobius/main.py | Adds runtime preflight gate and plumbs --runtime-version into runtime packaging. |
| scripts/generate_gguf_support_docs.py | Adds a script to refresh/check generated GGUF documentation blocks. |
| docs/cli_reference.md | Updates CLI documentation for the stricter runtime packaging gate and new options/semantics. |
| docs/api/build_from_gguf.md | Replaces hand-maintained matrices with generated blocks and expands policy/limitations documentation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+397
to
+402
| Runtime packaging requires a validated embedded `tokenizer.huggingface.json`; | ||
| opaque tokenizer pre-types are never reconstructed. Deferred/rejected | ||
| architecture, tokenizer, draft-pairing, or mmproj checks run before durable | ||
| output. Multimodal packages use `decoder`, `vision_encoder`, optional | ||
| `audio_encoder`, and `embedding`; an admitted trailing MTP head is persisted | ||
| under `mtp/`. |
Comment on lines
724
to
728
| gguf_path, | ||
| output_dir, | ||
| runtime=runtime, | ||
| runtime_version=getattr(args, "runtime_version", None), | ||
| external_data=args.external_data, |
justinchuby
added a commit
that referenced
this pull request
Aug 25, 2026
## Summary - clean up still-valid typing, documentation, test robustness, error-message, maintainability, and performance findings left by the GGUF PR stack - preserve the behavioral fixes merged in #625-#632 - hash reused multi-GB GGUF sources once, at the final pre-publication integrity gate, while retaining cheap identity checks around staging Exact base: `2db9d33debdc254d879a51b14434c9a81c230f4f` Exact head: `4541ed2bc9d2ab4227484510b4a85b2d9113eb25` ## Reconstructed original 17-item low-priority tranche The persisted audit retained only the totals, so this list was reconstructed from the live threads and current source. All but the already-fixed #596 comment are addressed in this PR. | PR | Comment | Disposition | |---|---:|---| | #550 | 3837144656 | Implemented: correct tuple return annotation | | #552 | 3837389183 | Implemented: multichannel waveform shape docs | | #559 | 3837716261 | Implemented: name-based cache assertions | | #573 | 3854308350 | Implemented: one final GGUF hash, with integrity regression coverage | | #574 | 3854345521 | Implemented: `TensorRole \| None` typing | | #574 | 3854345597 | Implemented: removed obsolete verdict filtering | | #577 | 3854472859 | Implemented: documented SSM sequence length | | #578 | 3843112613 | Implemented: documented F64 passthrough | | #579 | 3854518332 | Implemented: generalized fused-projection error | | #580 | 3854576300 | Implemented: removed brittle node counts | | #583 | 3854681386 | Implemented: documented conditional draft outputs | | #587 | 3854816872 | Implemented: corrected MTP output contract docs | | #596 | 3846110059 | Already fixed on base: unambiguous GQA bias comment | | #600 | 3855174281 | Implemented: metadata-count-only MTP error | | #607 | 3855776048 | Implemented: stable route-field assertions | | #607 | 3855776086 | Implemented: public tensor iterator | | #609 | 3856486136 | Implemented: fail-closed LM-head comment | ## Current unresolved-thread disposition This covers all 48 Copilot threads returned by the reproducible #600-#630 query. The one human #623 thread is excluded. | PR | Comment | Current-main disposition and evidence | |---|---:|---| | #600 | 3855174177 | Already fixed by #629: package cycle and reserved-sidecar validation | | #600 | 3855174238 | Already fixed by #629: explicit MTP sidecar naming/loading | | #600 | 3855174281 | Implemented here: error no longer invents an observed block count | | #602 | 3848155961 | Outside exact stack; already fixed: top-level `expert_dtype` is classified before early return | | #602 | 3848155983 | Outside exact stack; still-valid behavioral block-quant validation, unchanged | | #602 | 3848156000 | Outside exact stack; still-valid truncated-read behavioral finding, unchanged | | #602 | 3848156022 | Outside exact stack; still-valid descriptor byte/dtype validation, unchanged | | #602 | 3848156040 | Outside exact stack; still-valid expert-bank payload validation, unchanged | | #603 | 3855249765 | Already fixed by #630: runtime preflight preserves shard sets | | #603 | 3855249840 | Already fixed by #630: success output follows durable runtime publication | | #604 | 3855343082 | Implemented here: graph-only MTP persistence distinguished from runtime rejection | | #604 | 3855343131 | Already fixed by #630: runtime success messages are atomic | | #607 | 3855776001 | Implemented here: missing generation golden skips before provenance read | | #607 | 3855776048 | Implemented here: only stable route fields are asserted | | #607 | 3855776086 | Implemented here: tensor count uses `tensor_items_raw()` | | #608 | 3856079371 | Still-valid behavioral cache-symlink containment finding; unchanged | | #608 | 3856079415 | Still-valid behavioral lowercase-digest validation finding; unchanged | | #609 | 3856486136 | Implemented here: comment matches value-preserving policy | | #610 | 3855541683 | Already fixed by #628: Falcon bias precedence is explicit | | #610 | 3855541761 | Already fixed by #628: CTRL tiny config exercises projection biases | | #611 | 3856595840 | Implemented here: runtime test resolves the distribution providing the module | | #612 | 3855677383 | Already fixed by #625: supported-version endianness detection | | #612 | 3855677427 | Implemented here: shared `INT64_MAX` sentinel | | #612 | 3855677460 | Implemented here: shared PLaMo2 width inference | | #612 | 3855677486 | Implemented here: accepted PLaMo2 activation spellings are explicit | | #613 | 3855845678 | Implemented here: canonical issue URL | | #613 | 3855845757 | Implemented here: Mamba-1 function-registration docs | | #613 | 3855845806 | Implemented here: test expects the canonical issue URL | | #614 | 3855988545 | Implemented here: removed stale Nemotron-H divergence comments | | #614 | 3855988597 | Already fixed by #628: zero-head geometry raises actionable `ValueError` | | #615 | 3856082290 | Already fixed by #626: dense GraniteHybrid bias closure | | #618 | 3856729330 | Implemented here: required routes filter ORT GenAI evidence | | #618 | 3856729409 | Implemented here: env-selected runtime version is authoritative | | #618 | 3856729490 | Stale/N/A: PR-description-only matrix claim; repository workflow claims one pinned version | | #618 | 3856729563 | Implemented here: schema tail restored to normal indentation | | #619 | 3856342484 | Already fixed on base: Kimi Linear uses `/issues/605` | | #619 | 3856342532 | Already fixed by #628: config rejects convolution kernels below 2 | | #619 | 3856342580 | Already fixed by #628: GGUF contract rejects convolution kernels below 2 | | #620 | 3855717041 | Already fixed by #627: tied LM-head-only checkpoints are retained | | #621 | 3856722324 | Already fixed by #628: Kimi-K3 required metadata is complete | | #623 | 3855931210 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939302 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939358 | N/A to current main: comment belongs to open, unmerged #623 | | #623 | 3855939394 | N/A to current main: comment belongs to open, unmerged #623 | | #624 | 3856777152 | Implemented here: runtime compatibility reuses the emitted model type | | #625 | 3857049733 | Implemented here: unsupported header reports both endian candidates | | #629 | 3857313182 | Newer post-audit behavioral sidecar-symlink cleanup finding; unchanged | | #629 | 3857313251 | Newer post-audit cross-platform path-safety finding; unchanged | ## Validation - affected GGUF/package/ORT GenAI/model/schema tests: 1,040 passed - broad non-integration suite: 7,851 passed, 56 skipped, 1 subtest passed - generated GGUF docs checks: 7 passed - initialized `lintrunner`; full lint/format passed - GPT-5.6 Sol medium review: one integrity finding fixed; re-review found no significant issues Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
8d9af256337d1a501250f9bbf4c0859a654bddd6(2026-08-23T16:59:42Z)Final verdicts
Q2_0deferred because pinned gguf-py has no dequantizergemma4v,muse-glimmer); 0 runtime-supportedtokenizer.huggingface.jsonis presentLocal validation
python scripts/generate_gguf_support_docs.py --checklintrunner f --output oneline --all-fileslintrunner -aglm_moe_dsabaseline excluded)Waivers and limitations
This PR closes importer/registry/documentation policy; it does not add a new model architecture or claim downstream runtime readiness.
Stacked on #603 /
justinchuby-close-gguf-tokenizer-census.