Skip to content

Add specialized GGUF encoder architectures - #634

Merged
justinchuby merged 6 commits into
mainfrom
justinchuby-special-gguf-architectures
Aug 25, 2026
Merged

Add specialized GGUF encoder architectures#634
justinchuby merged 6 commits into
mainfrom
justinchuby-special-gguf-architectures

Conversation

@justinchuby

@justinchuby justinchuby commented Aug 25, 2026

Copy link
Copy Markdown
Member

Summary

Promotes four stateless specialized GGUF encoder architectures from the pinned llama.cpp census:

  • eurobert: pre-RMSNorm split-QKV RoPE encoder with parallel SwiGLU
  • neo-bert: fused contiguous QKV, interleaved RoPE, fused SwiGLU, and CLS pooling
  • nomic-bert (dense only): post-norm RoPE encoder with gated FFN
  • jina-bert-v2: ALiBi encoder with optional Q/K norms, optional second attention norm, and fused/separate GeGLU

All four use a dedicated GGUF feature-extraction task supporting NONE, MEAN, and CLS pooling. Tensor closure and shape validation are fail-closed. Quantized import is rejected until complete fused/split packed-weight routes exist; runtime packaging remains deferred pending representative real-weight evidence.

Authoritative evidence

  • llama.cpp pin: 8d9af256337d1a501250f9bbf4c0859a654bddd6
  • HF config pins:
    • EuroBERT/EuroBERT-210m@39b51e15dd1f1a06f58b5cbf6a8a188cec60bd0e
    • chandar-lab/NeoBERT@5424c8efeea6491b151d62dee55a752165407430
    • nomic-ai/nomic-embed-text-v1.5@e9b6763023c676ca8431644204f50c2b100d9aab
    • jinaai/jina-embeddings-v2-small-en@44e7d1d6caec8c883c2d4b207588504d519788d0

Tests cover tiny graph construction, pinned config transforms, exact tensor mapping/closure, contiguous NeoBERT QKV and interleaved RoPE, tanh GeGLU, FP16/BF16 Jina ALiBi graph loading, pooling ABI, left-padded CLS value selection, and float-only quantization policy.

Residual deferred architectures

  • jina-bert-v3: released artifacts require adapter handling; MoE scheduling is not covered
  • nomic-bert-moe: encoder MoE schedule/experts remain unsupported
  • talkie: dedicated gain/skip/logit-scale graph and value transforms required
  • wavtokenizer-dec: requires a code-token-to-ISTFT-feature task and waveform processor
  • rwkv6, rwkv6qwen2, rwkv7, arwkv7: token-shift plus matrix-state ABI is not KV/Mamba state
  • bailingmoe3: mixed KDA convolution/matrix state plus MLA cache
  • deepseek4: compressed cache/history/rollback ABI is not represented by the fallback graph
  • pockettts, qwen3tts: primary GGUFs require mandatory sidecar bundles and non-text orchestration

Validation

  • Shared registry/build/task/runtime/docs/coverage suite before the formatter-only follow-up: 4146 passed, 284 skipped, 16 deselected
  • Directly affected conventional decoder/docs tests after formatting: 53 passed
  • Broad non-integration suite before the formatter-only follow-up: 7983 passed, 57 skipped
  • Generated GGUF docs: exact generator output
  • Full initialized lintrunner f --output oneline --all-files: success and zero worktree changes after commit 4e1059eb
  • GPT-5.6 Sol medium review of the formatter-only follow-up: clean; semantic equivalence confirmed
  • Exact base: 3b2c9cd624c5f1e58a5948fb7e6af6453ea64ff2 (Add conventional dense GGUF architectures #635)
  • Head: 4e1059eb6516ba40d7e059a799ab47d1f420835e

The post-format monolithic shared rerun was attempted twice, but the host dropped to 100–115 MiB free and pytest could no longer create temporary files (FileNotFoundError: No usable temporary directory; OpenMP /tmp allocation failures). The formatting-only affected tests pass, and the identical shared suite passed immediately before the semantic no-op formatting commit.

Waivers

  • Generic L2/L4/L5: the four registered module types are GGUF-only and have no native HF model_type route. Pinned HF-to-GGUF config semantics and synthetic ORT parity are covered by _specialized_encoders_test.py; misleading generic test_model_id claims are intentionally omitted.
  • Real-weight runtime generation: these are encoder-only GGUF routes and runtime support remains explicitly deferred; no runtime capability is claimed.
  • Quantized parity/Olive: packed fused/split projection ownership is intentionally rejected, not silently dequantized under keep_quantized=True.
  • Exact-main coverage baseline: kimi_k3 and lfm2_moe already have test IDs without YAML cases on Add conventional dense GGUF architectures #635; the shared run excludes those two cases and their aggregate sentinel. Specialized coverage checks pass.
  • Existing SmolLM OGA 0.15.2 evidence probe has a baseline route-hash mismatch (3ca3… produced vs b999… pinned) that reproduces unchanged on untouched Fix atomic GGUF runtime CLI publication #630; this branch preserves the baseline route fingerprint and does not update evidence.

@justinchuby
justinchuby requested review from a team and a lite review from Copilot August 25, 2026 22:10
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

Performance Comparison

Comparing 3b2c9cd4e1059e

Model Metric Baseline Current Delta
bert (feature-extraction) model_size_bytes 359 KB 359 KB +0.0%
bert (feature-extraction) num_nodes 68 68 +0.0%
falcon model_size_bytes 364 KB 364 KB +0.0%
falcon num_nodes 66 66 +0.0%
gemma2 model_size_bytes 428 KB 428 KB +0.0%
gemma2 num_nodes 105 105 +0.0%
gpt2 model_size_bytes 324 KB 324 KB +0.0%
gpt2 num_nodes 54 54 +0.0%
llama model_size_bytes 425 KB 425 KB +0.0%
llama num_nodes 60 60 +0.0%
llama (static-cache) model_size_bytes 425 KB 425 KB +0.0%
llama (static-cache) num_nodes 56 56 +0.0%
mamba (ssm-text-generation) model_size_bytes 296 KB 296 KB +0.0%
mamba (ssm-text-generation) num_nodes 94 94 +0.0%
phi3 model_size_bytes 421 KB 421 KB +0.0%
phi3 num_nodes 58 58 +0.0%
phi3 (static-cache) model_size_bytes 421 KB 421 KB +0.0%
phi3 (static-cache) num_nodes 54 54 +0.0%
qwen2 model_size_bytes 425 KB 425 KB +0.0%
qwen2 num_nodes 60 60 +0.0%
qwen2 (static-cache) model_size_bytes 425 KB 425 KB +0.0%
qwen2 (static-cache) num_nodes 56 56 +0.0%
qwen3_5_moe (hybrid-text-generation) model_size_bytes 506 KB 506 KB +0.0%
qwen3_5_moe (hybrid-text-generation) num_nodes 265 265 +0.0%
qwen3_5_text (hybrid-text-generation) model_size_bytes 458 KB 458 KB +0.0%
qwen3_5_text (hybrid-text-generation) num_nodes 127 127 +0.0%
qwen3_5_vl (hybrid-qwen-vl) model_size_bytes 977 KB 977 KB +0.0%
qwen3_5_vl (hybrid-qwen-vl) num_nodes 429 429 +0.0%
t5 (seq2seq) model_size_bytes 836 KB 836 KB +0.0%
t5 (seq2seq) num_nodes 176 176 +0.0%
whisper (speech-to-text) model_size_bytes 1008 KB 1008 KB +0.0%
whisper (speech-to-text) num_nodes 128 128 +0.0%

No performance regressions.

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

🏗️ Architecture Diff

Comparing 3b2c9cd4e1059e

Model Sub-model Changes Status
bert (feature-extraction) model 0
falcon model 0
gemma2 model 0
gemma4 (gemma4) decoder 0
gemma4 (gemma4) embedding 0
gemma4 (gemma4) vision_encoder 0
gemma4_text model 0
gpt2 model 0
llama model 0
llama (static-cache) model 0
mamba (ssm-text-generation) model 0
phi3 model 0
phi3 (static-cache) model 0
qwen model 0
qwen (static-cache) model 0
qwen2 model 0
qwen2 (static-cache) model 0
qwen2_moe model 0
qwen2_moe (static-cache) model 0
qwen3 model 0
qwen3 (static-cache) model 0
qwen3_5_moe (hybrid-text-generation) model 0
qwen3_5_text (hybrid-text-generation) model 0
qwen3_5_vl (hybrid-qwen-vl) decoder 0
qwen3_5_vl (hybrid-qwen-vl) embedding 0
qwen3_5_vl (hybrid-qwen-vl) vision_encoder 0
qwen3_moe model 0
qwen3_moe (static-cache) model 0
qwen3_next (hybrid-text-generation) model 0
t5 (seq2seq) decoder 0
t5 (seq2seq) encoder 0
whisper (speech-to-text) decoder 0
whisper (speech-to-text) encoder 0

No architecture changes detected.


Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds four specialized stateless GGUF encoder architectures (EuroBERT, NeoBERT, dense NomicBERT, JinaBERT-v2) to mobius’ GGUF import pipeline, including a dedicated feature-extraction task that matches the GGUF pooling ABI and strict tensor-closure validation.

Changes:

  • Add gguf_encoders.py encoder model implementations and register them in the mobius model registry.
  • Introduce GGUFEncoderFeatureExtractionTask and wire it through task registration and GGUF builder dispatch.
  • Extend GGUF architecture registry, config postprocessing, tensor mapping, upstream pin inventory, docs, and add targeted tests for the new encoders.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/_test_configs.py Adds tiny-config coverage entries for the new GGUF encoder module types.
src/mobius/tasks/_feature_extraction.py Allows task-defined output naming; adds GGUFEncoderFeatureExtractionTask for GGUF pooling ABI.
src/mobius/tasks/init.py Exports/registers GGUFEncoderFeatureExtractionTask and task-name mapping.
src/mobius/models/gguf_encoders.py New specialized encoder model graphs (RoPE/ALiBi variants + pooling implementation).
src/mobius/models/init.py Exports new GGUF encoder model classes.
src/mobius/integrations/gguf/_upstream_data/llamacpp_pin.json Adds pinned tensor inventories for the newly promoted encoder architectures.
src/mobius/integrations/gguf/_tensor_mapping.py Adds GGUF→HF tensor stem mappings for the new encoders.
src/mobius/integrations/gguf/_specialized_encoders_test.py New tests validating config transforms, closure, mapping, pooling ABI, and graph properties.
src/mobius/integrations/gguf/_config_mapping.py Adds specialized-encoder postprocess resolving optional tensor-selected variants and pooling type.
src/mobius/integrations/gguf/_builder.py Wires specialized closure validation + dispatches to the GGUF encoder feature-extraction task.
src/mobius/integrations/gguf/_arch_registry.py Promotes four encoder architectures to supported graph import with specialized registry entries.
src/mobius/integrations/gguf/_arch_registry_test.py Updates expected supported counts and registry/task resolution assertions.
src/mobius/_registry.py Registers new module types to models + default GGUF encoder feature-extraction task.
src/mobius/_configs/_base.py Adds encoder-specific config fields needed for specialized GGUF variants.
docs/api/build_from_gguf.md Updates support matrix and docs to reflect promoted encoder architectures and quantized-import policy.
Suppressed comments (1)

src/mobius/integrations/gguf/_builder.py:3831

  • For specialized encoder GGUF architectures, build_from_gguf() validates task overrides against only "feature-extraction", but the code later auto-selects GGUFEncoderFeatureExtractionTask and the task is publicly registered as "gguf-encoder-feature-extraction". As a result, passing task="gguf-encoder-feature-extraction" will be rejected even though it’s the intended task for these architectures.
        expected_task = (
            "t5-text-encoding" if model_type == "t5encoder" else "feature-extraction"
        )
        if task is not None and task != expected_task:
            raise ValueError(

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Implement exact float GGUF import paths for EuroBERT, NeoBERT, dense NomicBERT, and JinaBERT-v2 with strict config and tensor closure validation. Keep quantized import and runtime packaging fail-closed until real artifact evidence exists.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Copilot-Session: c9f552d7-041b-482e-b1a6-7227d980ed5f
Correct NeoBERT per-head QKV packing and interleaved RoPE, use JinaBERT-v2 tanh GELU, reject unsupported pooling, and preserve established GGUF route fingerprints.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Copilot-Session: c9f552d7-041b-482e-b1a6-7227d980ed5f
Apply the initialized repository formatter after the post-rebase validation pass.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Copilot-Session: c9f552d7-041b-482e-b1a6-7227d980ed5f
Implement GGUF NONE/MEAN/CLS pooling through a dedicated encoder task and cast JinaBERT-v2 ALiBi bias to the model dtype for FP16/BF16 graph loading.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Copilot-Session: c9f552d7-041b-482e-b1a6-7227d980ed5f
Use contiguous full-width Q/K/V views from pinned build_qkv and select the first valid sequence token for CLS pooling, including a left-padding value test.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Copilot-Session: c9f552d7-041b-482e-b1a6-7227d980ed5f
@justinchuby
justinchuby force-pushed the justinchuby-special-gguf-architectures branch from 0960104 to 780267a Compare August 25, 2026 22:44
Apply the repository-pinned Ruff formatter to the Xverse permutation helper added by #635 so all-files lint is idempotent on the specialized branch.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c9f552d7-041b-482e-b1a6-7227d980ed5f
Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
@justinchuby
justinchuby merged commit 6ff8f08 into main Aug 25, 2026
10 of 17 checks passed
@justinchuby
justinchuby deleted the justinchuby-special-gguf-architectures branch August 25, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants