Looking for the guided, from-scratch journey? Zhongkai Fu's From Tensors to Tokens uses Gemma 4 E4B to connect tensor foundations, multimodal model execution, and a complete TensorSharp inference application. View the paperback on Amazon.
| Property | Value |
|---|---|
| Provider | |
| GGUF architecture key | gemma4 |
| Source class | Gemma4Model (legacy per-seq) + Gemma4Model.BatchedForward.cs (IBatchedPagedModel) |
| Vision encoder | Gemma4VisionEncoder (SigLIP-style ViT) |
| Audio encoder | Gemma4AudioEncoder (USM-style chunked transformer) |
| Audio frontend | Gemma4AudioPreprocessor (16 kHz mono → 128-bin log-mel) |
| Image processor | Gemma4ImageProcessor |
| Example models | gemma-4-E4B (8B effective), gemma-4-12B, gemma-4-31B, gemma-4-26B-A4B (MoE) |
| Modalities | Text, image, video (frame stack), audio |
| Thinking mode | Yes (<|channel>thought ... <channel|>) |
| Tool calling | Yes (<|tool_call>call:name{...}<tool_call|>) |
| Batched / paged forward | Default-on — IBatchedPagedModel.ForwardBatch with per-layer paged K/V (handles dual head dims, KV donor sharing, PLE injection, SWA + global mix). Set TS_GEMMA4_BATCHED=0 to force the legacy per-seq KV-swap path. See §11. |
| MTP speculative decoding | Optional — loads a separate gemma4-assistant EAGLE-style draft GGUF via --draft-model (TS_SPEC_DRAFT_MODEL); naming the file enables speculation by itself (an explicit --no-spec vetoes it). The flag works on either host: TensorSharp.Cli and TensorSharp.Server share SpeculativeCliFlags. Profitable on ggml backends and the pure-C# cuda backend. See §12. |
| Output parser | Gemma4OutputParser |
Verified GGUF pointers:
| Model | HF repo | Recommended file | mmproj (image / video / audio) | MTP draft |
|---|---|---|---|---|
| gemma-4-E4B-it | ggml-org/gemma-4-E4B-it-GGUF | gemma-4-E4B-it-Q8_0.gguf (8.031 GB; also gemma-4-E4B-it-Q4_0.gguf, 4.588 GB) |
mmproj-gemma-4-E4B-it-Q8_0.gguf (0.560 GB; same repo; alt: mmproj-F16.gguf, 0.990 GB, in unsloth/gemma-4-E4B-it-GGUF) |
gemma-4-E4B-it-assistant.Q8_0.gguf (0.100 GB) from AtomicChat/gemma-4-E4B-it-assistant-GGUF |
| gemma-4-12B-it (QAT) | unsloth/gemma-4-12B-it-qat-GGUF | gemma-4-12B-it-qat-UD-Q4_K_XL.gguf (6.716 GB) |
mmproj-BF16.gguf (0.175 GB; same repo) |
mtp-gemma-4-12B-it.gguf (0.254 GB; same repo root; quantized variants under MTP/) |
| gemma-4-26B-A4B-it (MoE, QAT) | unsloth/gemma-4-26B-A4B-it-qat-GGUF | gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf (14.249 GB) |
mmproj-BF16.gguf (1.195 GB; same repo) |
mtp-gemma-4-26B-A4B-it.gguf (0.252 GB; same repo), or gemma-4-26B-A4B-it-assistant.Q8_0.gguf (0.462 GB) from AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF |
| gemma-4-26B-A4B-it (MoE) | ggml-org/gemma-4-26B-A4B-it-GGUF | gemma-4-26B-A4B-it-Q4_K_M.gguf (16.796 GB) or gemma-4-26B-A4B-it-Q8_0.gguf (26.860 GB) |
mmproj-gemma-4-26B-A4B-it-Q8_0.gguf (0.806 GB) / mmproj-gemma-4-26B-A4B-it-bf16.gguf (1.195 GB) |
same drafts as the QAT row |
| gemma-4-31B-it | ggml-org/gemma-4-31B-it-GGUF | gemma-4-31B-it-Q4_K_M.gguf (18.687 GB) or gemma-4-31B-it-Q8_0.gguf (32.636 GB) |
mmproj-gemma-4-31B-it-Q8_0.gguf (0.810 GB) / mmproj-gemma-4-31B-it-bf16.gguf (1.201 GB) |
— |
Hugging Face identifies every row above as derived from the matching Google Gemma 4 base model. Some conversion cards do not declare a license; anonymous download does not replace the upstream model terms, so review the linked base model and conversion card before redistribution.
The MTP draft head ships as a separate gemma4-assistant GGUF whose backbone
dimension must equal the target's hidden size — always pair draft and target
from the same size (E4B draft ↔ E4B target, 12B ↔ 12B, 26B-A4B ↔ 26B-A4B).
A mismatched draft fails fast at server startup (§12.2).
Command-line download (one line per file; requires pip install -U huggingface_hub):
python -m pip install -U huggingface_hub
hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
hf download ggml-org/gemma-4-E4B-it-GGUF mmproj-gemma-4-E4B-it-Q8_0.gguf --local-dir models
hf download unsloth/gemma-4-12B-it-qat-GGUF gemma-4-12B-it-qat-UD-Q4_K_XL.gguf --local-dir models
hf download unsloth/gemma-4-12B-it-qat-GGUF mtp-gemma-4-12B-it.gguf --local-dir modelsThis is TensorSharp's verified quick-start lane. The E4B Q8_0 family/path
uses the public
ggml-org/gemma-4-E4B-it-GGUF
artifact, a normal native build, and a native GGML GPU backend. Verification is
at the E4B Q8_0 family/path level; it does not assert that a particular public
file checksum was the benchmark input. This copy/paste block is for
Linux + NVIDIA; platform-specific backend choices follow it. On a new machine,
install and verify the .NET 10 SDK
before running the build command; the runtime alone is not sufficient.
python -m pip install -U huggingface_hub
hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
printf '%s\n' 'Answer in one short sentence: what is TensorSharp?' > prompt.txt
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gemma-4-E4B-it-Q8_0.gguf \
--input prompt.txt --max-tokens 64 --backend ggml_cuda
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model models/gemma-4-E4B-it-Q8_0.gguf \
--backend ggml_cuda --max-tokens 128Use ggml_cuda on Windows/Linux with NVIDIA, ggml_metal on Apple Silicon,
or ggml_vulkan on Windows/Linux with a Vulkan-capable AMD, Intel, or NVIDIA
GPU. The matching mmproj-gemma-4-E4B-it-Q8_0.gguf is optional for text-only
requests; download it from the same repository and pass --mmproj for image,
video, or audio input.
The fast-path claim is backed by three independent routing details:
- Multi-token prefill/verify runs through the fused whole-model
NativeGemma4ModelVerifygraph, including E-series in-kernel PLE gathering and shared-KV donor handling. - Single-token dense decode runs the complete transformer through
NativeGemma4ModelDecodein one GGML graph dispatch. - With one scheduled sequence, the default
TS_BATCHED_N1_FAST_PATH=1scheduler route selects the linearForward()path that reaches that fused whole-model decode instead of the general batched per-op route.
See the engine-comparison report, the measured E4B prefill performance record, and the N=1 scheduler documentation.
Multimodal CLI one-shot (the text prompt comes from a file via --input; with
--image and no --input a default describe-the-image prompt is used; CLI
sampling defaults to greedy and --max-tokens defaults to 100):
dotnet run --project TensorSharp.Cli -c Release -- --model models/gemma-4-E4B-it-Q8_0.gguf \
--mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf \
--image photo.png --max-tokens 512 --backend ggml_cudaWith MTP speculative decoding. --draft-model is parsed by the
same SpeculativeCliFlags
on both hosts, so the spelling is identical on TensorSharp.Cli and
TensorSharp.Server. Naming the draft GGUF enables speculation by itself — no
--spec is needed beside it (an explicit --no-spec vetoes it). It must be on
the command line that loads the model — the draft GGUF is
attached to the target at startup, and a --draft-model that cannot be
activated is a fail-fast startup error rather than a silent fallback (§12.2):
# server
dotnet run --project TensorSharp.Server.Host -c Release -- --model models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
--backend ggml_cuda --draft-model models/mtp-gemma-4-12B-it.gguf
# CLI — speculation engages on --input, --input-jsonl, --multi-turn-jsonl and --interactive
dotnet run --project TensorSharp.Cli -c Release -- --model models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
--backend ggml_cuda --draft-model models/mtp-gemma-4-12B-it.gguf --input prompt.txt --max-tokens 512Then open http://localhost:5000 for the chat UI.
Gemma 4 is the most feature-rich architecture currently supported by TensorSharp. It pushes Google's Gemma line into:
- Per-layer heterogeneity: each layer can independently choose its attention pattern (SWA / global), its head dimension, its KV head count, and whether it shares KVs with an earlier "donor" layer.
- Per-Layer Embedding (PLE): a per-layer side embedding pulled from a
small
per_layer_token_embd.weighttable is mixed into the residual stream inside every block. - Mixture-of-Experts variants: e.g.
gemma-4-26B-A4B, where every block runs a dense MLP and a sparse MoE branch in parallel and adds them after separate post-norms. - True multimodality: vision, video frame stacks, and audio all flow into the same residual stream.
The SWA mask and RoPE caches are precomputed across all layers in a forward pass, the SWA cache is a circular buffer (so memory is bounded regardless of context length), and the entire transformer can be executed as a single fused GGML graph during decode.
tokens (int[])
│
token_embd.weight × sqrt(hidden)
│
[optional] InjectVisionEmbeddings (image / video frames)
[optional] InjectVisionEmbeddings (audio embeddings; same op)
│
[if PLE] ComputePLE(tokens, hidden) ──► perLayerInputs
│
┌────── × NumLayers ─────────────────────────────────────┐
│ HeadDim, KVHeads, attn pattern (SWA/global) per layer │
│ RMSNorm(attn_norm) │
│ QKV (fused or Q-only on KV-shared layers) │
│ per-head RMSNorm(Q,K) + unweighted RMSNorm(V) │
│ RoPE (NeoX local OR global+proportional/partial) │
│ Attention (SWA window OR full causal) │
│ attn_output ─► RMSNorm(post_attn_norm) + residual │
│ │
│ if MoE layer: │
│ RMSNorm(ffn_norm) ─► GeGLU(dense MLP) ─► PostNorm1 │
│ RMSNorm(pre_ffw_norm_2) ─► MoE(route+experts) │
│ ─► PostNorm2 │
│ residual += PostNorm(PostNorm1 + PostNorm2) │
│ else: │
│ RMSNorm(ffn_norm) ─► GeGLU ─► RMSNorm(post_ffw) │
│ residual += branch │
│ │
│ if PLE: │
│ residual += proj(GELU(inp_gate(hidden)) * pleInput) │
│ │
│ hidden *= layer_output_scale[layer] │
└─────────────────────────────────────────────────────────┘
│
RMSNorm(output_norm)
│
LM head (output.weight or tied)
│
[optional] tanh-softcap
│
▼
logits
hidden ─► RMSNorm(attn_norm)
─► QKV matmul (fused weight) ─► split into Q [seq, qDim],
K [seq, kvDim],
V [seq, kvDim]
─► per-head RMSNorm(Q) * attn_q_norm.weight
─► per-head RMSNorm(K) * attn_k_norm.weight
─► unweighted RMSNorm(V) // V-norm: weight ≡ 1
─► RoPE(Q, K, freqs[layer])
• local layers: standard NeoX RoPE on full headDim
• global layers: NeoX RoPE on the first _partialRotaryDims dims
with proportional frequency factors from rope_freqs.weight
─► Q ← Q * (1/sqrt(headDim))
─► append (K, V) to per-layer cache (circular for SWA)
─► attention(Q, K_cache, V_cache,
window = slidingWindow if SWA else totalSeq)
─► attn_output matmul → o
─► RMSNorm(post_attn_norm)
─► residual = hidden + o
─► h2 = RMSNorm(ffn_norm) on residual
─► ffn_gate_up matmul → [gate ‖ up]
─► g = GELU(gate)
─► h3 = ffn_down × (g * up)
─► RMSNorm(post_ffw_norm)
─► residual += h3
─► (PLE branch — see § 4.4)
─► hidden = residual * layer_output_scale[layer]
... attention block as above ...
# Dense MLP branch
b1 = RMSNorm(ffn_norm) on residual
b1 = GeGLU(b1) using ffn_gate_up.weight + ffn_down.weight
b1 = RMSNorm(post_ffw_norm_1) on b1
# MoE branch
b2 = RMSNorm(pre_ffw_norm_2) on residual
logits = ffn_gate_inp.weight × b2
logits = unweighted_RMSNorm(logits) * ffn_gate_inp.scale # learned scale
weights, idx = topK(softmax(logits), _numExpertsUsed)
b2 = weighted_sum_experts(SwiGLU on b2 using ffn_gate_up_exps[idx] +
ffn_down_exps[idx])
b2 = RMSNorm(post_ffw_norm_2) on b2
# combine
combined = RMSNorm(post_ffw_norm)(b1 + b2)
residual += combined
- Decode (
seqLen == 1) on a GGML backend with all dense layers and all weights quantized: a single native call (Gemma4ModelDecode) processes the whole stack, including PLE, per-layer head dims, circular SWA caches, and per-layer scalars in one GPU graph dispatch. - Prefill (
seqLen > 1): eligible dense models first use the fused whole-model verify graph, including E-series PLE and shared-KV donor layers. If that gate is unavailable, each eligible dense, non-shared, non-PLE layer can still use the per-layerGemma4LayerPrefillgraph; remaining layers use the per-op path. Long prompts are chunked to bound SWA score tensors.
_slidingWindowPattern[layer]fromgemma4.attention.sliding_window_pattern.IsLocalLayer(layer)returns this entry.true⇒ SWA layer;false⇒ full causal layer._localHeadDimfromgemma4.attention.key_length_swa(default 256),_globalHeadDimfromgemma4.attention.key_length(default 512).HeadDimForLayer(layer)selects which one to use._numGlobalKVHeadsfromgemma4.attention.global_head_count_kv. Local layers useConfig.NumKVHeads.KVHeadsForLayer(layer)resolves the count.DetectHeadDimsFromWeights()rebinds the head dims when the GGUF metadata contradicts the actual attention weight shapes.
The last _sharedKVLayers layers (from
gemma4.attention.shared_kv_layers) reuse another layer's KV cache. The
mapping is built by BuildKVDonorMap(); shared layers project only Q and skip
K/V matmuls and cache writes. The shared cache implementation:
_kvCacheK[shared] = _kvCacheK[donor](alias).- During chunked prefill, the donor's freshly-computed K and V are stashed in
_prefillSWAKVso KV-shared SWA layers can attend to the full chunk's K/V rather than the partial rolling cache.
When gemma4.embedding_length_per_layer_input > 0:
perLayerEmbeddings = lookup(per_layer_token_embd.weight, tokens)
perLayerEmbeddings = RMSNorm(per_layer_proj_norm.weight) on perLayerEmbeddings
perLayerEmbeddings = perLayerEmbeddings × per_layer_model_proj.weight^T
# inside each layer:
ple = GELU(inp_gate.weight × hidden) * extract_layer_slice(perLayerInputs, l)
ple = proj.weight × ple
ple = RMSNorm(post_norm.weight) on ple
residual += ple
ComputePLE() runs the full PLE pipeline once per forward pass and returns
[seqLen, NumLayers * pleDim] so each layer can Narrow out its slice.
- Local layers: standard NeoX RoPE on the full headDim, base
_ropeLocalBase(fromgemma4.rope.freq_base_swaorgemma4.rope.local.freq_base, default 10000). Implemented asApplyNeoXRoPEDecode/ApplyNeoXRoPEPrefillwith vectorized cos/sin tables. - Global layers: partial NeoX RoPE applied to the first
_partialRotaryDimsdims of the headDim (the rest passes through). The frequency vector is the standard NeoX schedule scaled by per-frequency factors loaded fromrope_freqs.weight. The full cos/sin lookup table is cached across global layers in_neoXRopeCos/_neoXRopeSin, eliminating ~35MMathF.Cos/MathF.Sincalls per chunk.
After the V projection, ApplyUnweightedRMSNorm() runs RMSNorm with an
all-ones weight tensor (_onesForVNorm) on each value vector. This is unique
to Gemma 4 in TensorSharp's matrix.
HasMoE(layer) is true when blk.{L}.ffn_gate_inp.weight exists. The MoE
branch:
- RMSNorm(pre_ffw_norm_2) on residual.
- Router matmul → unweighted RMSNorm → multiply by learned scale
ffn_gate_inp.scale. - softmax → TopK(
_numExpertsUsed). - For each selected expert: SwiGLU(
ffn_gate_up_exps.{e}.weight,ffn_down_exps.{e}.weight) followed by weighted accumulation. - RMSNorm(post_ffw_norm_2).
The dense and MoE branches are then summed and passed through
post_ffw_norm.
_layerScalars[layer] is loaded from blk.{L}.layer_output_scale.weight
(scalar [1]). Each layer's hidden output is multiplied by this scalar before
returning to the next layer.
Uses tanh(logits / cap) * cap when
gemma4.final_logit_softcapping > 0.
Gemma4VisionEncoder is a SigLIP-style ViT with 2D position embeddings, a
gated GELU-tanh MLP (gelu_pytorch_tanh, as the vision config declares; the
fused native block and the managed fallback both use it, where earlier builds
used QuickGELU), and a pooled projection to the LM hidden dim: an unweighted
RMSNorm over the vision hidden size (embedding_pre_projection_norm) followed
by the linear projector. Earlier builds normalized after the projection
instead; the order now matches upstream Gemma4MultimodalEmbedder.
The tower loads from an mmproj GGUF or, through the same --mmproj option,
from a HuggingFace .safetensors shard that holds it under
model.encoder.vision_tower.* with the projector
model.encoder.embed_vision.embedding_projection.weight
(Gemma4VisionEncoder.Safetensors.cs).
Geometry comes from the tensor shapes; the pooling factor and RMSNorm epsilon
come from a config.json beside the shard when its vision_config matches the
tower's width, otherwise from the Gemma 4 defaults with a one-time warning.
This path exists for checkpoints published without an mmproj
(DiffusionGemma is the case it was written for). A shard
carries no audio tower, so audio input still needs an mmproj GGUF. Video frames
are extracted from MP4 inputs using time-based sampling (one frame per second by
default via OpenCV, configurable with VIDEO_SAMPLE_FPS; VIDEO_MAX_FRAMES adds
an optional upper bound) and each frame is encoded independently; the resulting
embeddings are concatenated and injected at successive <|image> markers.
Still images are sized by projector type. A gemma4v (SigLIP) tower, such as
E4B's or DiffusionGemma's, gets the reference Gemma4ImageProcessor sizing: the
image is scaled up or down to the largest canvas within the 280-soft-token budget
whose sides are multiples of 48 px (patch 16 x pooling 3), then resized with an
antialiased bicubic filter and stretched to fill, with no letterbox. Earlier
builds used a letterboxing port of llama.cpp's smart resize that padded the
picture with black bars, so together with the activation and norm fixes above,
image answers on these towers change after upgrading. The resized pixels are
checked against a numpy transcription of the reference (Gemma4VisionOracleTests,
which needs a local fixture directory and is skipped without one). The 12B's
gemma4uv unified embedder, a block-less path (one large-patch conv, LayerNorms,
learned 2D positions, an RMSNorm and a linear projection) rather than the SigLIP
stack above, keeps the older letterboxed sizing it was validated on. Video frames
use a fixed 70-soft-token budget with that older sizing.
Gemma4AudioPreprocessor decodes WAV / MP3 / OGG, resamples to 16 kHz mono,
emits a 128-bin log-mel spectrogram in 10 ms hops, and pads to the encoder's
chunk size (12 frames × 12 chunks).
Gemma4AudioEncoder is a chunked-attention USM-style transformer with:
- Conv subsampling in the time dimension.
- Per-chunk causal attention with a 12-frame past context window.
- A
logit_cap = 50clip on attention logits (the same idea as the LM logit softcap). - A residual scaling factor of
0.5. - A final linear projector to the LM hidden dim.
The resulting audio embeddings flow into the same InjectVisionEmbeddings
path as image embeddings; from the language model's point of view, audio and
image tokens are interchangeable carriers of pre-computed embeddings.
| Key | Type | Meaning |
|---|---|---|
gemma4.attention.sliding_window_pattern |
bool[] | Per-layer SWA pattern |
gemma4.attention.sliding_window |
uint32 | SWA window size (default 512) |
gemma4.attention.key_length |
uint32 | Global head dim (default 512) |
gemma4.attention.key_length_swa |
uint32 | Local head dim (default 256) |
gemma4.attention.global_head_count_kv |
uint32 | KV heads for global layers |
gemma4.attention.head_count_kv |
int32[] | Per-layer KV head counts |
gemma4.attention.shared_kv_layers |
uint32 | Number of KV-sharing tail layers |
gemma4.rope.dimension_count |
uint32 | Partial rotary dims |
gemma4.rope.partial_rotary_factor |
float32 | Fraction of head dim to rotate |
gemma4.rope.freq_base_swa |
float32 | Local RoPE base |
gemma4.embedding_length_per_layer_input |
uint32 | PLE dim (0 disables PLE) |
gemma4.expert_count |
uint32 | MoE expert count (0 ⇒ dense) |
gemma4.expert_used_count |
uint32 | TopK routed experts per token |
gemma4.final_logit_softcapping |
float32 | LM head softcap |
token_embd.weight
output_norm.weight
output.weight # (optional if tied to token_embd)
blk.{L}.attn_norm.weight
blk.{L}.attn_qkv.weight # fused QKV (non-shared layers)
blk.{L}.attn_q.weight # Q-only (KV-shared layers)
blk.{L}.attn_q_norm.weight
blk.{L}.attn_k_norm.weight
blk.{L}.attn_output.weight
blk.{L}.post_attention_norm.weight # or attn_post_norm.weight
blk.{L}.ffn_norm.weight
blk.{L}.ffn_gate_up.weight # fused gate+up
blk.{L}.ffn_down.weight
blk.{L}.post_ffw_norm.weight # or ffn_post_norm.weight
blk.{L}.layer_output_scale.weight # scalar [1]
# MoE only:
blk.{L}.ffn_gate_inp.weight # router
blk.{L}.ffn_gate_inp.scale # learned router scale
blk.{L}.ffn_gate_up_exps.{E}.weight # fused expert gate+up
blk.{L}.ffn_down_exps.{E}.weight # expert down
blk.{L}.pre_ffw_norm_2.weight # MoE input norm
blk.{L}.post_ffw_norm_1.weight # dense MLP post-norm
blk.{L}.post_ffw_norm_2.weight # MoE post-norm
# PLE only:
per_layer_token_embd.weight
per_layer_model_proj.weight
per_layer_proj_norm.weight
blk.{L}.inp_gate.weight # PLE gate
blk.{L}.proj.weight # PLE projection
blk.{L}.post_norm.weight # PLE post-norm
# Global RoPE:
rope_freqs.weight # proportional frequency factors
Constructor (Gemma4Model(string ggufPath, BackendType backend)):
ParseBaseConfig()(general fields).- Reads SWA pattern and window, both head dims, both KV head counts, both
RoPE bases, partial rotary dims, PLE dim, shared KV layer count, and the
MoE expert counts. Sets
Config.UsesCircularKvCachefrom the SWA pattern. BuildKVDonorMap()— produces_kvDonorMap[layer] → donorLayer.ParseTokenizer(),LoadWeights()._hasTiedOutputdetection.DetectHeadDimsFromWeights()— repairs head dims if metadata lies.LoadLayerScalars()— loads_layerScalars[NumLayers].FuseQKVWeights(),FuseGateUpWeights(),FuseExpertGateUpWeights().PrepareCudaQuantizedWeightsForInference()— uploads / reorders quant blobs for the direct CUDA backend.PrecomputeRoPE()— fills both local and global frequency tables.InitKVCache(maxSeqLen)— per-layer cache with capacityslidingWindowfor SWA layers,maxSeqLenfor global layers; aliases shared layers to their donors.BuildGemma4DecodeArrays()— packs per-layer pointers, types, dims, and the optional MoE / PLE flags into the_decodeArraysstruct used by the fused single-graph decode kernel.
Forward(int[] tokens) orchestrates:
- Embedding lookup + scaling.
- Optional vision / audio injection (drains the pending lists, marking the
injected positions in
exceptPositionsso the layer code skips RoPE / cache writes for those positions). - Optional PLE computation.
- Either:
- Fused decode (one native call, see § 9), or
- Per-layer C# loop with optional
TryFusedLayerPrefillper dense layer.
- Final RMSNorm, LM head, optional softcap, copy to
_logitsBuffer.
ForwardRefill(int[] tokens) is the prefill-then-decode entry point. It
splits the prefix into min(2 × slidingWindow, 2048)-token chunks
(overridable via TS_PREFILL_CHUNK) and ends with Forward([lastToken]).
Multimodal prompts skip chunking because injection positions are absolute.
On ggml backends, ordinary multi-token prefill is served by the same fused
whole-model kernel that MTP verification uses (§12): all layers run in a
single GGML graph dispatch with activations device-resident, instead of one
graph per layer. CanUseWholeModelPrefillVerify() gates the path — dense
models only, including E-series in-kernel PLE and shared-KV donor layers, with
multimodal chunks eligible at any start position via the kernel's
bidirectional-span mask (TS_G4_MM_PREFILL=0 reverts multimodal to the per-op
path; see Image and audio turns after a reused prefix). SWA-wrapped chunks at startPos > 0 stay on the fused path through the
kernel's in-kernel swaPrev gather (TS_G4_VERIFY_SWAPREV=0 disables).
All-MoE variants (e.g. 26B-A4B) have a sibling fused path,
CanUseWholeModelMoEPrefillVerify() / TryFusedMoEModelVerify(). Set
TS_G4_WHOLE_PREFILL=0 to force the per-op chunked path for A/B. Note that
block-quantized (q8_0 / q4_0) KV caches require this path for
multi-token prefill — the per-op fallback cannot walk block-quantized cache
layouts.
The scheduler feeds a solo (uncontended) prompt to this path in big chunks
capped by TS_SCHED_SOLO_PREFILL_CHUNK (default 8192). See
docs/perf/gemma4-prefill-cuda-graph-design.md
for the measured design.
An image, video frame or audio clip enters the prompt as a span of soft tokens that
attend to each other in both directions. The prefill kernels encode that as one byte
per chunk token (is_except): a soft-token query also reads the chunk's soft-token
keys ahead of it, while every query reads the keys before it causally, inside its
sliding window on local layers. When a conversation continues its cache into an image
turn, the image chunk starts at a non-zero position P, after the text the earlier turns
left behind.
Until this change the fused kernels (dense TSGgml_Gemma4ModelVerify, MoE
TSGgml_Gemma4MoEModelVerify) applied the soft-token clause only at P = 0, because they
compared a key's buffer index with the chunk's bytes. The gates therefore sent a media
chunk at P > 0 to the per-op path, which was slower (E4B/Metal, a 457-token image turn
reusing 179 tokens: 1.25 s to first token instead of 0.64 s cold) and, once the ring had
wrapped, wrong: its mask read the buffer index as the absolute position, but past the
window the gathered window starts at P - 511, not 0. Phase 0 had to make such a turn
reuse nothing past its public prefix (CanPrefillMediaAfterReusedPrefix).
Every buffer the fused kernels attend keeps the chunk as its last N real keys: the chunk
alone at P = 0, the global cache [0, P + N), an unwrapped sliding-window cache, or the
previous window gathered from a wrapped ring prepended to the chunk. A key's chunk index
is therefore its buffer index minus the number of real keys before the chunk, and the row
builders in gemma4_mm_mask.h apply that offset. At P = 0 and for text chunks they
produce the same rows as before. The per-op path shifts the absolute soft-token positions
into the frame of its key buffer (ShiftPositions), as the tensor-parallel per-op path
already did; the tensor-parallel fused verify passes the chunk mask at any P too (its dense
variant used to run a media chunk at P > 0 with no soft-token mask at all).
So Gemma 4 no longer overrides CanPrefillMediaAfterReusedPrefix: an image turn reuses
the conversation's text at any prompt length, and the image chunk runs on the fused
graph. Measured on E4B (Q8_0, Metal, M5 Pro) with the IMG conversations (text, text,
image, text; greedy; median of two rounds; the Web UI column is time to first token, the
OpenAI column wall time for the whole non-streamed request):
| Image turn 3 | 6db6dbf6 | Phase 0 | this change |
|---|---|---|---|
| Web UI, 457-token prompt: reused / time to first token | 0 / 0.64 s | 179 / 1.25 s | 179 / 0.57 s |
| OpenAI, 457-token prompt: reused / wall | 0 / 1.80 s | 179 / 2.09 s | 179 / 1.46 s |
| Web UI, 889-token prompt (past the window): reused / time to first token | 0 / 0.85 s | 0 / 0.90 s | 611 / 0.62 s |
| OpenAI, 946-token prompt (past the window): reused / wall | 0 / 1.86 s | 0 / 1.60 s | 668 / 1.29 s |
Every reply in those runs (24 short-conversation and 16 long-conversation turns) is the
same text as the Phase 0 head's and as this change's with the prefix cache off
(TS_SCHED_PREFIX_CACHE=0), and text-only turns keep their reuse and timing
(AgentTurnBench --scenarios short,long,tool, two runs each: within 0.5% of 6db6dbf6 and
of the Phase 0 head on every row, identical tokens).
On CUDA (E4B Q8_0, A40, one round) the image turn gets the same reuse and runs faster too: Web UI 449-token prompt 1.10 s (6db6dbf6) / 2.85 s (Phase 0) / 0.93 s, OpenAI 1.45 / 2.49 / 0.98 s; past the window, Web UI 893 tokens reusing 615: 1.10 / 1.12 / 0.91 s, OpenAI 870 tokens reusing 592: 2.05 / 1.06 / 0.87 s. There, however, a turn continued from a cache is not token-for-token the cold reply on any build: on text turns and image turns alike, and on 6db6dbf6 as well, the greedy reply after a reused prefix leaves the cold one after a few dozen to a few hundred characters, because prefilling the same tokens in two chunks instead of one moves the logits by 0.65 to 0.8 on E4B with these kernels (the text control of the test below). With identical prompts, the Phase 0 head's image-turn reply equals this change's in the short conversation, where both reuse the same prefix, and differs past the window, where the Phase 0 head prefilled from zero.
Coverage: the native gemma4-multimodal-mask-after-reused-prefix test keeps a verbatim
copy of the old rows, checks they are unchanged at P = 0 and for text, and checks that a
chunk at P > 0 sees exactly what the same queries see in a cold prefill in every buffer
layout (the old start-position gate fails 1,102 of its 1,200 media cases).
Gemma4MediaAfterReusedPrefixExactnessTests (model-gated, TS_TEST_MODEL_DIR) prefills a
text turn and then an image or audio turn with and without reuse, inside and past the
window, on the fused and the per-op path. Its tolerance is twice the larger of two noise
estimates measured in the same row: the same split with the media left out, and the cold
media prefill on the other path. On Metal (E4B, E2B) both estimates are about 0.02 logits,
the fused greedy streams are identical and the prefill logits agree within 0.026, while on
the Phase 0 head they differed by 3.2 for an image and 2.0 to 7.1 for audio past the window
(per-op path: 3.2). On CUDA (E4B, 12B, 26B-A4B) the estimates are 0.35 to 1.4 (2.7 for
12B's short text control), every row passes, and the Phase 0 head's long-prefix rows
differ by 3.3 to 3.8 (image) and 2.1 (audio), above the tolerances measured for the same
rows here (1.3 to 2.8); the greedy streams there are compared only up to the first
near-tie within the noise, which is usually the first token or two. The per-op path's
greedy stream is not compared token for token: it is not reproducible against itself (four
identical cold prefills flipped a near-tie at decode step 16). Gemma4SoftTokenMaskTests
pins the two position conversions.
Two limits carry over from the cold path. The mask marks soft tokens, not which media item they belong to, so two images prefilled in the same chunk attend to each other in both directions. A conversation that sent an image in an earlier turn prefilled that image in its own chunk, where it could not see a later image; a cold prefill of the whole history lets it, so a reply after two images can differ between the two. And a chunk boundary that falls inside a span, which the scheduler's 256-token contention chunks can cause, splits its bidirectional attention.
The per-layer embeddings (PLE) are gathered inside the fused verify graph via
ggml_get_rows on the resident quantized per_layer_token_embd table,
instead of computing them in C# and shuttling the ~88 MB result
device→host→device every chunk. Default on; TS_G4_PLE_IN_KERNEL=0 reverts
to the uploaded path.
At the start of a fresh prefill the grow-on-demand global KV cache is
pre-sized to the whole prompt (PrepareForPrefill(totalPromptTokens)). At
start_pos == 0 the cache holds no committed K/V, so growing to the final
size copies nothing — this eliminates the incremental doubling grows that
each re-copied and device↔host round-tripped the whole global cache (a
measured ~7% at 64k). GGML GPU backends only, clamped to the model context.
ggml-cuda runs the 512-dim global layers only on its grouped-query flash
kernel, which needs a KV window that is a multiple of 256 rows. The window is
padded to 256 but never past the allocated cache, so a cache that has grown to
16 rows (TS_KV_INITIAL_TOKENS=8) or that is capped at a length like 4000
(MAX_CONTEXT=4000, past 3840 tokens) used to abort the process in
ggml-cuda/fattn.cu. Those global layers now run as explicit attention with the
same result and a one-time has no flash-attention kernel warning, and return
to the kernel once the cache length is a multiple of 256 again; the default
cache sizes never take this path. See
Flash attention on shapes a backend has no kernel for.
For each eligible layer (dense, non-shared KV, no PLE injection in the
current chunk, all weights quantized), TryFusedLayerPrefill() invokes a
single GGML graph that performs the entire transformer block in one dispatch:
- RMSNorm(attn_norm)
- Fused QKV matmul → split Q, K, V
- Per-head QK RMSNorm, V unweighted RMSNorm
- RoPE (local NeoX or global proportional)
- Causal attention (SWA or full causal) with KV cache append
- Output projection + RMSNorm(post_attn_norm) + residual
- RMSNorm(ffn_norm) → ffn_gate_up matmul → GELU(gate)*up → ffn_down matmul
- RMSNorm(post_ffw_norm) + residual + layer_output_scale
Layers with MoE, KV sharing, or active PLE injection fall back to the
standard per-op C# path. Setting TS_FUSED_LAYER_PREFILL=0 disables the
fused path entirely (useful for debugging or A/B benchmarking).
prefillChunkSize = min(2048, max(2 × slidingWindow, 1024)). Each chunk
processes through all layers before advancing, keeping the SWA score tensor
small. Override with TS_PREFILL_CHUNK=N for tuning.
Three caches re-used across layers within one forward pass:
_cachedSWAMaskWidths— per-row sliding-window mask widths for the current(queryLen, startPos); recomputed only whenseqLenorstartPoschanges._neoXRopeCos/_neoXRopeSin— global-layer NeoX RoPE cos/sin tables built once per forward and reused across all global layers._cachedRoPEPosQ/_cachedRoPEPosK— local-layer RoPE position tensors cached across layers whenseqLenandstartPosmatch.
Long prompts wrap the SWA cache mid-chunk, so a chunk-2 query near the start
needs the (W − 1) positions that the chunk just overwrote. Before any layer in
the new chunk runs, PrepareSwaPrevWindowsForChunk(startPos, seqLen) snapshots
the live SWA window; SWA layers then concatenate the snapshot in front of the
freshly-computed K/V before running attention.
When KV-shared SWA layers exist, the donor layer publishes its freshly-
computed K/V into _prefillSWAKV so shared layers attend to the full chunk's
K/V instead of the partial rolling cache.
When _canUseFusedDecode (all dense, all quantized) and seqLen == 1, the
decode path collapses to a single native call. BuildGemma4DecodeArrays()
packs the following per layer once at load time:
- Quantized weight metadata (
type,ne0,ne1, raw byte pointers). - Per-layer head dim, KV head count, RoPE base / kind.
- KV cache pointers (writable scratch on GPU).
- Layer scalar value.
- PLE flags / projection pointers.
NativeGemma4ModelDecode() then runs the entire stack — embedding lookup is
in C# but every layer (RMSNorm + QKV + QK-norm + V-norm + RoPE + circular
cache append + attention + output projection + post-attn norm + GeGLU FFN +
post-FFN norm + residual + layer scalar) executes in one GGML graph dispatch
on Metal / CUDA. This eliminates hundreds of CPU↔GPU round-trips per token.
EnsureKvCacheHostSynchronized() bridges between the fused and unfused
paths: when the next forward is unfused (e.g. mid-conversation prefill), the
host KV cache copy is updated from the device.
CopyToCacheCircular() writes new K/V slots at pos % cacheSize for SWA
layers. AttentionDecodeCircular() traverses the circular buffer for read.
SWA layers therefore allocate slidingWindow slots regardless of context
length — the resident set is bounded.
With N >= 2 requests in flight the engine does not round-robin N single-token
graphs: Gemma4Model.TryForwardBatchedFusedDecode decodes one token for every
sequence in ONE fused graph (TSGgml_Gemma4ModelDecodeBatchedEx2 in
ggml_ops_gemma4_batched.cpp),
so every weight is loaded once per step and applied to N tokens. Decode is
bandwidth-bound, which is where the aggregate throughput comes from. Each
sequence keeps its own per-request KV holder; the kernel takes the holders as a
[layer * N + seq] pointer array, runs the projections / FFN / LM head over
[hidden, N], and runs one single-row flash-attention per sequence over a
direct view of that sequence's cache window.
The v1 kernel covered dense models without per-layer embeddings, without
shared-KV layers and only while every sequence fitted its SWA ring, so the
E2B/E4B checkpoints (PLE dim 256, 18 KV-donor layers, a 512-slot ring most
chats outgrow) always declined and the server logged "The model declined the
default batched fused-decode path ... concurrency stays near 1x". The Ex
entry closes those three gaps:
- PLE per row. The per-layer embeddings are gathered inside the graph
from the resident quantized
per_layer_token_embdtable with oneget_rowsover the N token ids (plus theper_layer_model_projprojection, its RMSNorm and the1/sqrt(2)combine, exactly as the single-token decode andComputePLE), and injected per layer as a strided[ple_dim, N]column slice. When the in-kernel form is unavailable (CanGatherPleInKernelfalse: an unsupportedget_rowstype or a scaled projection) the caller uploadsComputePLE's rows instead, so the PLE term is never dropped. - KV-donor layers.
kv_source_arr[l]names the layer whose cache layerlattends. A shared layer runs the Q-only projection and reads the donor's per-sequence window and mask; it writes nothing. - SWA wrap. A local layer whose sequence is past its ring writes at
pos % cache_sizeand reads the whole ring flat with every slot valid; decode softmax is permutation-invariant over keys, so the rotation needs no concat. Global (linear) layers must still fit their cache. A sequence that needs growth takes a serial step while other ready sequences can continue batching.
Ex2 additionally accepts each sequence's actual cache capacity as
cache_size_arr[layer * N + seq]. Retained-prefix clones are intentionally
smaller than fresh caches, and each request grows independently. The earlier
uniform-capacity ABI rejected such batches even when every request had enough
space. The new kernel uses each allocation's real KV-head stride and its own
padded attention window and mask. It does not expand short children to the
largest request's allocation. Shared-KV layers use their donor's capacity for
the same sequence.
The native side reports what it supports through
TSGgml_Gemma4BatchedDecodeCapabilities() (bits: PLE=1, KV donor=2, SWA wrap=4,
per-sequence cache sizes=8).
The managed gate keeps the v1 restriction for every bit the loaded native
library lacks, so an older libGgmlOps (no probe symbol) behaves exactly as
before, and TSGgml_Gemma4ModelDecodeBatched keeps its v1 ABI as a thin
wrapper. The old Ex ABI also retains its layer-only capacity array.
TS_GEMMA4_BATCHED_CAPS=0 forces the v1 gates for an A/B;
TS_GEMMA4_BATCHED_CAPS=7 restores the uniform-capacity gate while retaining
PLE, shared-KV and SWA-wrap support. These are diagnostic overrides, not flags
needed to enable batching. Rebuild both the managed server and native GgmlOps
library to use Ex2; an older native library continues safely through fallback.
Every per-step input (hidden rows,
positions, per-(layer, seq) set_rows write rows, per-sequence F16 masks, the
PLE token ids or uploaded PLE rows) is a graph input refreshed with
ordered backend uploads, and the graph lives in its own context and own-slot
buffer, so a recurring request set replays a captured graph at stable
addresses. Capture identity includes all K/V pointers, capacities and windows;
cache growth invalidates captured graphs before replacing storage. The MoE
batched kernel (TSGgml_Gemma4MoEModelDecodeBatched) keeps
its no-PLE / no-donor scope.
The server logs the first successful fused batch and gives a reason for the
first declined batch. One decline does not mean every later step is serial.
Gemma4Model.BatchedFusedDecodeSteps and BatchedFusedDecodeTokens count only
successful native batched executions; BatchedFusedDecodeDeclines counts
rejected attempts. The regression and benchmark in
eng/validation/Gemma4BatchedCacheProbe
exercise real prefix clones, different capacities, independent growth, reordered
requests, and subsequent solo continuation.
Verified (Gemma4BatchedFusedDecodeParityTests,
TS_TEST_GGML_BACKEND=cuda, gemma-4-E4B-it-Q8_0): for 2, 3 and 4 concurrent
sequences, one of them prefilled past the 512-token SWA ring, the 12-step greedy
continuations of the batched path equal the round-robin single-token decodes
token for token, and every step ran on the batched kernel.
Measured (gemma-4-E4B-it-Q8_0, NVIDIA A40, ggml_cuda, f16 KV, prefill
chunk 512, 4 running sequences; the round-robin column forces the v1 gates with
TS_GEMMA4_BATCHED_CAPS=0, everything else identical):
| Workload | Round-robin (before) | Token-batched (after) | Ratio |
|---|---|---|---|
AgentTurnBench conc, 1 request (decode tok/s) |
68.8 | 68.5 | 1.0× |
AgentTurnBench conc, 2 concurrent (aggregate decode tok/s) |
65.1 | 99.0 | 1.5× |
AgentTurnBench conc, 4 concurrent (aggregate decode tok/s) |
66.5 | 148.7 | 2.2× |
validate_inference.py decode (512 new tokens), concurrency 1 (end-to-end tok/s) |
76.8 | 77.7 | 1.0× |
validate_inference.py decode, concurrency 4 (aggregate end-to-end tok/s) |
70.3 | 148.6 | 2.1× |
validate_inference.py decode_8k (8k prompt), concurrency 1 |
63.8 | 64.3 | 1.0× |
validate_inference.py decode_8k, concurrency 4 |
59.0 | 101.7 | 1.7× |
The validate_inference.py rows compare the server built from this tree
against the server built from the previous commit; their solo (concurrency 1)
outputs are byte-identical. What "token for token" does and does not mean
here: the AgentTurnBench conc streams (compare.py, identical output token
ids required) and the 12-step in-process parity test match the round-robin
decode exactly, but over hundreds of greedy tokens a low-margin token can flip
(ParityHarness --batched at 256 steps: one of the sequences in each 2/3/4-way
set diverged), and the pre-existing v1 kernel does the same on gemma-4-12B
(no PLE, no shared KV) under the identical run — batching changes GEMM shapes
and therefore rounding, the caveat already documented for the GLM batched
decode. Through the HTTP server at concurrency 4 even two round-robin runs
differ (10/16 identical cases), because prefill/decode interleaving is
scheduling-dependent, so parity has to be judged in-process.
- SWA layers: capacity
_slidingWindowslots, circular write/read viaCopyToCacheCircular()andAttentionDecodeCircular(). - Global layers: capacity
maxSeqLenslots, linear append/read viaCopyToCacheLinear()andAttentionDecodeWithWindow(). - Shared layers: alias the donor's
_kvCacheK[donor]/_kvCacheV[donor]— no separate allocation. - Quantized weight binding: zero-copy mmap on GGML CPU / Metal / CUDA (the
GGUF file is
MemoryMappedFile+QuantizedWeight.CreateExternalView). Direct CUDA uploads quantized blobs to device memory once and frees the host copy.
Under the default radix prefix cache (TS_PREFIX_CACHE_MODE=tree) the tree
decides what a finished request leaves behind, and its minimum is 32 tokens
(MinRetainTokens). The one-block rule below belongs to the legacy
retained-holder path (TS_PREFIX_CACHE_MODE=legacy).
A finished concurrent request's per-request holder is kept for its conversation's next
turn only when it holds at least one scheduler block (256 tokens by default), by the same
executor rule Qwen 3.5 uses (BatchExecutor.TryRetainReleasedFusedCache and
DonateFinishedLiveCacheToRetained). A shorter conversation re-prefills its whole prompt on
every turn that runs beside another request; one that runs alone continues from the live
cache, which has no minimum. Holders do not need the block granularity (they are matched
token by token and adoption reserves ceil(lcp / BlockSize) placeholder blocks), but lowering
the minimum failed its exactness validation on Qwen 3.5 on Metal, with a cause not yet
identified; see Qwen 3.5, Retained holders: the one-block
minimum. Shorter Gemma 4 holders were not
validated, so the minimum stays here as well.
Gemma 4 has a full IBatchedPagedModel.ForwardBatch port
(Gemma4Model.BatchedForward.cs)
that runs through the shared InferenceEngine continuous-batching stack
(docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md).
Unlike most batched ports, Gemma 4 is enabled by default; set
TS_GEMMA4_BATCHED=0 to force the legacy per-seq KV-swap path for
debugging. A solo request already reaches the fused single-graph decode
through the default N=1 fast path (TS_BATCHED_N1_FAST_PATH=1).
Gemma 4 is the hardest model TensorSharp ports to paged batching because of three sources of per-layer heterogeneity that the engine assumes is uniform across layers:
- Heterogeneous head dims and KV head counts. Local layers use
head_dim_local(typically 256) and a differentnum_kv_headsthan global layers (head_dim_global512).EnsureGemma4PagedBuffersallocates one paged K/V buffer per layer with its own per-layernumKvHeads * headDimsize._g4PagedKvDimPerLayer[layer]records the dimensionality so the scatter / gather steps know how to stride into each buffer. - Per-layer SWA dispatch. The native paged-attention kernel takes a
sliding_windowparameter per call. The batched path computesIsLocalLayer(l) ? _slidingWindow : 0per layer and threads it intoGgmlBasicOps.PagedAttentionForwardso local layers get the SWA window mask and global layers get full attention — within the sameForwardBatchinvocation. - KV donor layer aliasing. Layers 24-41 share K/V with earlier
layers. The batched buffer for a "receiver" layer is a pointer alias
of the donor layer's buffer (refcount-shared inside the
BlockPool) rather than a separate allocation. This preserves the legacy KV-share behaviour while keeping the scheduler oblivious to the aliasing.
The remaining batched-path mechanics mirror Mistral 3:
- Per-token NeoX RoPE dispatched through
Ops.RoPEExWithFreqFactorswith the per-layer proportional / partial frequency factors and an explicitpositions[]tensor. - K/V scatter via
slotMappinginto the per-layer paged buffer.EnsureGemma4PagedBufferscopies existing K/V across grow events so already-scheduled sequences keep their state. - Per-Layer Embedding (PLE) is computed once per batch at the
start of
ForwardBatch(ComputePLE(flatTokens, hiddenStates, numTokens)) and the per-layer slice is added after the post-attention residual, matching the legacy injection point. - Multimodal embedding injection uses the same row-wise
InjectMultimodalEmbeddingspath as the legacy forward; embeddings land at the right absolute token positions inside the concatenated batched hidden state.
Verified correctness
(Gemma4BatchedForwardTests):
- Per-layer checksum trace matches the legacy unfused path within FP noise for all 42 layers (SWA + GQA + PLE + KV-donor sharing at L24+).
- Logit-cosine ≥ 0.99 against the legacy non-batched path.
EngineParallelInferenceTests.Gemma4_ThreeLongGenerationsParallelexercises the multi-sequence batched path through the engine.
Throughput (gemma-4-E4B-it-Q8_0, Apple M4 Pro, GgmlMetal — toggling
TS_GEMMA4_BATCHED in-process, see
Gemma4BatchedPerfBench.cs):
| Workload | n | Prompt tok | Legacy tps | Batched tps | Speedup |
|---|---|---|---|---|---|
| Single sequence, short prompt | 1 | 29 | 14.0 | 4.9 | 0.35× (batched slower) |
| 5 short prompts parallel | 5 | 142 | 10.2 | 13.5 | 1.32× |
| 8 short prompts parallel | 8 | 218 | 10.0 | 15.1 | 1.51× |
| 4 long-context prompts parallel | 4 | 3 293 | 3.4 | 5.4 | 1.61× |
The speedup grows with batch size: at batch=8 short prompts the per-call paged graph build / gather cost is fully amortised. Single-sequence was a net loss in this in-process toggle because the legacy fused single-graph decode won by ~3× without any batching to amortise. These numbers predate the N=1 fast path: with it on (the default) a lone sequence runs the fused single-graph decode instead of the batched per-op route.
Two known bring-up bugs fixed during port (now regressions-tested):
TSGgml_PagedAttentionForwardwas missingggml_conton Q after the permute.ggml_flash_attn_extsilently produced wrong output on Metal when Q was a non-contiguous view.EnsurePagedBufferspreviously rebuilt K/V arrays destructively on grow, wiping K/V already written for previously-scheduled sequences in the same batch. The first sequence in a multi-sequence batch would then degenerate into a single-token loop on its first decode after later sequences joined. The fix (copy-on-grow) applies to Mistral 3 too.
Gemma 4 supports multi-token-prediction (MTP) speculative decoding
for solo (non-concurrent) sequences on both hosts. Unlike Qwen 3.6,
whose NextN block is embedded in the trunk GGUF, the Gemma 4 draft head ships as
a separate small gemma4-assistant GGUF loaded with --draft-model
(env TS_SPEC_DRAFT_MODEL, legacy TS_MTP_DRAFT_MODEL)
and attached to the target at startup by
SpeculativeDraftHeadLoader.
Source:
Gemma4Model.Speculative.cs,
driven through the shared
SpeculativeExecution
draft / verify / rollback core — Gemma 4 is the draft-head algorithm of the
pluggable --spec-type layer, which auto already picks from the checkpoint.
The draft head is an EAGLE-style recurrent drafter (mirrors llama.cpp's
gemma4-assistant.cpp + speculative.cpp draft-mtp). For each drafted token:
x = target.tok_embd[token] * sqrt(n_embd_backbone) // backbone embedding (e.g. 3840)
xh = concat(x, h_prev) // [2*backbone]
cur = nextn.pre_projection @ xh // -> draft hidden (e.g. 1024)
for il in 0..n_nextn-1: // a few Gemma-style decoder blocks
Q = rope(q_norm(wq @ attn_norm(cur))) // draft computes ONLY Q
a = attn(Q, target_KV) // attends the TARGET's last local
// (layer N-2) / global (layer N-1) K/V
cur = block_residuals(a, ffn, out_scale)
cur = output_norm(cur)
logits = draft.tok_embd @ cur // draft's own LM head
h_next = nextn.post_projection @ cur // -> backbone, chains the next draft step
The draft holds no K/V of its own (no wk/wv tensors): every draft layer
queries into the target model's existing per-layer KV cache, reusing the same
position for every drafted token (the recurrence flows purely through h). That
makes the drafter stateless given (token, h), so the only rollback on rejection
is an attention-KV position rewind — no recurrent-state snapshot is needed (in
contrast to Qwen 3.6's GatedDeltaNet trunk).
The draft's output backbone dimension ({arch}.embedding_length_out) must equal
the target's hidden size — pair the 12B target with its 12B draft, not the
26B-A4B draft. When --draft-model is given but the draft can't be activated
(file missing, hidden-size mismatch, or required draft tensors absent), the server
fails fast at startup with a remediation hint
(SpeculationStartupValidation)
rather than silently running without speculation.
SpeculationProfitable
(Gemma4Model.Speculative.cs)
gates whether speculation actually engages:
- ggml backends (CUDA / Metal;
ggml_vulkanandggml_cpupass the same gate) — run fused single-graph kernels: a multi-token verify (NativeGemma4ModelVerify, orTryFusedMoEModelVerifyfor the 26B-A4B MoE) and a fused draft step (NativeGemma4DraftStep). On partial acceptance a dense fast-rollback avoids re-running the kept prefix (escape hatchTS_GMTP_NO_FAST_ROLLBACK=1). The verify trunk runs the linear path by default for solo speculation;TS_GMTP_BATCHED_TRUNK=1opts into the batched paged trunk.TS_GMTP_NO_FUSED=1falls back to the per-op path for A/B testing. - Direct CUDA (
cuda, pure C#) — has no fused kernels, but its per-op verify and draft are fully GPU-resident: the draft attends the donor cache on-device, global verify attention runs the GQA decode kernel per row against the live cache, and global RoPE uses a GPU kernel — so the verify layer loop issues zero host-sync stalls. A win on prose/chat workloads, ~break-even on low-acceptance greedy. - Pure-C#
cpuand Applemlx— no fused kernels and no GPU-resident per-op path, so speculation stays off and the engine serves the standard fused decode.
The 26B-A4B MoE target additionally required a load-time OOM fix on ggml_cuda
(skipping the per-expert device preload) before MTP speculation became a net win.
Opt-in is the --draft-model file itself; opt-out (--no-spec) and tuning
(--spec-draft / --spec-pmin) use the shared flags, which
TensorSharp.Cli and TensorSharp.Server parse from the same
SpeculativeCliFlags;
see Speculative decoding for the full
flag list and the other algorithms — --spec-type ngram in particular needs no
draft GGUF at all and so runs on a Gemma 4 checkpoint with no assistant file
beside it.
Every emitted token is drawn from a trunk row, so speculation cannot change
which prefix a token is conditioned on, only the kernel that computed its row.
Two consequences, both measured with AgentTurnBench --spec-diagnostic --spec-diagnostic-teacher-force:
- Fixed: on the dense checkpoints without per-layer embeddings (12B measured; 31B takes the same path)
under the ggml and CPU backends, a fully accepted verify past the sliding
window used to put the evicted positions back over its committed rows, so the
next verify was 30-47 logits off and 12B streams diverged from plain greedy
within ten tokens.
SpecOnVerifyAcceptednow restores slots only for a window the executor re-forwards; 12B spec and checkpoint-clone streams on an A40 are identical to plain greedy again, and n-gram acceptance on the spec prompt went from 57% to 89%. - Tolerance: a K+1-row verify and a one-row decode are different kernels. On
ggml_cudatheir rows differ by 0.5 logits median on E4B (whose BF16 per-layer-embedding projection is batch-shape dependent in ggml-cuda) and 1.7-2.2 on 12B/26B-QAT; on Metal 0.003 (E4B) and 0.15 (12B); onggml_cpu, 0. A greedy token whose top-two margin is inside that error can differ: over 512 prose tokens on E4B that happened 4-5 times, every one at a margin under 0.25.
Details and the full table: What greedy parity delivers.
Gemma4OutputParser understands two structural framings:
- Thinking —
<|channel>thought ... <channel|>chain-of-thought, then the final answer. - Tool calling —
<|tool_call>call:function_name{...args...}<tool_call|>blocks, whichOutputParserextracts into structured tool calls regardless of the surrounding content. The arguments use Gemma's own syntax (bare keys, strings wrapped in<|"|>), and the model regularly writes a string value bare when it looks like an identifier —call:read_invoice{invoice_id:INV-472},{path:src/main.py},{ids:[INV-1, INV-2]}. The converter quotes every bare value that is not a JSON number /true/false/null(inside arrays too; numbers stay numbers). A call whose arguments still do not parse is surfaced verbatim as content rather than as an empty message, and each call in a multi-call turn carries its ownindexso streaming clients can pair the argument deltas.
Chat template falls back to the hardcoded Gemma 4 template when the GGUF does not ship a Jinja2 one.
No Gemma 4 template primes anything after a tool result when thinking is off:
the canonical E2B/E4B template and the 12B/26B/DiffusionGemma templates all
end the prompt at <tool_response|> and let the model continue its own turn
(the larger templates prime a closed <|channel>thought\n<channel|> only for a
fresh <|turn>model). The renderer keeps that prompt. Priming the closed block
after a tool result anyway was measured on E4B (Metal, Q8_0, skills preamble)
and made it worse: the model wrote its reasoning unmarked, closed the block
again and only then answered, so the reasoning reached content.
What E4B does at that boundary (campaign 2026-09-16, B10) is handled while
sampling instead, from the protocol's ThinkingBudgetOpenToken /
ThinkingBudgetEndToken / SuppressUnopenedThinkingEndAfter:
- A channel the model opens with thinking off is capped. The budget counts
from
<|channel>and closes the channel with<channel|>at the first line break past a quarter ofmax_tokens(at most 64 tokens, hard stop at twice that). The parser hides the thought, the answer follows. Before this theagenticfinal turn spent all 256 tokens in the channel and returned empty content. Closing at exactly 16 or 64 tokens (mid-sentence) was also measured: the model then leaked reasoning or turned the cut-off sentence into a tool call. At the first line break past 16, 32, 48 and 64 tokens it answered correctly every time. - After a tool result, a
<channel|>with no open channel is masked. E4B wrote the answer, closed a channel it never opened, and wrote the answer again. A stream had already sent the first copy, so the client got both. With the close masked the model ends the turn after the first copy. Masking needs host logits, so these requests do not use the device-argmax path. The mask is limited to that boundary: elsewhere a stray close still splits reasoning from the answer in the parser. - With thinking on, the same budget counts from the model's
<|channel>(or from the first token when the prompt primed<|channel>thought\nafter a tool result) and closes it at exactlyTS_THINKING_BUDGET. Before, Gemma 4 had no trained end token declared, and the generic hard stop looked for</think>, which Gemma never writes.TS_THINKING_BUDGET=0disables both caps.
response_format combines with "think": true: the JSON grammar stays dormant
through the thought channel and arms after <channel|>
(ThinkingGrammarActivationTrigger). With thinking off it enforces from the
first token, which also excludes <|channel>.
Gemma 4 runs under --tp N on the direct cuda backend and on the GGML CUDA /
Vulkan backends, with multimodal embeddings injected into the TP path so
vision/audio prompts survive sharding.
The MoE variants need special handling on GGML. The whole-model MoE kernels
(TSGgml_Gemma4MoEModelDecode / ...Verify) compute the router inside the
graph and ggml_top_k returns global expert ids, which whole-expert sharding
cannot feed. Under TP the MoE layers are therefore sharded the Megatron way
inside each expert: every rank keeps all 128 experts and holds 1/tp of each
expert's FFN width (gate/up column-parallel on the intermediate dim keeping the
fused [gate_r | up_r] layout, down row-parallel on it over whole quant
blocks). sel stays global, the graph is the single-GPU graph with narrower
expert matrices, and the expert-sum output becomes a third row-parallel partial
per layer — reduced right after the routing-weighted sum, which is linear, so
summing then weighting equals weighting then summing. The slices are
materialized at load (~10.5 GB on the 26B, ~36 s); TS_GEMMA4_TP_FUSED_MOE=0
falls back to the whole-expert per-op path.
Measured on 2× RTX 2000 Ada (prefill 512 / decode 64, tok/s): E4B Q8_0 goes from
2760 / 37.3 on one GPU to 2488 / 51.7 at --tp 2; 26B-A4B IQ4_XS from
1845 / 48.5 to 2537 / 51.2. Both produce output byte-identical to their
single-GPU runs. Full detail:
TENSOR_PARALLELISM_PLAN.md.
- Fused MoE kernels for mixed dense+MoE layouts — all-MoE variants
(e.g. 26B-A4B) now run fused whole-model MoE decode
(
TryFusedMoEModelDecode/TSGgml_Gemma4MoEModelDecode) and prefill/verify (TryFusedMoEModelVerify), but a hypothetical model mixing dense and MoE layers would still fall back to the per-layer graphs. (The expert-batched FFN kernel described below already removes the sequential per-expert dispatches from the unfused path.) - Audio prefill on GPU — the audio encoder currently dispatches its conv subsampling on CPU. Moving the conv stack onto Metal / CUDA would improve TTFT for long audio prompts.
- MTP speculative decoding (gemma4-assistant draft) — a separate EAGLE-style
draft GGUF (
--draft-model) accelerates solo decode (§12). ggml backends run fused multi-token-verify (denseNativeGemma4ModelVerifyand MoETryFusedMoEModelVerify) and fused draft-step (NativeGemma4DraftStep) kernels, with a dense fast-rollback on partial acceptance; the pure-C#cudabackend runs a fully GPU-resident per-op verify/draft. The 26B-A4B MoE target also needed aggml_cudaload-time OOM fix (skip per-expert device preload) before speculation became a net win. Mismatched / incomplete draft GGUFs fail fast at startup. - Expert-batched FFN (GEGLU) —
MoEForwardpreviously ran a batched-by-expert loop that degenerated intonum_experts_usedsingle-row matmuls on the decode path. It now dispatches the entire layer throughGgmlBasicOps.MoEFFNPrefill(..., MoEActivation.GEGLUSplit), which issues two or threeggml_mul_mat_idops (gate[+up] / down) plus a fusedggml_geglu_splitactivation and expert aggregation — a constant-bounded graph submission per MoE layer regardless ofseq_lenornum_experts_used. The kernel consumes the original 3Dffn_gate_up_exps.weight/ffn_down_exps.weightblocks as zero-cost stacked views (mmap-backed on Apple Silicon, single shared buffer on Windows / Linux), and the pre-existingffn_down_exps.scaleper-expert factor is folded into the routing weights in C# so the native kernel stays activation-agnostic. Falls back to the previous batched-by-expert C# path when a layer's stacked views are unavailable (e.g. F32-only tensors).