Skip to content

Generate graph-driven ORT GenAI decoder configs - #611

Open
justinchuby wants to merge 2 commits into
justinchuby-fix-gguf-q4-parityfrom
justinchuby-generic-genai-configs
Open

Generate graph-driven ORT GenAI decoder configs#611
justinchuby wants to merge 2 commits into
justinchuby-fix-gguf-q4-parityfrom
justinchuby-generic-genai-configs

Conversation

@justinchuby

Copy link
Copy Markdown
Member

Summary

  • emit the released architecture-neutral model.type: "decoder" contract for compatible single-model decoder-only text graphs
  • derive semantic inputs, cache templates, outputs, sparse global cache slots, recurrent pairs, and sliding-window metadata from the optimized ONNX graph
  • preserve runtime-specific LFM2 and Phi-3 LongRoPE behavior, fail closed for Mobius GPT-2's incompatible separate-cache ABI and unsupported state layouts, and record runtime compatibility without emitting main-only state_groups
  • keep GGUF runtime promotion evidence-gated while validating the exact SmolLM F16/CPU/tokenizer route end to end

Validation

  • 228 focused ORT GenAI/GGUF config tests
  • 2,871 affected CLI/ORT GenAI/GGUF tests
  • exact SmolLM deterministic generation on onnxruntime-genai 0.14.1 and 0.15.2
  • exact Q4_K_M fail-closed/dequantized parity test from the stacked base
  • full initialized lintrunner pass
  • two independent high-confidence reviews plus a final fix verification

Stacked on #609.

justinchuby and others added 2 commits August 24, 2026 20:31
Derive generic decoder inputs, outputs, and cache topology from the optimized ONNX graph instead of architecture-name registration. Preserve runtime-specific behavior, fail closed on unsupported state layouts, and publish released-version compatibility metadata.\n\nValidate the exact SmolLM GGUF route with deterministic generation on ORT GenAI 0.14.1 and 0.15.2.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

🏗️ Architecture Diff

Comparing d673f9dc684c1e

Model Sub-model Changes Status

No architecture changes detected.


Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed)

@github-actions

Copy link
Copy Markdown

Performance Comparison

Comparing d673f9dc684c1e

Model Metric Baseline Current Delta
bert (feature-extraction) model_size_bytes 359 KB 359 KB +0.0%
bert (feature-extraction) num_nodes 68 68 +0.0%
falcon model_size_bytes 364 KB 364 KB +0.0%
falcon num_nodes 66 66 +0.0%
gemma2 model_size_bytes 428 KB 428 KB +0.0%
gemma2 num_nodes 105 105 +0.0%
gpt2 model_size_bytes 388 KB 388 KB +0.0%
gpt2 num_nodes 54 54 +0.0%
llama model_size_bytes 425 KB 425 KB +0.0%
llama num_nodes 60 60 +0.0%
llama (static-cache) model_size_bytes 425 KB 425 KB +0.0%
llama (static-cache) num_nodes 56 56 +0.0%
mamba (ssm-text-generation) model_size_bytes 296 KB 296 KB +0.0%
mamba (ssm-text-generation) num_nodes 94 94 +0.0%
phi3 model_size_bytes 421 KB 421 KB +0.0%
phi3 num_nodes 58 58 +0.0%
phi3 (static-cache) model_size_bytes 421 KB 421 KB +0.0%
phi3 (static-cache) num_nodes 54 54 +0.0%
qwen2 model_size_bytes 425 KB 425 KB +0.0%
qwen2 num_nodes 60 60 +0.0%
qwen2 (static-cache) model_size_bytes 425 KB 425 KB +0.0%
qwen2 (static-cache) num_nodes 56 56 +0.0%
qwen3_5_moe (hybrid-text-generation) model_size_bytes 506 KB 506 KB +0.0%
qwen3_5_moe (hybrid-text-generation) num_nodes 264 264 +0.0%
qwen3_5_text (hybrid-text-generation) model_size_bytes 458 KB 458 KB +0.0%
qwen3_5_text (hybrid-text-generation) num_nodes 126 126 +0.0%
qwen3_5_vl (hybrid-qwen-vl) model_size_bytes 977 KB 977 KB +0.0%
qwen3_5_vl (hybrid-qwen-vl) num_nodes 428 428 +0.0%
t5 (seq2seq) model_size_bytes 836 KB 836 KB +0.0%
t5 (seq2seq) num_nodes 176 176 +0.0%
whisper (speech-to-text) model_size_bytes 1008 KB 1008 KB +0.0%
whisper (speech-to-text) num_nodes 128 128 +0.0%

No performance regressions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant