Add GLM-5.2 NVFP4 B300 SGLang single-node agentic benchmarks#2268
Add GLM-5.2 NVFP4 B300 SGLang single-node agentic benchmarks#2268Oseltamivir wants to merge 8 commits into
Conversation
Add glm5.2-fp4-b300-sglang-agentic from the SGLang cookbook B300 NVFP4 single-node recipes (STP only, no spec decoding): - Low-latency arm: TP8 with fp8 KV cache and cutedsl bf16 GEMM backend, conc [1, 2, 4, 8, 16, 32] - High-throughput arm: TP8/DP8 attention-DP behind sglang-router consistent hashing for session affinity, conc [48-512] - Image: lmsysorg/sglang:v0.5.15.post1-cu130 (ships sglang-router 0.3.2) - benchmark_lib.sh: glm5.2* added to the unfiltered agentic corpus branch (GLM-5.2 is a 1M-context model) Both arms validated on b300-nv: LL conc16 8k1k 10,043 total tok/s (TPOT 12.1 ms); HT conc256 34,429 total tok/s (TPOT 48.9 ms). Weights pre-staged at /data/models/GLM-5.2-NVFP4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Note on the failing Meanwhile, the full agentic sweep for this PR's configs was dispatched directly against Once #2267 merges I'll update this branch from main (the changelog-touching merge re-fires the label-gated sweep) so the official |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| glm5.2-fp4-b300-sglang-agentic: | ||
| image: lmsysorg/sglang:v0.5.15.post1-cu130 | ||
| model: nvidia/GLM-5.2-NVFP4 | ||
| model-prefix: glm5.2 | ||
| runner: cluster:b300-nv | ||
| precision: fp4 | ||
| framework: sglang | ||
| multinode: false | ||
| scenarios: | ||
| agentic-coding: | ||
| - dram-utilization: 0.80 | ||
| search-space: | ||
| - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32] } | ||
| - { tp: 8, dp-attn: true, kv-offloading: none, conc-list: [48, 64, 96, 128, 192, 256, 512], router: { name: sglang-router, version: "0.3.2" } } |
There was a problem hiding this comment.
🔴 The high-throughput arm ({ tp: 8, dp-attn: true, ... }) omits the ep key, so generate_sweep_configs.py defaults EP_SIZE to 1, and glm5.2_fp4_b300_sglang.sh unconditionally passes --ep-size $EP_SIZE to the server. This launches attention-DP (dp=8) with ep-size=1, i.e. no expert parallelism across the 8 DP ranks for this MoE model — every other dp-attn: true entry in this file (including the DSv4 recipe this PR says it copies) pairs it with an explicit ep equal to tp. Add ep: 8 to match the intended/validated recipe.
Extended reasoning...
The bug: The high-throughput search-space entry for glm5.2-fp4-b300-sglang-agentic is:
- { tp: 8, dp-attn: true, kv-offloading: none, conc-list: [48, 64, 96, 128, 192, 256, 512], router: { name: sglang-router, version: "0.3.2" } }It sets tp: 8 and dp-attn: true but never sets ep. In utils/matrix_logic/generate_sweep_configs.py (single-node agentic branch, line 1055 — mirrored at lines 756/943 for the other scenario branches), the EP field is computed as:
Fields.EP.value: ep if ep is not None else 1,Since the config never supplies ep, this resolves to EP_SIZE=1. That value flows straight into benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh, which builds PARALLEL_ARGS=(--tp "$TP" --ep-size "$EP_SIZE") unconditionally (no override or reset for the dp-attn branch — it only adds --dp "$TP" --enable-dp-attention ... on top). So the actual server launch for the documented '34,429 tok/s' high-throughput arm is --tp 8 --ep-size 1 --dp 8 --enable-dp-attention.
Why this is wrong: GLM-5.2 is an MoE model (glm_moe_dsa architecture). The whole point of pairing attention-DP with expert parallelism is to shard the MoE experts across the 8 DP ranks instead of replicating the full expert set on every GPU. With ep-size=1, SGLang runs with no expert parallelism — each of the 8 DP ranks holds/replicates the entire MoE expert set, which is a materially different (and much more memory- and compute-heavy) execution than the intended sharded layout.
Why nothing catches this today: There's no validation in generate_sweep_configs.py (or the launch script) requiring ep to be set — or to equal tp — whenever dp-attn: true is set for an MoE recipe. The default-to-1 behavior is silent, so a missing key produces a syntactically valid but semantically wrong config rather than an error.
Comparison to every other recipe in the file: Grepping configs/nvidia-master.yaml shows dozens of dp-attn: true entries (dsv4, minimaxm3, qwen3.5, etc.), and every single one pairs it with an explicit ep: equal to tp (e.g. { tp: 8, ep: 8, dp-attn: true }, { tp: 4, ep: 4, dp-attn: true }, { tp: 16, ep: 16, dp-attn: true } at lines 9892-9893 just above this diff). The PR description itself says the router setup is 'same pattern as the DSv4 B300 agentic recipe' (dsv4-fp4-b300-sglang-agentic-hicache), whose equivalent high-throughput arm is { tp: 8, ep: 8, dp-attn: true, ... }. This confirms the missing ep: 8 here is a copy-paste omission, not an intentional deviation.
Step-by-step proof:
- Config entry:
{ tp: 8, dp-attn: true, kv-offloading: none, conc-list: [...], router: {...} }— noepkey. generate_sweep_configs.pyreadsep = bmk.get(Fields.EP.value)→None(key absent).- It writes
Fields.EP.value: ep if ep is not None else 1→EP_SIZE=1in the generated matrix entry/env. glm5.2_fp4_b300_sglang.shbuildsPARALLEL_ARGS=(--tp "$TP" --ep-size "$EP_SIZE")=--tp 8 --ep-size 1, then appends--dp "$TP" --enable-dp-attention ...sinceDP_ATTENTION=true.- Resulting server launch:
--tp 8 --ep-size 1 --dp 8 --enable-dp-attention— attention-DP is on, but expert parallelism is off. - This is not the config that was on-node validated to get 34,429 tok/s @ conc 256 if that run used
--ep-size 8(matching every analogous recipe); the committed sweep will benchmark a different, degraded parallelism layout than what's documented.
Fix: add ep: 8 to the high-throughput arm, matching tp and the DSv4 reference recipe:
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, conc-list: [48, 64, 96, 128, 192, 256, 512], router: { name: sglang-router, version: "0.3.2" } }| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| set -x | ||
|
|
||
| # Agentic trace replay benchmark for GLM-5.2 NVFP4 on B300 using SGLang. |
There was a problem hiding this comment.
🟡 This PR's title and description are English-only, violating the AGENTS.md rule that all PR titles/descriptions must be bilingual (title format ' / <中文标题>' plus a mirrored Chinese section in the body).
Extended reasoning...
AGENTS.md line 7 states as an explicit, mandatory rule: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary...)". This applies to every PR, with no carve-out for benchmark-recipe PRs.
This PR's title is 'Add GLM-5.2 NVFP4 B300 SGLang single-node agentic benchmarks' — no '/ <中文标题>' suffix — and the description's Summary, Changes, and Validation sections are entirely in English with no mirrored '## 中文说明' section or equivalent.
This is not a matter of subjective style: the repo's own commit log shows a sibling recipe PR following the rule correctly. Commit d85fa13 (PR #2182) has the bilingual title 'Add MiniMax M3 8k/1k Dynamo vLLM B300 EAGLE recipes / 新增 MiniMax M3 8k/1k Dynamo vLLM B300 EAGLE 配方', proving other contributors in this exact benchmark-recipe workflow are expected to (and do) satisfy the rule. This PR is the outlier.
Step-by-step proof:
- AGENTS.md:7 mandates bilingual title format ' / <中文标题>' and a mirrored Chinese section in the body for every PR.
- This PR's title metadata reads 'Add GLM-5.2 NVFP4 B300 SGLang single-node agentic benchmarks' — checking the string for '/' followed by CJK characters finds none.
- The description contains only '## Summary', '## Changes', '## Validation' — all English — with no '## 中文说明' or any CJK text anywhere in the body.
- Comparing against PR Add MiniMax M3 8k/1k Dynamo vLLM B300 EAGLE recipes / 新增 MiniMax M3 8k/1k Dynamo vLLM B300 EAGLE 配方 #2182 in the git log (same repo, same benchmark-recipe category, merged 3 commits prior) shows the correctly-formatted bilingual title, confirming the rule is actively enforced/followed elsewhere and this PR is non-compliant.
Impact: none on benchmark correctness or CI — this is a process/documentation compliance gap, not a code defect. Fix is simple: the author (or whoever finalizes the merge) should append '/ <一句中文标题>' to the PR title and add a '## 中文说明' section mirroring the Summary/Changes/Validation content before merge.
One warmup request hit ECONNRESET (uvicorn closes idle connections after SGLANG_TIMEOUT_KEEP_ALIVE=5s while AIPerf reuses one pooled connection per session with a 300s client keep-alive across inter-turn gaps of up to 10s), and any terminal warmup failure aborts the AIPerf run by design. Set SGLANG_TIMEOUT_KEEP_ALIVE=900 so the server outlasts the client pool. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Sweep status: the TP8 conc-4 job in run 29634867738 failed with Root cause is a keep-alive lifecycle race, not the recipe/engine: AIPerf pins one pooled connection per agentic session with a 300s client-side keep-alive, while SGLang's uvicorn closes idle connections after Fixed in 8933d94 by exporting |
…b300 recipe The cookbook HT cell's --chunked-prefill-size 8192 is a whole-engine budget: under dp8 each rank prefills 1,024 tokens/step, which starves prefill on the 1M-context agentic corpus. A conc-256 warmup timed out after AIPerf's 1800s drain grace period with 15 giant sessions still prefilling while KV usage sat at ~0.01 (prefill-rate-bound, not memory-bound). Use the cookbook's own dp8 lever from the B200 cells: 32768 total = ~4096 tokens/rank/step. The TP arm keeps 8192 (full budget per step, passed warmup fine). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29637258921 |
… evals
The agentic SWE-bench eval drives mini-swe-agent through native tool
calling. Without --tool-call-parser, GLM-5.2's tool-call tokens stay as
raw text in message.content, every instance dies with RepeatedFormatError
("No tool calls found in the response"), 287/300 patches come back empty,
and swebench_lite scores 0.0000 against the 0.50 default threshold.
Per the GLM-5.2 cookbook: the model emits the GLM-4.7-style
<tool_call>/<arg_key>/<arg_value> format, so it needs the glm47 parser
(glm45 does not parse it); the glm45 reasoning parser separates hybrid
thinking into reasoning_content. Neither flag affects trace-replay
throughput (pre-canned replay discards live responses).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Agentic SWE-bench eval failures in run 29637532824 — root cause found. The eval jobs score Fix ready per the GLM-5.2 cookbook (§3.2): the model emits the GLM-4.7-style |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29637532824 |
The DPA conc-512 warmup drain converges healthily (~0.45 req/s, zero errors) but needs ~2500s end to end - the shared 1800s grace cuts it off with ~300 requests in flight. Make the grace period overridable via AGENTIC_WARMUP_GRACE_PERIOD (default unchanged at 1800) and set 3600 for the glm5.2 DP-attention arm. Grace is a maximum wait, not a fixed sleep, so lower-conc points are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Run 29637532824 post-mortem + final fix pushed. The TP8 matrix completed clean (conc 1–32, all green). The DPA matrix lost conc 512 to a warmup drain timeout even with the wider chunked prefill — but the log shows the drain converging healthily (~0.45 req/s, zero errors, 302 in flight at cutoff): it needed ~2,500s end-to-end against the shared 1,800s grace. fail-fast then cancelled the in-flight c128/c192/c256 (collateral, not failures). The eval jobs failed on the known pre-parser-fix tree. Pushed two commits:
This push re-fires the label-gated sweep with all four fixes (keep-alive, DP chunked prefill, parsers, grace). That run should be the green, merge-eligible one. |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29640703615 |
With the glm47 parser fix in, the agentic SWE-bench eval produces real work: 10/11 submitted trajectories resolved (0.91 precision), but 12/23 exited LimitsExceeded before submitting - GLM-5.2's chat template defaults to reasoning_effort=Max when the client passes no chat_template_kwargs, and the heavy thinking exhausts the default 75-step budget. Score landed at 0.4348 vs the 0.50 floor. Export SWEBENCH_AGENT_STEP_LIMIT=150 in the recipe's eval path only; the shared default stays 75. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Throughput matrix fully green in run 29640703615 — all 13 jobs passed with the four runtime fixes. The AgentX pareto frontier (mean interactivity = tok/s/user, total throughput = tok/s/GPU):
The pareto-efficient set is TP8 c1–c8 (interactivity end, 82–134 tok/s/user) plus TP8/DPA c48 (throughput end, 19.5k tok/s/GPU). Beyond those, the GPU KV pool (3.3M tokens vs 131k-token mean prompts) saturates: cache hit collapses from ~0.97 theoretical to 0.05–0.39 actual, and re-prefill storms dominate both axes — the classic no-offload cliff on this corpus, and a strong case for a Eval status: with the glm47 parser fix, the SWE-bench eval now does real work — 10/11 submitted trajectories resolved (0.4348 vs the 0.50 floor), but 12/23 exited |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29651235293 |
|
✅ Run 29651235293 is fully green — all 13 throughput jobs plus the agentic SWE-bench eval, which now scores Final recipe deltas vs the cookbook cells, all empirically forced by the AgentX corpus/harness rather than tuning taste: Ready for CODEOWNER review. Suggested follow-up (separate PR): a |
1 participant
Summary
Adds
glm5.2-fp4-b300-sglang-agentic— GLM-5.2 NVFP4 single-node agentic (AgentX) benchmarks on B300 with SGLang, derived from the SGLang cookbook GLM-5.2 B300 NVFP4 single-node recipes. STP only — the cookbook's low-latency/balanced cells use EAGLE MTP, which is deliberately not wired up yet (an-mtpvariant is a natural follow-up).Two arms form the throughput-vs-interactivity pareto frontier:
--kv-cache-dtype fp8_e4m3,--bf16-gemm-backend cutedsl,--max-prefill-tokens 8192— at conc[1, 2, 4, 8, 16, 32]consistent_hashingonx-correlation-id, same pattern as the DSv4 B300 agentic recipe) so multi-turn sessions keep radix-cache affinity to their DP rank, at conc[48, 64, 96, 128, 192, 256, 512]Conc lists are disjoint between arms so exp-names stay unique.
Changes
configs/nvidia-master.yaml: newglm5.2-fp4-b300-sglang-agenticconfig (agentic section); image pinned tolmsysorg/sglang:v0.5.15.post1-cu130(verified to ship--bf16-gemm-backendand sglang-router 0.3.2)benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh: new benchmark script (modeled ondsv4_fp4_b300_sglang.sh)benchmarks/benchmark_lib.sh: addglm5.2*to the unfiltered agentic corpus branch — GLM-5.2 is a 1M-context model (max_position_embeddings: 1048576)perf-changelog.yaml: changelog entry appended at tailValidation
generate_sweep_configs.py full-sweep --model-prefix glm5.2emits 13 valid entries; 223 matrix_logic tests pass; changelog parses and config key resolves/data/models/GLM-5.2-NVFP4(465 GB, 47 shards, matches the launcher's writable-models path); squash image pre-imported at/data/squash/lmsysorg_sglang_v0.5.15.post1-cu130.sqsh🤖 Generated with Claude Code