Add GLM-5.2 FP8 MI355X SGLang single-node agentic benchmarks#2272
Add GLM-5.2 FP8 MI355X SGLang single-node agentic benchmarks#2272Oseltamivir wants to merge 10 commits into
Conversation
Add glm5.2-fp8-mi355x-sglang-agentic from the SGLang cookbook MI355X FP8 single-node recipes (TP8, DSA tilelang prefill/decode backends, STP only; MTP/EAGLE is not validated on ROCm gfx950): - Single TP8 arm at conc [1-64]; the cookbook's low-latency vs balanced/high-throughput strategies differ only in batch-shaping levers, derived from CONC in the benchmark script - Image pinned to the cookbook-verified lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618 (carries the gfx950 block-FP8 accuracy fix); glm47/glm45 parsers verified present - launch_mi355x-amds.sh: serve zai-org/GLM-5.2* from the NFS hf-hub-cache (MiniMax-M3 pattern; 756 GB checkpoint) and exclude broken-but-idle mia1-p01-g09/g11 from single-node salloc - Carries the B300 lessons: SGLANG_TIMEOUT_KEEP_ALIVE=900 and AGENTIC_WARMUP_GRACE_PERIOD=3600 for the conc>=32 band Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| SGLANG_CMD=( | ||
| python3 -m sglang.launch_server | ||
| --model-path "$MODEL_PATH" | ||
| --served-model-name "$MODEL" | ||
| --host 0.0.0.0 | ||
| --port "$PORT" | ||
| --trust-remote-code | ||
| --tp "$TP" | ||
| --ep-size "$EP_SIZE" | ||
| --dsa-prefill-backend tilelang | ||
| --dsa-decode-backend tilelang | ||
| # GLM-5.2 emits the GLM-4.7-style tool-call format; glm47 is required for | ||
| # structured message.tool_calls (SWE-bench agentic evals die without it). | ||
| # The glm45 reasoning parser keeps hybrid thinking in reasoning_content. | ||
| --tool-call-parser glm47 | ||
| --reasoning-parser glm45 | ||
| --chunked-prefill-size "$CHUNKED_PREFILL_SIZE" | ||
| --mem-fraction-static "$MEM_FRACTION_STATIC" | ||
| --max-running-requests "$MAX_RUNNING_REQUESTS" | ||
| --cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS" | ||
| --watchdog-timeout 1800 | ||
| --enable-metrics | ||
| ) |
There was a problem hiding this comment.
🔴 SGLANG_CMD in glm5.2_fp8_mi355x_sglang.sh never passes --kv-cache-dtype, so SGLang falls back to bf16 KV cache even though this is the FP8 cookbook recipe (precision: fp8, zai-org/GLM-5.2-FP8). Every other GLM-5/5.1 SGLang script (12 files) and the architecturally-closest sibling, dsv4_fp4_mi355x_sglang.sh (also DSA/tilelang on MI355X), explicitly set --kv-cache-dtype fp8_e4m3. Add --kv-cache-dtype fp8_e4m3 to the SGLANG_CMD array (around line 90-112).
Extended reasoning...
The bug: SGLANG_CMD in benchmarks/single_node/agentic/glm5.2_fp8_mi355x_sglang.sh (lines 90-112) builds the sglang.launch_server invocation with --tp, --ep-size, --dsa-prefill-backend tilelang, --dsa-decode-backend tilelang, chunked-prefill/mem-fraction/max-running-requests/cuda-graph flags, etc., but never sets --kv-cache-dtype. SGLang's default for that flag is auto, which tracks the model's compute dtype — bf16 for this checkpoint — not the weight quantization. So despite this recipe's entire purpose being to reproduce the SGLang cookbook's MI355X FP8 GLM-5.2 recipe (configs/amd-master.yaml sets precision: fp8, model zai-org/GLM-5.2-FP8), the KV cache silently runs in bf16, doubling bytes/token relative to what the recipe intends.\n\nWhy this isn't a stylistic nit: every other SGLang launch script for this model family in the repo passes the flag explicitly: glm5_fp8_mi355x.sh:47, glm5_fp8_mi325x.sh:53, glm5_fp8_b200.sh:44, glm5_fp8_b300.sh:58, glm5_fp4_b200.sh:42, glm5_fp4_b300.sh:57, glm5.1_fp4_mi355x.sh:53, and their _mtp variants — 12 files, zero exceptions. Most tellingly, the closest architectural sibling in this very PR's lineage, benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh:140, is also an MI355X SGLang agentic recipe using DSA-style sparse attention with tilelang backends, and it too sets --kv-cache-dtype fp8_e4m3. That rules out the only plausible defense ("DSA/sparse attention can't do fp8 KV here") — the same attention path already runs fp8 KV cache elsewhere in the repo. The agentic SGLang sibling qwen3.5_fp8_b300_sglang.sh:100 also sets it. GLM-5.2's script is the sole SGLang FP8 GLM/DSA recipe that omits it.\n\nWhy nothing catches this today: check_env_vars only validates that required env vars are non-empty; it has no knowledge of server flags. The server boots fine with bf16 KV (mem-fraction-static is just a fraction of total GPU memory, so there's no hard startup failure), so wait_for_server_ready succeeds and the PR's own on-node validation ("server boots and serves") does not surface the problem.\n\nConcrete impact — step-by-step:\n1. At CONC=64, the script computes MAX_RUNNING_REQUESTS=128 and CUDA_GRAPH_MAX_BS=128, and switches to MEM_FRACTION_STATIC=0.85 — sizing explicitly derived (per the script's own comments) from the cookbook's fp8-KV-cache assumption.\n2. Because --kv-cache-dtype is never passed, SGLang instead allocates the KV cache in bf16 (2 bytes/element) instead of fp8_e4m3 (1 byte/element) — double the memory per token.\n3. Against the 1M-context agentic corpus (resolve_trace_source explicitly routes glm5.2* to the unfiltered, non-256k-capped Weka corpus), sessions carry much larger working sets than a fixed-seq-len benchmark.\n4. With half the effective KV token capacity the mem-fraction/max-running-requests levers were sized for, the conc 32/48/64 bands (which the PR description itself flags as still 'in flight'/unvalidated) are far more likely to hit KV-cache-full backpressure or OOM than the recipe's own batch-shaping math assumed.\n5. Even in runs that don't fail outright, every throughput/TTFT/ITL number collected is measuring a bf16-KV server, not the FP8 cookbook configuration this PR claims to add — silently invalidating the benchmark's stated purpose.\n\nThe fix is a one-line addition to SGLANG_CMD, consistent with every sibling script: --kv-cache-dtype fp8_e4m3.
| --trust-remote-code | ||
| --tp "$TP" | ||
| --ep-size "$EP_SIZE" | ||
| --dsa-prefill-backend tilelang | ||
| --dsa-decode-backend tilelang |
There was a problem hiding this comment.
🔴 The server launch command in glm5.2_fp8_mi355x_sglang.sh uses --dsa-prefill-backend tilelang/--dsa-decode-backend tilelang, but every other GLM SGLang recipe in this repo (GLM-5, GLM-5 MTP, GLM-5.1) selects the identical tilelang sparse-attention backend via --nsa-prefill-backend/--nsa-decode-backend instead — these --dsa-* flag names appear nowhere else in the repo. Since sglang.launch_server uses argparse (which aborts on unrecognized arguments), this will very likely fail the server to start on every conc arm of the new recipe; the fix is to rename both flags to --nsa-prefill-backend/--nsa-decode-backend.
Extended reasoning...
glm5.2_fp8_mi355x_sglang.sh:96-100 launches SGLang with --dsa-prefill-backend tilelang --dsa-decode-backend tilelang to select the tilelang sparse-attention kernel. A repo-wide search shows --dsa-prefill-backend/--dsa-decode-backend occur in exactly one place: this new script. Every other GLM SGLang recipe that configures the same tilelang backend — glm5_fp8_mi355x.sh:45-46, glm5_fp8_mi355x_mtp.sh:50-51, glm5_fp8_mi325x.sh:51-52, glm5_fp8_mi325x_mtp.sh:56-57, and the direct predecessor glm5.1_fp4_mi355x.sh:51-52 — uses --nsa-prefill-backend/--nsa-decode-backend instead. Even the DeepSeek-V4 SGLang scripts (which genuinely implement DeepSeek Sparse Attention) don't use --dsa-* flags; they use --attention-backend dsv4. There is no precedent for a --dsa-*-backend SGLang CLI flag anywhere in this codebase.
A verifier raised a plausible-sounding refutation: that GLM-5.2's architecture is _dsa (glm_moe_dsa, "DeepSeek Sparse Attention"), distinct from GLM-5/5.1's NSA architecture, and that this could legitimately expose --dsa-*-backend flags on the newer pinned SGLang image (v0.5.13.post1 vs. v0.5.12-era). I checked this directly: benchmarks/multi_node/amd_utils/setup_deps.sh:169 and experimental/token_position_decode_slo/glm-5/serve_glm5_sbatch.sh:30 both confirm that GLM-5 itself (not just GLM-5.2) already uses the glm_moe_dsa transformers architecture — "GLM-5 (zai-org/GLM-5-FP8) requires a transformers build that includes the glm_moe_dsa model type." Yet GLM-5's SGLang launch scripts uniformly use --nsa-prefill-backend/--nsa-decode-backend, not --dsa-*. This means the HuggingFace architecture class name (glm_moe_dsa) and the SGLang CLI backend-flag family (--nsa-*) are two independent naming schemes for this entire model family — the "DSA" in the architecture name does not translate into a "--dsa-" SGLang flag. The refutation's central architectural argument doesn't hold up under this repo's own evidence.
The refutation's remaining support is the PR's self-reported Validation section, which claims the server "boots and serves" with the exact committed flags. This is a single unverified assertion in an auto-generated PR description, and it directly contradicts the universal --nsa-* convention this same model family has used across every prior SGLang release (v0.5.12 through v0.5.12.post1) without the backend-selection flag name changing. It is not credible that a single point release (v0.5.12.post1 -> v0.5.13.post1) silently renamed the flag for the same tilelang backend value while every other flag in the command stayed compatible.
Impact: sglang.launch_server uses argparse, which raises SystemExit on any unrecognized CLI argument. If --dsa-prefill-backend/--dsa-decode-backend are not registered flags (as the overwhelming repo-wide pattern indicates), the server process exits immediately on startup. wait_for_server_ready detects the dead PID via kill -0 "$server_pid" and calls exit 1, failing the whole job. Since set -eo pipefail is set at the top of the script, this aborts every one of the 8 conc arms [1, 2, 4, 8, 16, 32, 48, 64] in the new glm5.2-fp8-mi355x-sglang-agentic recipe — the entire PR's benchmark would produce no usable results.
Step-by-step proof: (1) sglang.launch_server is started via "${SGLANG_CMD[@]}" > "$SERVER_LOG" 2>&1 & with --dsa-prefill-backend tilelang --dsa-decode-backend tilelang in SGLANG_CMD. (2) argparse parses sys.argv; if --dsa-prefill-backend isn't a registered option (which the repo's --nsa-* convention strongly implies), argparse prints "unrecognized arguments" to stderr and calls sys.exit(2). (3) The background process dies within milliseconds, long before the health endpoint comes up. (4) wait_for_server_ready polls kill -0 "$server_pid", finds it dead, prints "Server died before becoming healthy" and calls exit 1. (5) Because of set -eo pipefail, the whole script (and thus every conc arm using it) exits non-zero, breaking the entire recipe.
Fix: rename --dsa-prefill-backend tilelang -> --nsa-prefill-backend tilelang and --dsa-decode-backend tilelang -> --nsa-decode-backend tilelang at lines 99-100, matching every other GLM SGLang recipe in the repo.
| # mia1-p01-g09 and mia1-p01-g11 are broken but sit idle in slurm (g09 was | ||
| # already excluded on the multi-node disagg path); keep single-node jobs | ||
| # off them too. |
There was a problem hiding this comment.
🟡 The new comment claims g09 was 'already excluded on the multi-node disagg path,' but the disagg exclude list (benchmarks/multi_node/amd_utils/submit.sh:169, default SLURM_EXCLUDE_NODES='mia1-p01-g11,mia1-p01-g12,mia1-p01-g15') does not include g09. This repeats an inaccurate claim from PR #2094's changelog and could mislead future engineers into thinking multi-node disagg jobs are already protected from g09; the actual salloc change (excluding g09/g11 for single-node) is correct and unaffected.
Extended reasoning...
The bug: the new comment in runners/launch_mi355x-amds.sh:211-213 reads:
# mia1-p01-g09 and mia1-p01-g11 are broken but sit idle in slurm (g09 was already excluded on the multi-node disagg path); keep single-node jobs off them too.
The parenthetical asserts g09 is already excluded on the multi-node disagg path. That is factually wrong.
Where this is checkable: benchmarks/multi_node/amd_utils/submit.sh:169 is the only place that sets the multi-node disagg exclude list, and it defaults SLURM_EXCLUDE_NODES to mia1-p01-g11,mia1-p01-g12,mia1-p01-g15. g09 is not in that list, and a repo-wide search for SLURM_EXCLUDE_NODES/mia1-p01-g09 turns up no other override in .github/, runners/, or benchmarks/ that adds it. So today, a multi-node disagg job can still land on g09.
Why the claim looked plausible but is wrong: perf-changelog.yaml (the PR #2094 entry) claims that PR added g09 to the disagg exclude list, but that PR's actual diff only touched configs/amd-master.yaml (image bump / conc-list trim) — it never modified submit.sh. So the claim was already false when it first appeared in the changelog, and this PR's new comment simply repeats/perpetuates that stale, inaccurate assertion in a second location (code comment, not just changelog prose).
Step-by-step proof:
- Read the new comment at
runners/launch_mi355x-amds.sh:211-213: it says g09 'was already excluded on the multi-node disagg path'. - Grep the repo for where multi-node disagg exclusions are actually configured:
benchmarks/multi_node/amd_utils/submit.sh:169,SLURM_EXCLUDE_NODES=${SLURM_EXCLUDE_NODES:-mia1-p01-g11,mia1-p01-g12,mia1-p01-g15}. - Compare the node list against the claim:
g11,g12,g15— nog09. - Grep for any other place that could add g09 (
.github/,runners/, otherbenchmarks/multi_nodefiles) — none found. - Conclusion: a multi-node disagg SLURM job is not currently prevented from being scheduled on
mia1-p01-g09, contradicting the comment.
Impact: this is comment-only — it does not change any runtime behavior. The actual code change in this PR (salloc --exclude=mia1-p01-g09,mia1-p01-g11 for single-node jobs) is correct and functions exactly as intended. The risk is purely that a future engineer reads the comment, trusts it, and does not realize multi-node disagg jobs on this broken node still need to be excluded via submit.sh's SLURM_EXCLUDE_NODES default.
Fix: update the parenthetical to either drop the disagg claim entirely, or state accurately that g09 is not yet excluded on the multi-node disagg path (and optionally file a follow-up to add it to submit.sh's SLURM_EXCLUDE_NODES default alongside g11/g12/g15).
Severity: nit — this is purely a misleading code comment with no functional impact on this PR's actual change.
|
|
||
| - config-keys: | ||
| - glm5.2-fp8-mi355x-sglang-agentic | ||
| description: | ||
| - "Add GLM-5.2 FP8 MI355X SGLang single-node agentic benchmarks from the SGLang cookbook MI355X FP8 recipes (TP8, DSA tilelang backends, STP only - MTP not validated on ROCm)" | ||
| - "Image: lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618 (cookbook-verified build with the gfx950 block-FP8 accuracy fix); model zai-org/GLM-5.2-FP8" |
There was a problem hiding this comment.
🟡 This PR's title and description are English-only, but AGENTS.md requires every PR title/description to be bilingual (title format / <中文标题>, plus a Chinese section like "## 中文说明" in the body). Please add the Chinese title suffix and a matching Chinese translation section to the description, following the convention used by #2182 and #2094.
Extended reasoning...
AGENTS.md states explicitly: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary)... This applies to every PR and every issue." This is not a style suggestion — it is phrased as a hard requirement covering every PR, with no carve-out for infra/config-only changes.
PR #2272's title is plain English ("Add GLM-5.2 FP8 MI355X SGLang single-node agentic benchmarks") with no "/ <中文标题>" suffix, and the description body — Summary, Changes, Validation sections — is entirely in English with no "## 中文说明" (or equivalent) section anywhere.
The convention is actively enforced in this repo's own history, not just an aspirational doc line: the immediately preceding merged PRs use bilingual titles, e.g. #2182 "Add MiniMax M3 8k/1k Dynamo vLLM B300 EAGLE recipes / 新增 MiniMax M3 8k/1k Dynamo vLLM B300 EAGLE 配方" and #2094, which is also bilingual. That makes this PR a deviation from a convention its own sibling PRs follow, not an ambiguous or unenforced rule.
Nothing in the existing tooling blocks a non-bilingual PR from being opened or merged (there is no CI check enforcing this), so the gap is not caught automatically — it only surfaces on manual/author review, which is exactly why it is worth flagging here.
Proof / how to verify:
- Open AGENTS.md at the repo root and read the callout right after the CONTRIBUTING.md pointer — it states the bilingual title/body requirement in bold, applying to "every PR and every issue."
- Compare PR Add GLM-5.2 FP8 MI355X SGLang single-node agentic benchmarks #2272's title against the required format / <中文标题> — there is no "/" separator or Chinese text.
- Scan the PR body — Summary / Changes / Validation — for a "## 中文说明" section or any Chinese text mirroring the English summary; there is none.
- Compare against Add MiniMax M3 8k/1k Dynamo vLLM B300 EAGLE recipes / 新增 MiniMax M3 8k/1k Dynamo vLLM B300 EAGLE 配方 #2182's title in the git log, which correctly follows the bilingual format, confirming the convention is actively practiced, not dead documentation.
Fix: rename the PR title to append "/ 添加 GLM-5.2 FP8 MI355X SGLang 单节点智能体基准测试" (or an equivalent natural translation), and append a "## 中文说明" section to the description mirroring the Summary/Changes/Validation content in Chinese.
This is filed against the in-diff perf-changelog.yaml entry as the closest in-repo anchor, since the actual defect lives in the PR's title/description metadata on GitHub rather than in any single file's content. It is a mandatory process/metadata requirement with no effect on the benchmark recipe's correctness or CI runtime behavior — nothing breaks if this merges as-is, so it does not block merge, but should still be fixed to keep the PR consistent with the repository's stated (and actively followed) contribution convention.
Mirror the B300 sibling fix: GLM-5.2's chat template defaults to reasoning_effort=Max when the client passes no chat_template_kwargs, and the heavy thinking exhausts mini-swe-agent's default 75-step budget before submission (B300: 12/23 trajectories LimitsExceeded unsubmitted, 10/11 submitted resolved). SWEBENCH_AGENT_STEP_LIMIT=150, recipe-scoped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29650272796 |
The cookbook-verified v0.5.13.post1-rocm720-mi35x-20260618 build segfaults in AITER gemm_a8w8_blockscale_bpreshuffle mid-profiling under the agentic corpus (KV usage 0.95+, 5.3M pending tokens, untuned GLM-5.2 GEMM shapes falling back to default configs - gfx950 tuned configs are not upstream yet, ROCm/aiter#4243). Move to the newest rocm720 build for the latest AITER fallback fixes. Cap the conc list at 32 (was 64): the B300 sibling frontier showed no-offload points beyond ~c48 are KV-cliff-dominated on the 131k-token mean corpus, and MI355X prefill is slower still - c48/c64 only added crash surface and sweep time for dominated points. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Run 29651566222 post-mortem. The TP8 c32 job got through warmup and ~30 min of profiling, then AIPerf aborted on Pushed e9225ac:
If the segfault reproduces at c32 on the new image, the fallback plan is to cap the arm at the validated conc ≤16 band and mark the saturation points blocked on aiter#4243. |
…-sglang-agentic # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29651566222 |
v0.5.15.post1-rocm720-mi35x-20260717 cannot boot DSA models on ROCm at all: the dpsk_v4_topk_v2 JIT kernel includes the CUDA-only cooperative_groups.h header and fails to compile for gfx950, killing the server at startup. Pin v0.5.14-rocm720-mi35x-20260710 instead - the build the DSv4 MI355X agentic recipe (same DSA architecture family) runs green on with the same harness, corpus, and cluster, and whose AITER is three weeks newer than the segfaulting cookbook 20260618 pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29660172009 |
The topk v2 kernel (default since sglang v0.5.14) is JIT-compiled from CUDA-only source (cooperative_groups.h) and cannot build for gfx950, killing decode CUDA-graph capture at boot. v1 dispatches to the precompiled HIP op in sgl-kernel; upstream's MI355X CI runs DSA models with SGLANG_OPT_USE_TOPK_V2=false for the same reason. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Image pin + ROCm DSA topk fixTwo commits landed after on-node validation on
Validation evidence (short-prompt probes do not reproduce the AITER segfault, so a dedicated storm was used): 48 unique ~120k-token prompts at concurrency 24 against the pinned image + fix — all requests completed, no segfault, structured |
…ntry Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29673718350 |
1 participant
Summary
Adds
glm5.2-fp8-mi355x-sglang-agentic— GLM-5.2 FP8 single-node agentic (AgentX) benchmarks on MI355X with SGLang, from the SGLang cookbook MI355X FP8 recipes (FP8 is the cookbook-verified MI355X quant; NVFP4 is NVIDIA-only). TP8 with the DSA tilelang prefill/decode backends, STP only — MTP/EAGLE is not validated on ROCm gfx950, per the cookbook.The cookbook's low-latency / balanced / high-throughput strategies differ only in batch-shaping levers, so a single TP8 arm at conc
[1, 2, 4, 8, 16, 32, 48, 64]traces the throughput-vs-interactivity frontier, with the script deriving the levers from CONC (conc ≤16: chunked-prefill 131072, mem 0.80; conc ≥32: chunked-prefill 32768, mem 0.85).Changes
configs/amd-master.yaml: new entry onrunner: cluster:mi355x-amds; image pinned to the cookbook-verifiedlmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618(carries the gfx950 block-FP8 accuracy fix;glm47/glm45parsers verified present at that tag)benchmarks/single_node/agentic/glm5.2_fp8_mi355x_sglang.sh: new script; carries the lessons from the B300 bring-up (Add GLM-5.2 NVFP4 B300 SGLang single-node agentic benchmarks #2268):SGLANG_TIMEOUT_KEEP_ALIVE=900(AIPerf per-session keep-alive race),--tool-call-parser glm47+--reasoning-parser glm45(SWE-bench agentic evals),AGENTIC_WARMUP_GRACE_PERIOD=3600for the conc ≥32 bandbenchmarks/benchmark_lib.sh: same two edits as Add GLM-5.2 NVFP4 B300 SGLang single-node agentic benchmarks #2268, byte-identical so whichever PR merges second auto-merges cleanly —glm5.2*added to the unfiltered 1M-context agentic corpus branch, and--warmup-grace-periodmade overridable viaAGENTIC_WARMUP_GRACE_PERIOD(default unchanged at 1800s)runners/launch_mi355x-amds.sh: servezai-org/GLM-5.2*from the NFS hf-hub-cache (MiniMax-M3 pattern — 756 GB checkpoint, staged and verified: 141/141 shards match the index, world-readable/writable for the runner user); excludemia1-p01-g09/g11from single-node salloc (broken but sitting idle in slurm — g09 was already excluded on the multi-node disagg path)perf-changelog.yaml: entry appended at tailValidation
cluster:mi355x-amds); 223 matrix_logic tests pass; changelog key resolvesreasoning_content(glm45); structured tool-calls parse correctly via glm47; conc-64 band validation in flight (note:sglang.bench_servingis broken in this image build — missingsglang.benchmark.datasets— irrelevant to CI, which drives AIPerf)Related: #2268 (B300 NVFP4 agentic recipes for the same model).
🤖 Generated with Claude Code