[AgentX] vLLM DeepSeek-V4 B200 aggregate MTP#2259
Conversation
…tion #1947 made single-node agentic recipes generate a SWE-bench eval row (run_eval/eval_only + agentic fields) but never widened the changelog matrix schema, so ChangelogMatrixEntry.evals (typed list[SingleNodeMatrixEntry], fixed-seq-len only) rejects every agentic eval row -- breaking check-changelog for any single-node agentic PR. Widen evals to the same Union single_node already uses, and give SingleNodeAgenticMatrixEntry optional run_eval/eval_only (None-default, so benchmark rows are unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r every agentic arm
Add an MTP speculative-decoding twin (spec-decoding: mtp,
num_speculative_tokens=3) for every arm of dsv4-fp4-b200-vllm-agentic
(TP8 GPU-resident, TP8 SimpleCPU, DEP8 SimpleCPU, DEP8 Mooncake), each
mirroring its non-MTP conc-list, routed via the launcher's
spec-decoding=mtp suffix to dsv4_fp4_b200_vllm_mtp.sh.
New dsv4_fp4_b200_vllm_mtp.sh forks dsv4_fp4_b200_vllm.sh with only the MTP
deltas: --speculative-config {"method":"mtp","num_speculative_tokens":3} and
--max-cudagraph-capture-size scaled to MAX_NUM_SEQS*(1+N) tokens so
FULL_DECODE_ONLY still covers the largest decode batches.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Document the dsv4-fp4-b200-vllm-agentic MTP twins (num_speculative_tokens=3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29565769390 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| --speculative-config "{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS}" | ||
| --no-disable-hybrid-kv-cache-manager | ||
| --disable-uvicorn-access-log | ||
| --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}' | ||
| --max-num-seqs "$MAX_NUM_SEQS" |
There was a problem hiding this comment.
🔴 The new agentic script benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh runs real (non-synthetic) MTP decoding instead of the fairness-mandated synthetic acceptance pinned to the golden AL curve. Its --speculative-config only sets method/num_speculative_tokens, missing rejection_sample_method=synthetic and synthetic_acceptance_length (2.49 thinking_on / 2.97 thinking_off from golden_al_distribution/dsv4_mtp.yaml), unlike the precedent kimik2.5_fp4_b300_mtp.sh script.
Extended reasoning...
The bug: docs/PR_REVIEW_CHECKLIST.md contains a mandatory, agentic-scoped CODEOWNER sign-off item (line 23):
For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in
golden_al_distribution/for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
golden_al_distribution/README.md's "Fairness Guidelines for AgentX" section reinforces this and shows the exact vLLM mechanism: inject the golden value via rejection_sample_method: "synthetic" + synthetic_acceptance_length: <value> in --speculative-config. The committed curve at golden_al_distribution/dsv4_mtp.yaml gives deepseek-v4-pro, num_speculative_tokens=3 → AL 2.49 (thinking_on) / 2.97 (thinking_off).
Where it breaks: In the new file benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh (line ~260), the --speculative-config passed to vllm serve is:
--speculative-config "{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS}"This has no rejection_sample_method or synthetic_acceptance_length field, so vLLM performs real (non-synthetic) MTP verification against the actual draft model's predictions rather than sampling acceptance at a fixed, pinned rate.
Why nothing else catches it: The only other agentic MTP/EAGLE script in the repo, benchmarks/single_node/agentic/kimik2.5_fp4_b300_mtp.sh, correctly follows the fairness contract — it sets rejection_sample_method:"synthetic" and synthetic_acceptance_length (2.88 / 3.24, selected by DCP_SIZE branch) directly in its --speculative-config string. The new dsv4 script has no equivalent branch or env-var override anywhere in the file that injects these fields; VLLM_CMD is built once, inline, with the literal string above, so there is no other code path that could compensate.
Impact: Without pinning acceptance to the golden curve, the measured acceptance length (and therefore decode throughput/speedup from MTP) is free to drift with whatever the real draft model happens to achieve on the agentic replay traffic. This defeats the stated purpose of the golden-AL mechanism — making agentic MTP speedup numbers comparable across submissions/vendors — and is exactly the failure mode the checklist item and Fairness Guidelines were written to prevent. It is also a mandatory, explicit CODEOWNER sign-off checklist item, not a style preference.
Step-by-step proof:
- A reviewer runs the new pure-TP+MTP tier added in
configs/nvidia-master.yaml(spec-decoding: mtp,conc-list: [1,2,4,6,8]). dsv4_fp4_b200_vllm_mtp.shlaunchesvllm servewith--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'.- vLLM's MTP draft head proposes tokens and the target model verifies them for real — acceptance depends on true model/draft agreement on the actual agentic trace content, which is not controlled.
- Compare to
kimik2.5_fp4_b300_mtp.sh: its config additionally carries"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.88, forcing every decode step to accept exactly the pinned AL regardless of true draft/target agreement. - Because dsv4's config lacks those two fields, its published throughput/TPOT numbers reflect whatever real AL the traces produce that day/run, not the committed
2.49/2.97golden value — so DeepSeek-V4 MTP results are not apples-to-apples comparable to other vendors' agentic MTP submissions the way the fairness guideline requires.
Fix: Add rejection_sample_method: "synthetic" and synthetic_acceptance_length: <2.49 or 2.97> (selected based on thinking mode, mirroring how kimik2.5_fp4_b300_mtp.sh branches on DCP_SIZE) to the --speculative-config JSON at line ~260.
Note: fixed_seq_len/dsv4_fp4_b200_vllm_mtp.sh also uses real decoding, but the checklist item is explicitly scoped to agentic workloads, so that script is out of scope for this finding.
| search-space: | ||
| # Pure TP at low concurrency. | ||
| - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 6, 8] } | ||
| # Pure TP GPU-resident + MTP (num_speculative_tokens=3) | ||
| - { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 6, 8] } | ||
| - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 12, 16] } | ||
| # Pure TP SimpleCPU + MTP (num_speculative_tokens=3) | ||
| - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, spec-decoding: mtp, conc-list: [8, 12, 16] } | ||
| # DEP | ||
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80], router: { name: vllm-router, version: "0.1.14" } } | ||
| # DEP SimpleCPU + MTP (num_speculative_tokens=3) | ||
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, spec-decoding: mtp, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80], router: { name: vllm-router, version: "0.1.14" } } | ||
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } } | ||
| # DEP Mooncake + MTP (num_speculative_tokens=3) | ||
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, spec-decoding: mtp, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } } | ||
|
|
||
| dsv4-fp4-b200-trt: |
There was a problem hiding this comment.
🔴 The PR adds 6 new MTP search-space rows (and the new dsv4_fp4_b200_vllm_mtp.sh script) to dsv4-fp4-b200-vllm-agentic in configs/nvidia-master.yaml, but never appends the required entry to perf-changelog.yaml. Since .github/workflows/run-sweep.yml gates both its push and pull_request triggers on paths: perf-changelog.yaml, the benchmark sweep will never trigger for this PR, leaving the new MTP arms unbenchmarked and unvalidated by CI. Please append a config-keys: [dsv4-fp4-b200-vllm-agentic] entry to perf-changelog.yaml before merging.
Extended reasoning...
What's missing
This PR introduces 6 new MTP-variant agentic search-space rows under dsv4-fp4-b200-vllm-agentic in configs/nvidia-master.yaml (MTP counterparts of the pure-TP, SimpleCPU-offload, and DEP/Mooncake tiers), plus the new benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh script that actually runs them. However, the PR touches only three files — the new script, configs/nvidia-master.yaml, and utils/matrix_logic/validation.py — and never appends an entry to perf-changelog.yaml.
Why this matters (not just a doc nit)
AGENTS.md's "Adding a benchmark configuration" section is explicit: "Add entries to configs/nvidia-master.yaml ..., append to perf-changelog.yaml, then validate...". That alone would just be a process nit, except perf-changelog.yaml is not merely documentation — it is the mechanism that drives CI. .github/workflows/run-sweep.yml gates both its push (to main) and pull_request triggers on:
paths:
- "perf-changelog.yaml"
I verified this directly in the workflow file (lines ~22-36). Since this PR never modifies perf-changelog.yaml, the check-changelog/validate-perf-changelog job and the downstream benchmark-sweep job never run for this PR at all — not "run and pass trivially," but never invoked in the first place.
Proof / concrete walkthrough
- I checked the tail of
perf-changelog.yaml: the most recent entries reference PR [AgentX] Update vLLM DeepSeek-V4 B300 aggregate / 更新 vLLM DeepSeek-V4 B300 聚合配置 #2241 (dsv4-fp4-b300-vllm-agentic), Add Kimi K2.6 NVFP4 B300 EAGLE3 AgentX benchmark / 新增 Kimi K2.6 NVFP4 B300 EAGLE3 AgentX 基准测试 #2228, and [NV] Add GLM-5 NVFP4 GB200 disaggregated Dynamo TensorRT-LLM MTP benchmarks / [NV] 新增 GLM-5 NVFP4 GB200 分离式 Dynamo TensorRT-LLM MTP 基准测试 #1800 — there is no entry for this PR or its new MTP config keys. run-sweep.yml'son.pull_request.pathsandon.push.pathsboth list onlyperf-changelog.yaml.- GitHub Actions path-filtered triggers only fire the workflow when a commit in the PR touches a file matching that path. This PR's diff (
dsv4_fp4_b200_vllm_mtp.sh,configs/nvidia-master.yaml,utils/matrix_logic/validation.py) matches none of those paths. - Therefore
run-sweep.ymldoes not trigger on this PR's pushes/synchronize events, socheck_changelog/process_changelog.pynever runs, never discovers the 6 new MTP rows (it discovers rows to benchmark by diffingperf-changelog.yamlbetween base and head), and the sweep infrastructure that would exercisedsv4_fp4_b200_vllm_mtp.shnever executes. - Net effect: the new MTP arms — the entire purpose of this PR — go unbenchmarked and unvalidated by CI, silently. Nothing fails loudly; the workflow simply never starts.
Precedent
Every comparable recent PR in this repo followed the append-to-changelog step: #2224 (dsv4-fp4-b200-vllm-agentic), #2241 (dsv4-fp4-b300-vllm-agentic), #2228 (kimik2.5-fp4-b300-vllm-agentic-mtp), and #1800 (glm5-fp4-gb200-dynamo-trt-mtp) all appended a perf-changelog.yaml entry alongside their config changes. This PR breaks that established, repository-mandated pattern.
Fix
Append a new entry to the end of perf-changelog.yaml, e.g.:
- config-keys:
- dsv4-fp4-b200-vllm-agentic
description:
- "Add MTP (num_speculative_tokens=3) variants of the pure-TP, SimpleCPU-offload, and DEP/Mooncake agentic tiers for B200 AgentX."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2259This is not a stylistic ask — it is the mechanism by which the sweep workflow actually triggers, so omitting it defeats the PR's own verification.
Restructure so the sweep runs only the new MTP work, not the existing aggregate: - Revert dsv4-fp4-b200-vllm-agentic to its main (#2224) search space -- unchanged vs main, so it is not re-run. - Add dsv4-fp4-b200-vllm-agentic-mtp: MTP twins (num_speculative_tokens=3) of the aggregate arms (TP8 GPU-resident, TP8 SimpleCPU, DEP8 SimpleCPU, DEP8 Mooncake), each mirroring its non-MTP conc-list. - Point the perf-changelog entry at only the new key, so only it sweeps. (No separate TP8 key: B200's TP8 arms already exist in the #2224 aggregate.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use MTP synthetic rejection sampling with acceptance length 2.49, the dsv4-pro golden AL (thinking_on, num_speculative_tokens=3) from golden_al_distribution/dsv4_mtp.yaml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-mtp # Conflicts: # perf-changelog.yaml
Lower gpu-memory-utilization for the B200 vLLM MTP recipe to 0.9 for extra headroom. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29566610531 |
Eval-only, agentic-coding-scoped entry for dsv4-fp4-b300-vllm-agentic so the PR sweep (and the on-push main sweep after merge) runs exactly one job: the DEP8 top-concurrency SWE-bench eval through the new sweep-agentic-evals dispatch, confirming the fix for the eval-only failures first seen on #2258/#2259. 中文：新增 changelog 条目以端到端验证 agentic 评估调度——针对 dsv4-fp4-b300-vllm-agentic 的 evals-only、仅 agentic-coding 场景条目，使 PR 扫描（及合并后 main 分支的 push 触发扫描）只运行一个任务：DEP8 最高并发 的 SWE-bench 评估经由新的 sweep-agentic-evals 调度执行，确认对 #2258/#2259 上首次出现的 eval-only 失败的修复。
… sweep job / 修复 agentic SWE-bench 评估调度：新增 agentic_evals 独立分桶与扫描任务 (#2267) * fix: dispatch agentic SWE-bench eval rows via a dedicated bucket and sweep job Since #1947, generate_sweep_configs --evals-only marks the top-concurrency agentic arm of each config as a SWE-bench eval row, but two consumers were never taught about agentic eval rows, so any PR touching a single-node agentic-coding config failed its sweep: 1. Schema: ChangelogMatrixEntry.evals only accepted SingleNodeMatrixEntry (fixed-seq-len), so process_changelog.py's final model_validate rejected the agentic row and check-changelog failed before any GPU work. 2. Dispatch: even with the schema widened, run-sweep.yml's sweep-evals job dispatches every eval row with fixed-seq-len inputs; an agentic row gets empty ISL/OSL/MAX_MODEL_LEN and scenario-type defaults to fixed-seq-len, so the launcher runs the fixed_seq_len benchmark script, which exits on check_env_vars. Fix: route agentic eval rows into a new agentic_evals bucket (mirroring the single_node['agentic'] design) dispatched by a new sweep-agentic-evals job cloned from sweep-agentic's input block plus run-eval/eval-only, so an agentic row can never reach the fixed-seq-len eval dispatch. The agentic model accepts optional run-eval/eval-only carried by eval rows; exclude_none keeps benchmark row output byte-identical. 中文：自 #1947 起，generate_sweep_configs --evals-only 会将每个配置中并发度最高的 agentic 搜索空间条目标记为 SWE-bench 评估行，但下游两个消费方并不认识 agentic 评估行，导致任何涉及单节点 agentic-coding 配置的 PR 扫描失败：其一，schema 层 ChangelogMatrixEntry.evals 仅接受固定序列长度条目，check-changelog 直接报错； 其二，调度层 sweep-evals 以固定序列长度的输入派发评估行，ISL/OSL/MAX_MODEL_LEN 为空且 scenario-type 回落为 fixed-seq-len，启动器因此运行错误的基准测试脚本并 在 check_env_vars 处退出。本修复新增 agentic_evals 独立分桶（与 single_node['agentic'] 设计一致），由新的 sweep-agentic-evals 任务按 sweep-agentic 的输入派发并附加 run-eval/eval-only，使 agentic 评估行不可能进入 固定序列长度的评估调度路径；基准测试行的输出保持逐字节不变。 * test: trim agentic eval bucket tests to essential coverage Collapse fixtures into module constants, merge the accept/reject checks into one test, and drop redundant cases; added test lines go from 212 to 100 with the two core guards kept (agentic rows validate only in agentic_evals; realistic rows split end-to-end through the real schema). 中文：精简 agentic 评估分桶测试——fixture 合并为模块常量，接受/拒绝检查合并 为单个测试并移除冗余用例，新增测试行数从 212 降至 100，保留两项核心守卫 （agentic 行仅能通过 agentic_evals 校验；真实行形状端到端经过真实 schema 拆分）。 * chore: add changelog entry to validate agentic eval dispatch end-to-end Eval-only, agentic-coding-scoped entry for dsv4-fp4-b300-vllm-agentic so the PR sweep (and the on-push main sweep after merge) runs exactly one job: the DEP8 top-concurrency SWE-bench eval through the new sweep-agentic-evals dispatch, confirming the fix for the eval-only failures first seen on #2258/#2259. 中文：新增 changelog 条目以端到端验证 agentic 评估调度——针对 dsv4-fp4-b300-vllm-agentic 的 evals-only、仅 agentic-coding 场景条目，使 PR 扫描（及合并后 main 分支的 push 触发扫描）只运行一个任务：DEP8 最高并发 的 SWE-bench 评估经由新的 sweep-agentic-evals 调度执行，确认对 #2258/#2259 上首次出现的 eval-only 失败的修复。 * ws
…x validation" This reverts commit 6efd246.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29615786644 |
1 participant
No description provided.