[LMCache] Add LMCache arm on tuned DSV4 FP4 B300 vLLM AgentX recipe / 在调优后的 DSV4 FP4 B300 vLLM AgentX 配方上新增 LMCache 分支

[ApostaC]

Description

Adds the LMCache KV-offload backend (#2153) to the DeepSeek-V4 FP4 B300 vLLM AgentX recipe, rebased on the official B300 recipe from #2241, mirroring the B200 arm in #2231. LMCache runs on the same image and serving flags as the other arms so backends are directly comparable.

• benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh: adds an lmcache offload-backend case on the [AgentX] Update vLLM DeepSeek-V4 B300 aggregate / 更新 vLLM DeepSeek-V4 B300 聚合配置 #2241 recipe — LMCache 0.5.1 MP server (lmcache_driven transfer mode) + LMCacheMPConnector, L1 pool derated to 75% of TOTAL_CPU_DRAM_GB. Two lmcache-only deviations, both required and validated: (1) no PYTORCH_ALLOC_CONF=expandable_segments:True (VMM allocations cannot be legacy-CUDA-IPC-exported to the LMCache server; register_kv_caches fails with cudaErrorInvalidValue), (2) gpu-memory-utilization 0.92 on DEP8 (same as the official DEP8 derate) and 0.94 on TP/DEP4 (at 0.96 the bring-up sweep hit torch-pool, DeepGEMM-JIT, and cuBLAS-workspace OOMs — the LMCache GPU worker plus no expandable segments leaves <1 GiB headroom).
• configs/nvidia-master.yaml: LMCache runs inside the official dsv4-fp4-b300-vllm-agentic config (no separate section) — one lmcache arm per official arm at a conc offset: TP4 +4 [32,36,40,44] vs SimpleCPU [28,32,36,40]; DEP4 +4 [36,44,52,60,68,76] vs SimpleCPU [32,40,48,56,64,72]; DEP8 +8 capped at 208 [72,104,120,136,152,168,184,200] vs GPU-resident [64,96,112,128,144,160,176,192,224]. All ladders sit inside the ranges validated green (23/23 points) in bring-up runs 29463061871/29535333851.

Validated bring-up results (run 29535333851): peak DEP8 ~41.8k total tok/s/GPU at conc 192; DEP4 peak ~34.2k at conc 48; pure TP4 up to ~28.8k at conc 16 with 42+ tok/s/user.

Known blocker (upstream, not this PR): #1947 (SWE-bench Lite eval) introduced a changelog-validation regression — its eval pass emits agentic eval rows that ChangelogMatrixEntry.evals rejects, so check-changelog fails for any PR whose changelog entry touches an agentic config, including this one. A standalone fix PR is being prepared; this PR's sweep is blocked until it merges. (The SWE-bench agentic eval job also failed at runtime in an earlier sweep on this PR, so the #1947 feature likely needs a runtime fix as well.)

中文说明

将 LMCache KV 卸载后端（#2153）加入 DeepSeek-V4 FP4 B300 vLLM AgentX 配方，基于 #2241 的官方 B300 配方，对应 B200 的 #2231。LMCache 与其他分支使用相同镜像和 serving 参数，便于后端间直接对比。

• 基准测试脚本在 [AgentX] Update vLLM DeepSeek-V4 B300 aggregate / 更新 vLLM DeepSeek-V4 B300 聚合配置 #2241 配方上新增 lmcache 卸载后端分支——LMCache 0.5.1 MP server（lmcache_driven 传输模式）+ LMCacheMPConnector，L1 池按 TOTAL_CPU_DRAM_GB 的 75% 降额。lmcache 分支仅有两处必要且已验证的差异：（1）不设置 PYTORCH_ALLOC_CONF=expandable_segments:True（VMM 分配无法通过传统 CUDA IPC 导出给 LMCache server，register_kv_caches 报 cudaErrorInvalidValue）；（2）gpu-memory-utilization 在 DEP8 为 0.92（与官方 DEP8 降额一致）、TP/DEP4 为 0.94（0.96 时调试扫描出现 torch 内存池、DeepGEMM JIT 与 cuBLAS workspace 三类 OOM——LMCache 的 GPU worker 加上无 expandable segments 导致余量不足 1 GiB）。
• 主配置：LMCache 直接运行在官方 dsv4-fp4-b300-vllm-agentic 配置内（不再单列配置段）——每个官方分支对应一个并发偏移的 lmcache 分支：TP4 偏移 +4，[32,36,40,44]（对比 SimpleCPU [28,32,36,40]）；DEP4 偏移 +4，[36,44,52,60,68,76]（对比 SimpleCPU [32,40,48,56,64,72]）；DEP8 偏移 +8 且最大并发限制为 208，[72,104,120,136,152,168,184,200]（对比 GPU 驻留 [64,96,112,128,144,160,176,192,224]）。所有阶梯均处于调试扫描 run 29463061871/29535333851 **验证通过（23/23 测试点）**的范围内。

已验证的调试结果（run 29535333851）：DEP8 峰值约 41.8k total tok/s/GPU（并发 192）；DEP4 峰值约 34.2k（并发 48）；纯 TP4 在并发 16 时约 28.8k，且交互速度 42+ tok/s/user。

已知阻塞（上游问题，非本 PR）： #1947（SWE-bench Lite 评估）引入变更日志校验回归——其评估选择会生成 ChangelogMatrixEntry.evals 无法接受的 agentic 评估行，导致任何变更日志条目涉及 agentic 配置的 PR（包括本 PR）的 check-changelog 失败。独立修复 PR 正在准备中；在其合并前本 PR 的扫描被阻塞。（此前本 PR 的一次扫描中 SWE-bench agentic 评估任务在运行时也失败了，#1947 的功能可能还需要运行时修复。）

Related Issue

Builds on #2241 (official B300 AgentX recipe), #2153 (LMCache backend), and #2231 (B200 counterpart). Blocked by the #1947 changelog-validation regression. / 基于 #2241（官方 B300 AgentX 配方）、#2153（LMCache 后端）与 #2231（B200 对应版本）。受 #1947 变更日志校验回归阻塞。

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have tested my changes locally
• I have updated documentation if necessary
• If I changed a container image or config, I have already updated perf-changelog.yaml
  • New perf-changelog.yaml entries are appended to the end of the file (the file is chronological: oldest at top, newest at bottom)
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /reuse-sweep-run on this PR — do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps (remove and re-add the label to force one)

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

🔴 The new perf-changelog.yaml entry's pr-link uses pull/XXXX (four X's), but validate_perf_changelog.py's PR_LINK_PLACEHOLDERS only recognizes XXX (three X's) or the real canonical PR link. This mismatch means the placeholder will never get canonicalized to the real PR link during the merge-with-reuse flow. Fix by changing it to pull/XXX (three X's) or the real PR number (2232).

Extended reasoning...

The appended entry in perf-changelog.yaml (line 4827) sets:

pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX

four X characters. utils/validate_perf_changelog.py defines:

CANONICAL_PR_LINK = re.compile(r"https://github.com/SemiAnalysisAI/InferenceX/pull/\d+")
PR_LINK_PLACEHOLDERS = {
    "XXX",
    "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX",
}

Both of these expect exactly three X's, not four. pull/XXXX therefore matches neither the canonical digit-based link pattern nor the placeholder set.

Step-by-step proof:

1. validate_added_pr_link() is invoked with link = "https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX".
2. If pr_number is None (e.g., a main-branch validation run), the code requires CANONICAL_PR_LINK.fullmatch(link). The regex requires \d+ after pull/; XXXX is not digits, so fullmatch returns None and the check fails.
3. If pr_number is set (a PR-run, e.g. 2232), the code computes expected = ".../pull/2232" and checks link not in PR_LINK_PLACEHOLDERS and link != expected. "https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX" is not in PR_LINK_PLACEHOLDERS (which only contains the three-X forms) and is not equal to expected, so the condition is true and ChangelogValidationError("new PR entry must use ... or an XXX placeholder; found ...") is raised.
4. Either way, the placeholder as written is rejected.

This check is reached at merge time via prepare_perf_changelog_merge.py's canonicalize/resolve-conflict subcommands, which utils/merge_with_reuse.sh invokes with the real --pr-number to rewrite placeholder links to the real PR URL. Because compare_entries() → validate_added_pr_link() rejects pull/XXXX outright, that canonicalization step will fail, meaning the reuse-merge tooling cannot resolve this entry to the real PR link and the merge flow breaks for this changelog entry. (Separately, the PR-time run-sweep.yml CI invocation of validate_perf_changelog.py only runs validate_generated_config(), which does not call this comparison path, so it will not itself fail at PR-review time — but the PR description's claim that "validate_perf_changelog.py against origin/main passes" is not evidence this placeholder is valid, since that path isn't exercised by the local check either.)

The fix is a one-character edit: change pull/XXXX to pull/XXX (three X's) to match the accepted placeholder, or simply write the real PR number 2232 directly.

[claude]

🔴 The new LMCache arm's --kv-transfer-config sets kv_connector":"LMCacheMPConnector" but omits kv_connector_module_path, unlike the two sibling scripts (dsv4_fp4_mi355x_vllm.sh:344, kimik2.5_fp4_b200.sh:150) that wire up the same connector. Since LMCacheMPConnector is out-of-tree, vLLM will not be able to locate the class and vllm serve should fail at KV-connector construction, breaking every point in the new dsv4-fp4-b300-vllm-agentic-lmcache config. Fix by adding "kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector" to the config dict.

Extended reasoning...

The bug: In the lmcache) case of benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh (around lines 254-259), OFFLOAD_ARGS is built as:

\n"{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{...}}"\n

This JSON blob is missing the kv_connector_module_path key.

Why it matters: LMCacheMPConnector is not a vLLM built-in connector (unlike MooncakeStoreConnector, which vLLM resolves through its internal connector-factory registry). It lives in the external lmcache package, at lmcache.integration.vllm.lmcache_mp_connector. vLLM's KVTransferConfig resolves out-of-tree connectors by dynamically importing the module named in kv_connector_module_path before looking up the connector class by name — without that field, vLLM has no way to find the class.

Confirmed by direct comparison with sibling scripts: I checked the two other scripts in this repo that wire up the identical LMCacheMPConnector:

• benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm.sh:344: "kv_connector\":\"LMCacheMPConnector\",\"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\"...
• benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh:150 (also lmcache 0.5.1, matching this PR's version exactly): same pattern.

Both sibling scripts explicitly set kv_connector_module_path alongside kv_connector. This new B300 arm is the only one of the three that omits it.

Why nothing else catches this: The script does run python3 -c \"import lmcache.integration.vllm.lmcache_mp_connector\" earlier, but that only verifies the module is importable in the launcher process — it says nothing to vLLM about where to import the connector class from when it parses --kv-transfer-config. bash -n, the matrix-logic pytest suite, and generate_sweep_configs.py all validate shell syntax and config-generation logic, not the semantics of the JSON payload passed to vLLM, so none of them would catch this.

Step-by-step proof of the failure:

1. KV_OFFLOAD_BACKEND=lmcache is selected, LMCache MP server starts and passes its healthcheck.
2. OFFLOAD_ARGS is set to --kv-transfer-config '{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{...}}' (no kv_connector_module_path).
3. vllm serve parses this into a KVTransferConfig. Since kv_connector_module_path is absent, vLLM falls back to its built-in connector-factory lookup by name.
4. LMCacheMPConnector is not registered there (it is only importable via the external lmcache package), so the lookup fails and vLLM raises an error during KV-connector construction, before the server can start serving.
5. Every concurrency point in the new dsv4-fp4-b300-vllm-agentic-lmcache search space (17 points total) uses this same code path, so the entire new config fails at server startup, not just some points.

Fix: add \"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\" to the JSON dict at the same spot the two sibling scripts do, e.g.:

\n"{\"kv_connector\":\"LMCacheMPConnector\",\"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{...}}"\n

Severity is normal, not a nit or pre-existing issue: this is new code introduced by this PR, and merging as-is causes a concrete, deterministic startup failure for the entire new lmcache arm — not a stylistic or description mismatch.

[claude]

🟡 Line 2 weakens set -euo pipefail to set -eo pipefail, dropping nounset — this is the only script among 23 in benchmarks/single_node/agentic/ that does so (its B200 sibling dsv4_fp4_b200_vllm.sh from #2231 handles the exact same optionally-unset vars with ${VAR:-} instead). Recommend restoring set -u and using ${KV_OFFLOAD_BACKEND:-}, ${SLURM_JOB_ID:-}, and ${MODEL_PATH:-} in the new unguarded expansions instead of disabling nounset for the whole ~350-line script.

Extended reasoning...

What changed: Line 2 of dsv4_fp4_b300_vllm.sh changes set -euo pipefail to set -eo pipefail, dropping the -u (nounset) flag. I verified this is the only script among all 23 scripts in benchmarks/single_node/agentic/ that lacks nounset — every other script, including the direct B200 counterpart dsv4_fp4_b200_vllm.sh introduced in #2231 (which this PR explicitly says it mirrors), keeps set -euo pipefail.

Why it was dropped: The rewrite replaces the old declare -p VAR >/dev/null 2>&1 && [ -n "$VAR" ] guards for MODEL_PATH/SLURM_JOB_ID with plain [[ -n "$VAR" ]], and adds a new case "$KV_OFFLOAD_BACKEND" in "") branch that relies on KV_OFFLOAD_BACKEND expanding to empty when unset. None of KV_OFFLOAD_BACKEND, SLURM_JOB_ID, or MODEL_PATH are in the check_env_vars allowlist, so under nounset these bare references would abort with "unbound variable" whenever the var isn't exported (e.g. a stand-alone run without SLURM, or the vllm-simple/mooncake KV_OFFLOAD_BACKEND path). Dropping -u script-wide is the blunt fix for that.

Why this is unnecessary: The B200 sibling script that this PR is explicitly mirroring solves the identical problem without sacrificing nounset, using the standard ${VAR:-} parameter-expansion idiom: [[ -n "${SLURM_JOB_ID:-}" ]], ${SLURMD_NODENAME:-unknown}, [[ -n "${MODEL_PATH:-}" ]]. benchmark_lib.sh itself uses this idiom throughout. So the intent (let three specific optional vars expand to empty) can be fully achieved without disabling the safety net for the other ~350 lines of the script — the fix conflated "these three vars are allowed to be unset" with "disable unbound-variable checking everywhere."

Impact: With nounset off, any future typo in a variable reference anywhere in this script (e.g. a misspelled $LMCACHE_PORT, $TOTAL_CPU_DRAM_GB, or $GPU_COUNT) will silently expand to an empty string instead of aborting immediately with "unbound variable." Concretely: GPU_COUNT=$TP further down is guarded by a regex check so a typo there would still be caught, but a hypothetical typo inside one of the arithmetic expressions (e.g. LMCACHE_L1_SIZE_GB=$((TOTAL_CPU_DRAM_GB * 3 / 4)) misspelled as TOTAL_CPU_DRM_GB) would silently evaluate to 0 rather than failing fast, producing a confusing downstream failure (or a nonsensical LMCache pool size) instead of an immediate, obvious error at the point of the typo.

Step-by-step proof of the regression mechanism (not a live failure today, since check_env_vars still validates required vars and the arithmetic guards catch $GPU_COUNT):

1. Suppose a future edit misspells $TOTAL_CPU_DRAM_GB as $TOTAL_CPU_DRAM_G inside the lmcache branch's LMCACHE_L1_SIZE_GB=$((TOTAL_CPU_DRAM_GB * 3 / 4)) line.
2. Under set -u (the previous/sibling-script behavior), bash would immediately abort with TOTAL_CPU_DRAM_G: unbound variable, pointing directly at the typo.
3. Under this PR's set -eo pipefail (nounset disabled), the misspelled var silently expands to empty string, the arithmetic becomes $(( * 3 / 4 )) → bash arithmetic treats an empty operand as 0, so LMCACHE_L1_SIZE_GB=0.
4. The script proceeds to start lmcache server --l1-size-gb 0 ..., which either fails deep inside the LMCache server startup logs (unclear connection to the root cause) or silently runs a degenerate 0-size cache — either way, a much harder failure to root-cause than an immediate bash error at the typo site.

Fix: Restore set -euo pipefail on line 2, and change the three unguarded references to use parameter expansion consistent with the B200 sibling: [[ -n "${SLURM_JOB_ID:-}" ]], ${SLURMD_NODENAME:-unknown}, [[ -n "${MODEL_PATH:-}" ]], and case "${KV_OFFLOAD_BACKEND:-}" in "").

On severity: All four verifiers independently confirmed the factual claim (only script in the directory without nounset, unnecessary given the ${VAR:-} idiom is used successfully in the sibling script) but converged on nit rather than normal, and I agree. check_env_vars still validates every genuinely required variable, GPU_COUNT is separately regex-validated, and the script runs correctly today as written on every current sweep point — the harm is a hypothetical future maintenance hazard, not a concrete failure in this PR. That's a real, worth-fixing consistency/robustness regression, but not a merge blocker.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29463061871
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29463061871

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29535333851
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29535333851

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29535333851
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29535333851

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29609383045
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29609383045