Fix agentic SWE-bench eval dispatch: dedicated agentic_evals bucket + sweep job / 修复 agentic SWE-bench 评估调度：新增 agentic_evals 独立分桶与扫描任务#2267
Conversation
…sweep job Since #1947, generate_sweep_configs --evals-only marks the top-concurrency agentic arm of each config as a SWE-bench eval row, but two consumers were never taught about agentic eval rows, so any PR touching a single-node agentic-coding config failed its sweep: 1. Schema: ChangelogMatrixEntry.evals only accepted SingleNodeMatrixEntry (fixed-seq-len), so process_changelog.py's final model_validate rejected the agentic row and check-changelog failed before any GPU work. 2. Dispatch: even with the schema widened, run-sweep.yml's sweep-evals job dispatches every eval row with fixed-seq-len inputs; an agentic row gets empty ISL/OSL/MAX_MODEL_LEN and scenario-type defaults to fixed-seq-len, so the launcher runs the fixed_seq_len benchmark script, which exits on check_env_vars. Fix: route agentic eval rows into a new agentic_evals bucket (mirroring the single_node['agentic'] design) dispatched by a new sweep-agentic-evals job cloned from sweep-agentic's input block plus run-eval/eval-only, so an agentic row can never reach the fixed-seq-len eval dispatch. The agentic model accepts optional run-eval/eval-only carried by eval rows; exclude_none keeps benchmark row output byte-identical. 中文：自 #1947 起，generate_sweep_configs --evals-only 会将每个配置中并发度最高的 agentic 搜索空间条目标记为 SWE-bench 评估行，但下游两个消费方并不认识 agentic 评估行，导致任何涉及单节点 agentic-coding 配置的 PR 扫描失败：其一，schema 层 ChangelogMatrixEntry.evals 仅接受固定序列长度条目，check-changelog 直接报错； 其二，调度层 sweep-evals 以固定序列长度的输入派发评估行，ISL/OSL/MAX_MODEL_LEN 为空且 scenario-type 回落为 fixed-seq-len，启动器因此运行错误的基准测试脚本并 在 check_env_vars 处退出。本修复新增 agentic_evals 独立分桶（与 single_node['agentic'] 设计一致），由新的 sweep-agentic-evals 任务按 sweep-agentic 的输入派发并附加 run-eval/eval-only，使 agentic 评估行不可能进入 固定序列长度的评估调度路径；基准测试行的输出保持逐字节不变。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — this correctly plugs the agentic SWE-bench eval dispatch gap by mirroring the existing single_node['agentic'] → sweep-agentic pattern.
Extended reasoning...
Overview
The PR adds a new agentic_evals bucket (schema in validation.py, split logic in process_changelog.py) and a dedicated sweep-agentic-evals job in run-sweep.yml to fix a real bug: single-node agentic-coding eval rows previously had no valid path through ChangelogMatrixEntry and, once schema-widened, would have been dispatched with fixed-seq-len inputs (empty ISL/OSL/MAX_MODEL_LEN), launching the wrong benchmark script. recover_failed_ingest.py is updated to count the new bucket in its eval_jobs stat.
Security risks
None. This is CI/workflow orchestration and Pydantic schema code with no user-facing surface, auth, or secrets handling.
Level of scrutiny
Moderate — this is CI-critical infrastructure (a broken change would block all PRs touching agentic configs, as the linked incident shows), but the change is well-contained: it adds a new parallel bucket/job rather than modifying existing dispatch paths, and the risk of the existing fixed-seq-len path being broken is essentially nil (the evals bucket still explicitly rejects agentic rows, verified by a new guard test).
Other factors
I independently verified the change is properly scoped: generate_sweep_configs.py's mark_eval_entries/mark_all_eval_entries only ever mark single-node agentic rows as eval rows (multi-node agentic eval is explicitly commented as unsupported), so the new agentic_evals bucket and sweep-agentic-evals job cover the entire reachable case — there's no multi-node agentic eval row that would still hit the old broken path. Tests exercise the real model_validate (not mocked), which is exactly what was missing before and what let this bug reach CI in the first place. The trigger-ingest/trigger-agentic-ingest non-goal called out in the PR description is accurate — it's a pre-existing gating condition left untouched, not a regression from this change.
Collapse fixtures into module constants, merge the accept/reject checks into one test, and drop redundant cases; added test lines go from 212 to 100 with the two core guards kept (agentic rows validate only in agentic_evals; realistic rows split end-to-end through the real schema). 中文：精简 agentic 评估分桶测试——fixture 合并为模块常量，接受/拒绝检查合并 为单个测试并移除冗余用例，新增测试行数从 212 降至 100，保留两项核心守卫 （agentic 行仅能通过 agentic_evals 校验；真实行形状端到端经过真实 schema 拆分）。
Eval-only, agentic-coding-scoped entry for dsv4-fp4-b300-vllm-agentic so the PR sweep (and the on-push main sweep after merge) runs exactly one job: the DEP8 top-concurrency SWE-bench eval through the new sweep-agentic-evals dispatch, confirming the fix for the eval-only failures first seen on #2258/#2259. 中文：新增 changelog 条目以端到端验证 agentic 评估调度——针对 dsv4-fp4-b300-vllm-agentic 的 evals-only、仅 agentic-coding 场景条目，使 PR 扫描（及合并后 main 分支的 push 触发扫描）只运行一个任务：DEP8 最高并发 的 SWE-bench 评估经由新的 sweep-agentic-evals 调度执行，确认对 #2258/#2259 上首次出现的 eval-only 失败的修复。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29633097900 |
1 participant
Problem
Since #1947,
generate_sweep_configs --evals-onlymarks the top-concurrency agentic arm of each config as a SWE-bench eval row, but two downstream consumers were never taught about agentic eval rows, so any PR touching a single-node agentic-coding config fails its sweep (first seen on #2258):ChangelogMatrixEntry.evalsis typedlist[SingleNodeMatrixEntry](fixed-seq-len only), soprocess_changelog.py's finalmodel_validaterejects the agentic eval row with 8 errors (missingisl/osl/max-model-len/disagg, forbiddenkv-offloading/total-cpu-dram-gb/duration/scenario-type).check-changelogfails before any GPU work: example run.agentx/dsv4-v300-vllm-mtp),sweep-evalsdispatches every eval row with fixed-seq-len inputs:ISL/OSL/MAX_MODEL_LENcome out empty,scenario-typedefaults tofixed-seq-len, and the launcher runs the fixed_seq_len benchmark script, which exits atcheck_env_vars: example run.Fix
Route agentic eval rows into their own bucket, mirroring how the repo already separates agentic benchmark rows (
single_node['agentic']→ dedicatedsweep-agenticjob with a static, correct input block):validation.py:ChangelogMatrixEntry.agentic_evals: list[SingleNodeAgenticMatrixEntry]; the agentic model accepts optionalrun-eval/eval-only(carried by eval rows;exclude_nonekeeps benchmark-row output byte-identical, preserving artifact-reuse row shapes).evalsstays fixed-seq-len-only, so an agentic row reaching the fixed-seq-len dispatch is now structurally impossible — it failscheck-changeloginstead.process_changelog.py: split single-node eval rows byscenario-typeintoevals/agentic_evals.run-sweep.yml: newsweep-agentic-evalsjob cloned fromsweep-agentic's input block (forwardsscenario-type,kv-offloading,kv-offload-backend(+metadata),total-cpu-dram-gb,duration, stubsisl/osl/max-model-lento'0') plusrun-eval: true/eval-only: true; wired intocollect-evals. This satisfies the agentic scripts'check_env_varsgate and routes the launcher tobenchmarks/single_node/agentic/, whoseEVAL_ONLY=truebranch runs the SWE-benchrun_evalpath from [WIP] add SWE-bench Lite accuracy eval / 添加 SWE-bench Lite 准确率评估 #1947.recover_failed_ingest.py: countagentic_evalsineval_jobs(uses.get, so pre-existing reuse artifacts without the key still work; on the workflow side a missing key yieldsnull, which the job's!= 'null'condition skips).process_changelogtests mocksubprocess.runwithstdout="[]", so no realistic row ever hit the schema — CI was the first place this could fail. Added an end-to-end split test with realistic fixed + agentic eval rows flowing through the realChangelogMatrixEntry.model_validate, plusChangelogMatrixEntrybucket tests (including a guard thatevalsstill rejects agentic rows, and that benchmark-row dumps don't gain eval flags).python -m pytest utils/ -q: 543 passed; the 2 failures inutils/evals/test_run_eval_dispatch.pyare pre-existing onmain(environment-dependent, they shell out to mini-swe-agent) and unrelated.Non-goal:
trigger-ingest/trigger-agentic-ingestgating is unchanged; whether a main-push run with only agentic evals (agenticevals-onlyentry) should route to the agentic ingest is left as a follow-up.After merge,
agentx/dsv4-v300-vllm-mtp(#2258) should rebase onto main and drop itsvalidation.pyunion commit.中文说明
问题
自 #1947 起，
generate_sweep_configs --evals-only会将每个配置中并发度最高的 agentic 条目标记为 SWE-bench 评估行，但下游两个消费方并不认识 agentic 评估行，导致任何涉及单节点 agentic-coding 配置的 PR 扫描失败（最早出现在 #2258）：ChangelogMatrixEntry.evals的类型是list[SingleNodeMatrixEntry]（仅固定序列长度），process_changelog.py最终的model_validate会以 8 个校验错误拒绝 agentic 评估行，check-changelog在任何 GPU 任务开始前即失败。agentx/dsv4-v300-vllm-mtp分支曾尝试），sweep-evals仍按固定序列长度的输入派发所有评估行：ISL/OSL/MAX_MODEL_LEN为空、scenario-type回落为fixed-seq-len，启动器因此运行 fixed_seq_len 目录下的基准测试脚本，并在check_env_vars处退出。修复
将 agentic 评估行放入独立分桶，与仓库现有的 agentic 基准测试行设计一致（
single_node['agentic']→ 专用sweep-agentic任务）：validation.py：新增ChangelogMatrixEntry.agentic_evals；agentic 模型接受可选的run-eval/eval-only（exclude_none保证基准测试行输出逐字节不变，不影响产物复用）。evals仍仅接受固定序列长度条目，agentic 行进入固定序列长度调度路径从结构上不再可能。process_changelog.py：按scenario-type将单节点评估行拆分到evals/agentic_evals。run-sweep.yml：新增sweep-agentic-evals任务，输入块复制自sweep-agentic（转发scenario-type、kv-offloading、kv-offload-backend(+metadata)、total-cpu-dram-gb、duration，isl/osl/max-model-len置为'0'），并附加run-eval: true/eval-only: true；同时接入collect-evals。recover_failed_ingest.py：eval_jobs统计计入agentic_evals（使用.get，兼容旧的复用产物）。stdout="[]"模拟子进程，真实行形状从未经过 schema 校验，CI 成了第一道防线。本 PR 新增端到端拆分测试（真实的固定 + agentic 评估行经过真实的model_validate）及ChangelogMatrixEntry分桶测试（含evals拒绝 agentic 行的守卫测试）。python -m pytest utils/ -q：543 通过；utils/evals/test_run_eval_dispatch.py的 2 个失败在main上即存在（依赖本地 mini-swe-agent 环境），与本 PR 无关。非目标：
trigger-ingest/trigger-agentic-ingest的触发条件未改动；仅含 agentic 评估的 main 推送应走哪条 ingest 路径留作后续跟进。合并后，
agentx/dsv4-v300-vllm-mtp（#2258）应 rebase 到 main 并移除其validation.py的 union 提交。Validation run / 验证运行
This PR carries an eval-only, agentic-scoped changelog entry for the existing
dsv4-fp4-b300-vllm-agenticconfig, so the sweep itself proves the fix (the same eval-only dispatch that failed on #2258/#2259): run 29633097900 —check-changelogpasses (previously 8 schema errors), the fixed-seq-leneval /job is skipped, and the newagentic eval /job dispatches the DEP8 conc-224 SWE-bench eval with agentic inputs (previously died in 7s oncheck_env_vars: ISL/OSL/MAX_MODEL_LEN). After merge, the same entry re-runs through theon: pushmain sweep, confirming the push path too.中文：本 PR 附带一条针对现有
dsv4-fp4-b300-vllm-agentic配置的 evals-only、仅 agentic 场景的 changelog 条目，使扫描本身即可验证修复（与 #2258/#2259 失败的 eval-only 调度路径完全一致）：见 run 29633097900 ——check-changelog通过（此前报 8 个 schema 校验错误），固定序列长度的eval /任务被正确跳过，新的agentic eval /任务以 agentic 输入派发 DEP8 conc-224 的 SWE-bench 评估（此前在 7 秒内因check_env_vars: ISL/OSL/MAX_MODEL_LEN退出）。合并后，同一条目会经由 main 分支的on: push扫描再次运行，同时验证 push 路径。