Goal
Let the ClawBench harness load + execute tasks from OSU-NLP-Group/Online-Mind2Web — paper titled "An Illusion of Progress? Assessing the Current State of Web Agents." It explicitly audits other live-web benchmarks. Methodologically the most direct conversation partner for our two-stage scoring.
Scope
- Task loader —
clawbench.corpus.adapters.online_mind2web ingests their task definitions (URL + natural-language instruction + step-level evaluation); maps onto our task schema.
- Step-level scoring passthrough — their original Mind2Web ranks DOM elements per step; Online-Mind2Web extends with live-web execution. Surface their step-level evaluation in
run-meta.json alongside our Stage 1 / Stage 2.
- Audit signal — they're explicit about which methodology choices inflate scores. Compare side-by-side with our numbers on the same agent runs; if our reward rate skews higher or lower than their step-level eval, that's a methodology-validation signal worth surfacing.
- CLI —
clawbench run --corpus online-mind2web --model <m> end-to-end.
Why now
The Online-Mind2Web paper explicitly critiques other benchmarks. The cleanest response is "run their audit methodology on ClawBench" — and the cleanest path to that is hosting their corpus inside our harness. Strong methodology positioning for V2.
Acceptance
Goal
Let the ClawBench harness load + execute tasks from OSU-NLP-Group/Online-Mind2Web — paper titled "An Illusion of Progress? Assessing the Current State of Web Agents." It explicitly audits other live-web benchmarks. Methodologically the most direct conversation partner for our two-stage scoring.
Scope
clawbench.corpus.adapters.online_mind2webingests their task definitions (URL + natural-language instruction + step-level evaluation); maps onto our task schema.run-meta.jsonalongside our Stage 1 / Stage 2.clawbench run --corpus online-mind2web --model <m>end-to-end.Why now
The Online-Mind2Web paper explicitly critiques other benchmarks. The cleanest response is "run their audit methodology on ClawBench" — and the cleanest path to that is hosting their corpus inside our harness. Strong methodology positioning for V2.
Acceptance
clawbench run --corpus online-mind2web --limit 5 --model <m>produces 5-layer bundlesrun-meta.jsonincludesonline_mind2web_step_scorealongside our metricseval/within ±3pp on a sampled subseteval/adapters/online-mind2web.md+ a short methodology-comparison appendix