Skip to content

feat: adapter — run Online-Mind2Web (live-web evolution of Mind2Web) under the ClawBench harness #189

Description

@reacher-z

Goal

Let the ClawBench harness load + execute tasks from OSU-NLP-Group/Online-Mind2Web — paper titled "An Illusion of Progress? Assessing the Current State of Web Agents." It explicitly audits other live-web benchmarks. Methodologically the most direct conversation partner for our two-stage scoring.

Scope

  • Task loaderclawbench.corpus.adapters.online_mind2web ingests their task definitions (URL + natural-language instruction + step-level evaluation); maps onto our task schema.
  • Step-level scoring passthrough — their original Mind2Web ranks DOM elements per step; Online-Mind2Web extends with live-web execution. Surface their step-level evaluation in run-meta.json alongside our Stage 1 / Stage 2.
  • Audit signal — they're explicit about which methodology choices inflate scores. Compare side-by-side with our numbers on the same agent runs; if our reward rate skews higher or lower than their step-level eval, that's a methodology-validation signal worth surfacing.
  • CLIclawbench run --corpus online-mind2web --model <m> end-to-end.

Why now

The Online-Mind2Web paper explicitly critiques other benchmarks. The cleanest response is "run their audit methodology on ClawBench" — and the cleanest path to that is hosting their corpus inside our harness. Strong methodology positioning for V2.

Acceptance

  • clawbench run --corpus online-mind2web --limit 5 --model <m> produces 5-layer bundles
  • run-meta.json includes online_mind2web_step_score alongside our metrics
  • Scoring matches upstream eval/ within ±3pp on a sampled subset
  • Docs: eval/adapters/online-mind2web.md + a short methodology-comparison appendix

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions