Skip to content

judge_llm.py has diverged from judge.py (no Gemini support, wrong responses endpoint, 800-token cap) — collapse the duplication #300

Description

@reacher-z

src/clawbench/runner/judge_llm.py:20-22 says it is a "drop-in replacement" for judge.py with the same interface. It has diverged in ways that matter, and it is the module behind the lenient rubric that produces the published Reward-lenient column:

judge.py judge_llm.py
judge_context kwarg (passed at run.py:587-597) yes missing (:169-171)
openai-responses api_type correct endpoint routed to /chat/completions (:180-181)
google-generative-ai (the /v1beta/openai fix) supported (:99-108, :217-220) absent
max_tokens 4096, with a comment that reasoning judges burn hidden tokens (:123) 800 (:114)

The default judge is deepseek-v4-pro, a reasoning model — an 800-token budget risks truncating its verdict, and truncation lands on the except path whose lenient default is match=True (see #295). On top of that, _post_json / _build_user_msg / _call_* are ~100 duplicated lines that now have to be fixed twice; the Gemini fix already only landed in one copy.

Ask: reduce judge_llm.py to its JUDGE_SYSTEM prompt and delegate to judge.py's _run_judge / _build_user_msg with the rubric as a parameter. That restores identical api_type coverage, token budget, and signature in one place.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions