Skip to content

Make grading-schema prose_fields extend the default set instead of replacing it #946

Description

@justinmclean

Summary

A per-step grading-schema.json replaces the runner's default prose-field set
rather than adding to it, so a prose field added later silently reverts to
exact-match grading. One schema is also now dead config.

Background

Prose fields are graded by a model; everything else is compared exactly. When a
step ships a grading-schema.json naming one prose field, the default set stops
applying to that step. Add a notes or summary field to that step's
expected.json later and it is exact-matched, failing on any wording
difference, with nothing to indicate why.

Where to look

  • evals/release-verify-rc/*/fixtures/grading-schema.json - both set
    prose_fields: ["paste_recipe"].
  • evals/release-audit-report/step-2-assemble-record/fixtures/grading-schema.json -
    sets prose_fields: ["record_markdown"], but that is no longer a compared
    expected.json key, so the schema acts on nothing.
  • tools/skill-evals/src/skill_evals/runner.py - where the default set and the
    per-step override are resolved.

Acceptance criteria

  • Document whether a per-step schema replaces or extends the default, in the
    skill-evals README.
  • The behaviour matches the documented decision, with a test.
  • The dead release-audit-report schema is removed or given a real field.
  • uv run --directory tools/skill-evals --group dev pytest passes.

Estimated effort

~2 hours for someone new to the codebase.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions