Summary
A per-step grading-schema.json replaces the runner's default prose-field set
rather than adding to it, so a prose field added later silently reverts to
exact-match grading. One schema is also now dead config.
Background
Prose fields are graded by a model; everything else is compared exactly. When a
step ships a grading-schema.json naming one prose field, the default set stops
applying to that step. Add a notes or summary field to that step's
expected.json later and it is exact-matched, failing on any wording
difference, with nothing to indicate why.
Where to look
evals/release-verify-rc/*/fixtures/grading-schema.json - both set
prose_fields: ["paste_recipe"].
evals/release-audit-report/step-2-assemble-record/fixtures/grading-schema.json -
sets prose_fields: ["record_markdown"], but that is no longer a compared
expected.json key, so the schema acts on nothing.
tools/skill-evals/src/skill_evals/runner.py - where the default set and the
per-step override are resolved.
Acceptance criteria
Estimated effort
~2 hours for someone new to the codebase.
Summary
A per-step
grading-schema.jsonreplaces the runner's default prose-field setrather than adding to it, so a prose field added later silently reverts to
exact-match grading. One schema is also now dead config.
Background
Prose fields are graded by a model; everything else is compared exactly. When a
step ships a
grading-schema.jsonnaming one prose field, the default set stopsapplying to that step. Add a
notesorsummaryfield to that step'sexpected.jsonlater and it is exact-matched, failing on any wordingdifference, with nothing to indicate why.
Where to look
evals/release-verify-rc/*/fixtures/grading-schema.json- both setprose_fields: ["paste_recipe"].evals/release-audit-report/step-2-assemble-record/fixtures/grading-schema.json-sets
prose_fields: ["record_markdown"], but that is no longer a comparedexpected.jsonkey, so the schema acts on nothing.tools/skill-evals/src/skill_evals/runner.py- where the default set and theper-step override are resolved.
Acceptance criteria
skill-evals README.
release-audit-reportschema is removed or given a real field.uv run --directory tools/skill-evals --group dev pytestpasses.Estimated effort
~2 hours for someone new to the codebase.