Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .agents/skills/ce-skill-work/references/evaluate.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ This repository does not ship an eval skill. Anthropic's `skill-creator` and the

Baseline, then compare: run the scenario against the pre-change skill first when a behavior is being *changed* (so you can see the failure the change fixes), then against the edited skill. Include a no-guidance control when the question is whether a line does anything at all. Read every result; do not score by keyword.

**Size the eval to the decision, not the workflow.** A routing change inside an already-active skill is one decision made early in a run — which output contract, which reviewer team, which branch — so the cell that proves it stops at that decision: prompt the skill to resolve its gate and end (announce the team, state the contract, name the route), and grade the declared result. A full-skill run per cell (a complete review, a plan written to disk) spends ten minutes of subagents to observe a line that was decided in the first thirty seconds, and that cost is what turns a proportionate eval into a matrix. The floor: one discriminating fixture per direction — the case the old prose got wrong and the new must get right, and the case both must still get right — plus one per additional leg of the condition; post arm on Claude and Codex; the pre arm only when no baseline evidence already exists (a run that already happened in the session is evidence, not something to reproduce). A cell that declares nothing fails — grade the declared line, never narration that mentions the right word. Six to twelve cells is the ordinary size; the pack runs its rows serially, so run one pack invocation per row side by side when wall time matters. If the change seems to need more cells, suspect the condition before growing the eval. This shortcut cannot test activation — the cell tells the agent to read the skill, so a skill that would never auto-trigger still resolves its gate; a description or trigger change is evaluated in a fresh host session with no skill injected, per the activation item below.

Cover, proportionally to risk: the path the change touches on the weakest realistic model tier; strong-model regression (did prose make a capable model worse); restraint (does the skill stop where it should, and does it avoid case catalogs, repeated autonomy gates, repeated command skeletons, and blanket brevity slogans); activation (positive, adjacent-negative, explicit invoke) when the description or trigger changed — with substantive prompts, since a trivial one-step ask never triggers a skill regardless of description; and the next consumer's contract when an envelope or handoff changed. Run on Claude and Codex by default — cross-host divergence is the biggest portability risk and the one a single-host run cannot see.

**Classify the key behavior before sizing the eval, because a fake-boundary run does not validate delegation.** A run with dispatch, `git`, and `gh` forbidden grades recognition, routing, and pointer-following, and it is the right first pass for any skill (`bun run test:skill-eval-cell -- --read-only`). It cannot grade a skill whose key behavior is live: peer or subagent dispatch, an orchestrator return contract, a mutation gated on what a delegate returned. There the model's account of what it would have done is the artifact under test, so it cannot also be the evidence — whether the delegate was dispatched, what payload it got, whether attribution waited on a real receipt, and whether the reconciliation was a synthesis are all invisible. For those skills the eval dispatches for real, pre- and post-change, on Claude and Codex, graded on subprocess logs and on-disk artifacts rather than the transcript, against a throwaway subject repo rather than the checkout. Record the paths one run could not reach — reconciliation when peers concur, degradation when a peer is unavailable — as unexercised, not as passing. A `codex exec` launched from a Claude Code shell inherits `CLAUDECODE=1` and makes the peer attest the wrong host; launch it under `env -u CLAUDECODE`.
Expand Down