From aa117ed2b460ca83b632829eb5ab03dfa7b71cbf Mon Sep 17 00:00:00 2001 From: Trevin Chow Date: Sun, 23 Aug 2026 00:09:51 -0700 Subject: [PATCH 1/2] docs(ce-skill-work): size a behavioral eval to the decision, not the workflow A routing or activation change is one early decision; the cell that proves it stops at that decision and grades the declared result. States the floor (one discriminating fixture per direction, post arm on Claude and Codex, pre arm only without baseline evidence, six to twelve cells) so an agent does not default to full-skill runs per cell. --- .agents/skills/ce-skill-work/references/evaluate.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.agents/skills/ce-skill-work/references/evaluate.md b/.agents/skills/ce-skill-work/references/evaluate.md index 8eebeb83f..3744fe52a 100644 --- a/.agents/skills/ce-skill-work/references/evaluate.md +++ b/.agents/skills/ce-skill-work/references/evaluate.md @@ -16,6 +16,8 @@ This repository does not ship an eval skill. Anthropic's `skill-creator` and the Baseline, then compare: run the scenario against the pre-change skill first when a behavior is being *changed* (so you can see the failure the change fixes), then against the edited skill. Include a no-guidance control when the question is whether a line does anything at all. Read every result; do not score by keyword. +**Size the eval to the decision, not the workflow.** A routing or activation change is one decision made early in a run — which output contract, which reviewer team, which branch — so the cell that proves it stops at that decision: prompt the skill to resolve its gate and end (announce the team, state the contract, name the route), and grade the declared result. A full-skill run per cell (a complete review, a plan written to disk) spends ten minutes of subagents to observe a line that was decided in the first thirty seconds, and that cost is what turns a proportionate eval into a matrix. The floor: one discriminating fixture per direction — the case the old prose got wrong and the new must get right, and the case both must still get right — plus one per additional leg of the condition; post arm on Claude and Codex; the pre arm only when no baseline evidence already exists (a run that already happened in the session is evidence, not something to reproduce). A cell that declares nothing fails — grade the declared line, never narration that mentions the right word. Six to twelve cells, run concurrently, is the ordinary size; if the change seems to need more, suspect the condition before growing the eval. + Cover, proportionally to risk: the path the change touches on the weakest realistic model tier; strong-model regression (did prose make a capable model worse); restraint (does the skill stop where it should, and does it avoid case catalogs, repeated autonomy gates, repeated command skeletons, and blanket brevity slogans); activation (positive, adjacent-negative, explicit invoke) when the description or trigger changed — with substantive prompts, since a trivial one-step ask never triggers a skill regardless of description; and the next consumer's contract when an envelope or handoff changed. Run on Claude and Codex by default — cross-host divergence is the biggest portability risk and the one a single-host run cannot see. **Classify the key behavior before sizing the eval, because a fake-boundary run does not validate delegation.** A run with dispatch, `git`, and `gh` forbidden grades recognition, routing, and pointer-following, and it is the right first pass for any skill (`bun run test:skill-eval-cell -- --read-only`). It cannot grade a skill whose key behavior is live: peer or subagent dispatch, an orchestrator return contract, a mutation gated on what a delegate returned. There the model's account of what it would have done is the artifact under test, so it cannot also be the evidence — whether the delegate was dispatched, what payload it got, whether attribution waited on a real receipt, and whether the reconciliation was a synthesis are all invisible. For those skills the eval dispatches for real, pre- and post-change, on Claude and Codex, graded on subprocess logs and on-disk artifacts rather than the transcript, against a throwaway subject repo rather than the checkout. Record the paths one run could not reach — reconciliation when peers concur, degradation when a peer is unavailable — as unexercised, not as passing. A `codex exec` launched from a Claude Code shell inherits `CLAUDECODE=1` and makes the peer attest the wrong host; launch it under `env -u CLAUDECODE`. From 339915dcc2aa098112d2d6270872cf332ca07e62 Mon Sep 17 00:00:00 2001 From: Trevin Chow Date: Sun, 23 Aug 2026 00:14:32 -0700 Subject: [PATCH 2/2] docs(ce-skill-work): the decision probe covers routing, not activation; packs run rows serially --- .agents/skills/ce-skill-work/references/evaluate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/skills/ce-skill-work/references/evaluate.md b/.agents/skills/ce-skill-work/references/evaluate.md index 3744fe52a..fa77f4323 100644 --- a/.agents/skills/ce-skill-work/references/evaluate.md +++ b/.agents/skills/ce-skill-work/references/evaluate.md @@ -16,7 +16,7 @@ This repository does not ship an eval skill. Anthropic's `skill-creator` and the Baseline, then compare: run the scenario against the pre-change skill first when a behavior is being *changed* (so you can see the failure the change fixes), then against the edited skill. Include a no-guidance control when the question is whether a line does anything at all. Read every result; do not score by keyword. -**Size the eval to the decision, not the workflow.** A routing or activation change is one decision made early in a run — which output contract, which reviewer team, which branch — so the cell that proves it stops at that decision: prompt the skill to resolve its gate and end (announce the team, state the contract, name the route), and grade the declared result. A full-skill run per cell (a complete review, a plan written to disk) spends ten minutes of subagents to observe a line that was decided in the first thirty seconds, and that cost is what turns a proportionate eval into a matrix. The floor: one discriminating fixture per direction — the case the old prose got wrong and the new must get right, and the case both must still get right — plus one per additional leg of the condition; post arm on Claude and Codex; the pre arm only when no baseline evidence already exists (a run that already happened in the session is evidence, not something to reproduce). A cell that declares nothing fails — grade the declared line, never narration that mentions the right word. Six to twelve cells, run concurrently, is the ordinary size; if the change seems to need more, suspect the condition before growing the eval. +**Size the eval to the decision, not the workflow.** A routing change inside an already-active skill is one decision made early in a run — which output contract, which reviewer team, which branch — so the cell that proves it stops at that decision: prompt the skill to resolve its gate and end (announce the team, state the contract, name the route), and grade the declared result. A full-skill run per cell (a complete review, a plan written to disk) spends ten minutes of subagents to observe a line that was decided in the first thirty seconds, and that cost is what turns a proportionate eval into a matrix. The floor: one discriminating fixture per direction — the case the old prose got wrong and the new must get right, and the case both must still get right — plus one per additional leg of the condition; post arm on Claude and Codex; the pre arm only when no baseline evidence already exists (a run that already happened in the session is evidence, not something to reproduce). A cell that declares nothing fails — grade the declared line, never narration that mentions the right word. Six to twelve cells is the ordinary size; the pack runs its rows serially, so run one pack invocation per row side by side when wall time matters. If the change seems to need more cells, suspect the condition before growing the eval. This shortcut cannot test activation — the cell tells the agent to read the skill, so a skill that would never auto-trigger still resolves its gate; a description or trigger change is evaluated in a fresh host session with no skill injected, per the activation item below. Cover, proportionally to risk: the path the change touches on the weakest realistic model tier; strong-model regression (did prose make a capable model worse); restraint (does the skill stop where it should, and does it avoid case catalogs, repeated autonomy gates, repeated command skeletons, and blanket brevity slogans); activation (positive, adjacent-negative, explicit invoke) when the description or trigger changed — with substantive prompts, since a trivial one-step ask never triggers a skill regardless of description; and the next consumer's contract when an envelope or handoff changed. Run on Claude and Codex by default — cross-host divergence is the biggest portability risk and the one a single-host run cannot see.