test(fabrika): drive the prototyping eval set's FORFEITED terminal with a sixth fixture

[usirin]

The prototyping skill lists nine ways a run can end, and one of them — FORFEITED, abandoning a
spike that never produced an answer — had no test behind it. The word showed up in a list and in a
"don't do this" assertion, and nowhere else. This adds a sixth eval fixture that actually ends there,
and rewrites the set's notes to say plainly what the recorded benchmark still does not measure.

Part of #5265

What's here

• claude-plugins/fabrika/skills/prototyping/evals/fixtures/eval-6.md — a spike on the fictional
  orrery/beacon whose three runs all died in setup (the sandbox will not open more than 64 of the
  500 sockets), so no run ever reached the phase the question was about. The session abandoned it
  with spike dispose --forfeit and is asked to finish the run.
• claude-plugins/fabrika/skills/prototyping/evals/evals.json — case 6 with six assertions, and a
  rewritten notes.

Why the fixture contains the forfeit instead of stopping before it

SKILL.md's A-TERMINAL-IS-AN-EXIT-YOU-READ rule forbids naming a terminal from an exit code the run
reasoned its way to rather than read, and FORFEITED is defined as 0 from spike dispose --forfeit. A fixture that stopped one command before the forfeit would therefore make the correct
answer "the terminal is pending" — the case could assert the word FORFEITED while no run could
honestly reach the state. So the fixture's transcript carries the --forfeit invocation as EXECUTED,
returning 0 with "workspace":"removed","forfeited":true, exactly the way eval-5 carries its
observed 17. What the case grades is the behaviour after that observed result: naming FORFEITED
rather than its near-miss DISPOSED (both are 0 from dispose; only one had a captured decision),
not dressing three aborted runs up as an answer, not trying to revive a workspace that is proven
gone, and routing the still-open question forward as fresh work.

The case derives the graded tier — all six assertions are judgements about what the run
concluded, none matches the mechanical cue lexicon. Verified through the real decoder, not asserted:
fabrika eval cases reports 6 case(s): 0 deterministic, 6 graded.

Deviations

• Scope narrowing — Said: prototyping skill evals never exercise the FORFEITED terminal outcome #5265's AC3 asks the set to be re-graded on both arms against the
  post-fix skill, and AC4 asks the PRE-FIX skill caveat to be removed or replaced with the new
  run's provenance. Did: shipped the fixture and the notes rewrite; did not re-grade. The
  notes keep the pre-fix caveat and now state exactly what it covers — five cases, pre-fix skill,
  eval-6 ungraded. Why: fabrika eval run takes --stage from the live STAGES vocabulary
  (triage, build, review, ship-it — packages/fabrika-cli/src/eval/corpus.ts) and
  prototyping is not a member, so the harness refuses this set at the flag. Admitting a stage has
  its own rules (the stage-admission rule in packages/fabrika-cli/src/eval/README.md) and belongs
  in its own change; writing numbers nobody ran would be worse than the gap. Disposition:
  follow-up re-grade the prototyping eval set (both arms, six cases) against the post-fix skill #5305 filed and named in the notes, and this PR is Part of #5265 per §9, so the issue
  stays open.

• Scope narrowing — Said: prototyping skill evals never exercise the FORFEITED terminal outcome #5265's AC1 names assertion 1c as the instrument that grades the
  §TERM closed set verbatim. Did: matched 1e (and its siblings 2f/5f) instead. Why:
  1c is about routing question 3 away from prototyping; 1e is the assertion that enumerates all
  nine terminals verbatim, so it is the instrument the AC means. Disposition: no action needed.

• Known defect left unfixed — Said: nothing in the issue asks for this. Did: left
  assertion 2c's wording alone, although its "by exit code 14" phrasing hits the cue lexicon and
  mis-derives that assertion as mechanical rather than the judgement it is. Why: prototyping skill evals never exercise the FORFEITED terminal outcome #5265 puts the
  weak-instrument defect on 1e/2f explicitly out of scope, and editing a graded set's existing
  assertions is the same class of change. Disposition: for the reviewer to judge; not filed
  separately, because it sits next to the already-tracked open question 2 on Authoring brief: the prototyping skill — throwaway code answering ONE question, never grown into the product #5020.

[github-actions]

No preview deploy

• No preview deploy for this PR — its diff touches no deploy-relevant path, so no preview stack was minted and e2e is not applicable. _(7b633fe)

• web — Stage pr-5306 torn down.

[usirin]

review-skill: PASS @ 7b633fe — merge-ready

Reviewed-head: @ 7b633fe

Verified PR #5306 against the acceptance criteria of #5265 + the skill-rigor checklist. Non-blocking (§CP: claude-plugins/fabrika/** matches no live CONTROL_PLANE_RE branch and no CODEOWNERS row — live CODEOWNERS covers kampus-pipeline only), so this PASS binds ship-it and the PR auto-ships with no human approval.

The central check — does the case genuinely reach FORFEITED, or merely assert the word?

It genuinely reaches it. Verified at source, not accepted from the PR body:

1. SKILL.md §TERM on main defines FORFEITED as 0 from spike dispose --forfeit — spike closed carrying a forfeit note and no decision, workspace proven gone.
2. The fixture transcript carries fabrika spike dispose --nonce 3d5b71ae --forfeit marked EXECUTED, answering {"spike":94518,...,"workspace":"removed","treeMatched":true,"runs":3,"forfeited":true}, with echo $? shown explicitly as 0. The run therefore reads the code rather than predicting it.
3. That satisfies the A-TERMINAL-IS-AN-EXIT-YOU-READ anchor, which I read on main: "Name a terminal from an exit code you actually read, never from one you reasoned your way to... If you could not run the verb, say which state you are actually in and that the terminal is pending." The stated counterfactual holds — a fixture stopping one command before the forfeit would make "pending" the correct answer, so the case could assert the word while no run honestly reached the state.
4. Structural identity to eval-5 confirmed line by line. eval-5 likewise ends its transcript on the dispose invocation EXECUTED with echo $? = 17, and 5f grades DISPOSAL-REFUSED off that observed code. Same shape, opposite code. The PR body claim is true.
5. The JSON is the contract shape, not invented. contract.md:780 seats the dispose answer as {"spike":…,"nonce":…,"workspace":"removed","treeMatched":true,"runs":…,"forfeited":false} — the fixture matches field-for-field and in key order, with forfeited:true on the forfeit path. Its spike status line matches contract.md:915 the same way.

Real discrimination, not a giveaway. 6e requires FORFEITED and explicitly fails DISPOSED, keyed on the correct §TERM discriminator (a captured decision); 6a independently requires that no decision was composed. The two lock together — the run must both not capture and name the terminal meaning "closed with no decision" — and the transcript grounds it observably via the status line's "captured":false. The --forfeit flag alone does not settle it: both DISPOSED and FORFEITED are 0 from dispose. The fixture also applies genuine counter-pressure ("nobody wants the three runs to have cost nothing", the standup deadline), which is what 6a/6b catch — the same pressure design as eval-4.

Non-blocking observation, for #5305 to measure rather than a defect here: the transcript prints "forfeited":true, so the token in 6e is echo-adjacent — the same class as the already-annotated 5a leak. It does not carry 6e's weight (which rests on choosing FORFEITED over DISPOSED, something the JSON does not disambiguate), and 6b/6c/6d/6f are substance regardless. Since eval-6 is disclosed UNGRADED, no strength claim is being made about it, so this is honest either way.

Acceptance criteria

• [PASS] AC1 — fixture drives a run to FORFEITED, graded verbatim against §TERM's closed set. eval-6.md + case 6's six assertions; 6e requires the verbatim member and fails both DISPOSED and the back-off terminals. See the central check above. The AC's own instrument pointer was mis-cited and the author corrected it in the open — correctly: I read both assertions at head; 1c is about routing question 3 away from prototyping, and 1e is the assertion enumerating all nine terminals verbatim. 1e is the instrument the AC means, so building 6e on it is the faithful reading, not a substitution. Disclosed as a Deviations entry rather than done silently, which is the difference between a correction and a defect.
• [PASS] AC2 — follows the set's ground rules. Fictional orrery/beacon; spike 94518 sits ~89k above the host repo's max issue number, so no fixture claim about the host is falsifiable; ground rules forbid invoking the not-yet-existing spike verbs and license reasoning forward from contract.md; the transcript stops one command before the behaviour under test — the behaviour under test is the write-up, and the dispose is its setup, exactly as in eval-5.
• [DEFERRED — disclosed partial-split, see the ruling below] AC3 — re-grade both arms against the post-fix skill. Not delivered.
• [PARTIAL — first half PASS, second half deferred] AC4 — notes updated. The COVERAGE GAP sentence is retired and restated ("FORFEITED, once a stated coverage gap, now has eval-6") — that half is satisfied. The PRE-FIX caveat is kept rather than removed/replaced, but is now scoped precisely (five cases, pre-fix skill, eval-6 ungraded) — deferred with AC3, since replacing it requires the run that cannot be made.
• [PASS] AC5 — lands where prototyping lives at pick time, and states which was true. Verified independently: PR feat(fabrika): author the prototyping skill and derive its CLI contract (#5020) #5261 merged at 2026-08-10T04:46:28Z, so the issue body's "not on main" precondition was stale and main was the correct base. The author states this explicitly in the format-3 progress comment on prototyping skill evals never exercise the FORFEITED terminal outcome #5265 ("PR feat(fabrika): author the prototyping skill and derive its CLI contract (#5020) #5261 merged at 2026-08-10T04:46Z, so prototyping is on main..."). Both halves met.
• [PASS] AC6 — out-of-scope items not silently absorbed. The 1e/2f weak-instrument defect and The eval corpus STAGES vocabulary has no ideation stage, so no quintet skill can declare an eval entry #5241 are untouched; the author cites AC6 as the reason for leaving 2c alone rather than quietly widening scope.

Ruling — partial delivery with the gap disclosed is CORRECT here; this should not have waited

The forcing constraint is real, verified at source. packages/fabrika-cli/src/eval/corpus.ts:32 declares STAGES = ["triage", "build", "review", "ship-it"] and prototyping is not a member, so fabrika eval run --stage refuses this set at the flag. Admitting a stage has its own rules — the stage-admission rule at packages/fabrika-cli/src/eval/README.md:71 ("a stage exists when its skill does", with a manifest key and a grader arm, enforced by stage-admission.data.unit.test.ts) — and that is plainly a different change from an eval-fixture chore.

So the two alternatives to shipping partial were both worse: fabricate numbers nobody ran — a false claim, and precisely the failure mode this ticket exists to close — or hold the fixture hostage to an unrelated stage-admission change. The author took neither.

And this shape is contract-sanctioned, not a tolerated lapse. §DEV class 1 says it outright: "A Part of #N partial-split per §9 is a disclosed narrowing: name it here too, don't let the token stand in for the reasoning." The author did all four things that requires — Part of #5265 so GitHub auto-closes nothing (verified: closing-keyword set is {}, #5265 is still OPEN), a named ## Deviations entry citing AC3 and AC4 specifically, the forcing constraint stated and checkable, and a real follow-up (#5305, OPEN) carrying the re-grade. The notes field states what remains unmeasured rather than deleting the caveat, so nothing downstream can read the five-case pre-fix benchmark as a measurement of this six-case set.

What this PASS does and does not certify. It certifies the scope this PR actually claims. It does not certify #5265 complete: AC3 and AC4's second half remain open work on #5265 and #5305, and #5265 must stay open after merge. That is the designed behaviour of Part of #N, not a gap I am waving through.

Ruling — the derived tier, re-derived independently

I did not take the author's number. I read the authoritative lexicon (CUE_PHRASES, 24 phrases across four cues, skill-eval-set.ts:91-120) and deriveTier (:195, deterministic only when a case has assertions and every one is mechanical), then ran deriveAssertion/deriveTier directly over the head's evals.json:

case 1 -> graded  (6 judgment, 0 mechanical)
case 2 -> graded  (5 judgment, 1 mechanical)
case 3 -> graded  (6 judgment, 0 mechanical)
case 4 -> graded  (6 judgment, 0 mechanical)
case 5 -> graded  (6 judgment, 0 mechanical)
case 6 -> graded  (6 judgment, 0 mechanical)   6a..6f all judgment
6 case(s): 0 deterministic, 6 graded

Matches the claim exactly, and the stated grounds hold: all six of case 6's assertions are judgements and none hits the cue lexicon. (6f is a near-miss worth noting — it says "returned 0", which narrowly avoids the exits 0 cue; the tier would be unchanged either way, since one mechanical assertion cannot make a case deterministic.)

Ruling — the deliberately-unfixed 2c mis-derivation: correctly left

The disclosure is true: 2c's "by exit code 14" hits the exit-status cue and derives mechanical rather than the judgement it plainly is — it is the single mechanical assertion in the whole set, which my run above isolates to case 2. But it is inert. deriveTier needs every assertion mechanical, and case 2 carries five judgements beside it, so case 2 still derives graded. The mis-derivation changes no tier, no routing decision, and no number. Fixing it would mean editing an existing assertion in an already-graded set — the same class of change #5265's AC6 puts out of scope, and the set's own notes warn that repairing a graded set retroactively falsifies the runs quoting it. Left correctly, and disclosed rather than hidden.

Skill rigor

• [PASS] Behavioral correctness — the fixture is a coherent, followable session brief; its ground rules match the other five; and every machine answer it shows is the contract's real shape (contract.md:780 dispose, :915 status), so it teaches the true output rather than an invented one.
• [PASS] Trigger / description quality — no frontmatter, SKILL.md, or trigger surface is touched; the diff is a fixture plus one case entry. Nothing to over- or under-trigger.
• [PASS] Cross-skill conflict / shadowing — no marker namespace, §CP set, routing rule, or shared contract is touched. evals.json is read by the fabrika harness alone.
• [PASS] Gate-invariant preservation — no gate invariant is in the diff's reach (nothing in §CP; no pipeline gate file). The one adjacent risk — that rewriting notes might launder a stale benchmark — resolves the safe way: the rewrite keeps the pre-fix caveat and adds the ungraded/unmeasured disclosure, so the set claims strictly less than before, not more.
• [PASS] Contract implementation ticket — out of scope; the diff's two files carry no fabrika contract.md.
• [PASS] deviation-disclosure (§DEV) — the literal ## Deviations heading is present with three entries, and each is true at source: (1) the STAGES vocabulary and the README stage-admission rule, both verified, with re-grade the prototyping eval set (both arms, six cases) against the post-fix skill #5305 filed and OPEN; (2) 1c vs 1e, verified by reading both assertions; (3) the 2c mechanical mis-derivation, independently reproduced. Nothing undisclosed that this gate could see.

Non-contradiction check (context only, not gated)

• ADR 0257 (unlanded, PR docs(decisions): 0257 — a platform incident is eval material only through an artifact, at the deterministic tier (#4824) #5282) rules platform incidents enter the eval bar only as artifact checks at the deterministic tier. eval-6 is a skill-behaviour case at the graded tier — not a platform incident and not an artifact check — so it falls outside 0257's scope rather than against it. No contradiction.
• PR docs(fabrika): record the trigger-coverage convention + the user-only exemption on the eval surface (#5239) #5303 (unlanded) records trigger coverage. eval-6 measures post-invocation behaviour, a different axis — the same distinction prototyping skill evals never exercise the FORFEITED terminal outcome #5265's triage already drew against Decide: retro-measure trigger coverage, or enumerate an explicit grandfather, for the 13 model-invoked fabrika skills already on main #5245. No contradiction.

Neither treated as settled law; both merely checked for collision.

Read the PR head (§HEAD): all text under review sourced from 7b633fe51a833031abb4b7a2ac04052b49361971 via a per-run ref + git show, never the launched checkout's working copy. main was fetched fresh for every shipped-state check (§TERM, contract.md, CODEOWNERS, STAGES, the cue lexicon).

All checks pass. This PR is merge-ready for the scope it claims. review-skill does not merge — ship-it is the authorized merge step. Merging closes nothing: the body carries Part of #5265, so #5265 stays open with AC3 and AC4's second half outstanding, tracked by #5305.

Verdict-written: 2026-08-10T07:20:58Z