docs(decisions): 0249 — trigger coverage lives in the eval set, not a fourth ship-gate part (#4750)

[usirin]

Fixes #4750

Records the ruling on #4750 as ADR 0249. One added file, no other changes:

• .decisions/0249-skill-trigger-coverage-lives-in-the-eval-set.md

The ruling this records

Per the newest comment on the issue (2026-08-10, founder-delegated judgment under the standing trust ruling of 2026-08-09, founder live with veto open):

• The fabrika §8 ship gate keeps its three parts. No fourth part. claude-plugins/fabrika/docs/skill-conventions.md is not edited by this PR, so §8's own prohibition on restating eval mechanics stays intact.
• Trigger coverage is a property of a skill's eval set, under the mechanics epic fabrika eval layer: the execution harness, the incident corpus, and the ruled bar #4649 owns. Since §8 part 3 already requires a green eval set, this adds a property to the eval set rather than a gate to the doc — one home for the number.
• User-only skills are exempt, stated: with disable-model-invocation there is no description in model context, so there is nothing to trigger. Precedent set live by the front-door session (Authoring brief: front-door — fabrika skill (the /fabrika operating front door) #4952), whose gate accepted the deviation.
• The convention gets written against the eval surface, not into §8 — a downstream build ticket, filed and linked in the progress comment.

What is superseded, and why the ADR says so out loud

An earlier ruling on the same issue (2026-08-02) split the gate and sent a routing-path check into skill-conventions.md beside the sizing band. The newest comment reverses that: nothing lands in §8. The ADR records the reversal explicitly so a reader who scrolls the issue top-down does not act on the 2026-08-02 answer.

The evidence behind that earlier split is carried forward, not dropped — the five optimizer iterations (recall 0–11%, precision 100% throughout, no should-fire query ever reaching 2/3) and the baseline arm that behaved correctly with no skill at all by reading CLAUDE.md. That is what stops someone re-litigating a naive recall floor in a month, so the ADR keeps it as the three constraints #4649's mechanics inherit.

Acceptance criteria

• The ruling is recorded: §8 gains no triggering gate.
• The single home for the number is named — the eval set / fabrika eval layer: the execution harness, the incident corpus, and the ruled bar #4649's mechanics — and the user-only exemption is stated. No floors are invented here; the ruling named none, and the ADR does not fabricate them.
• The two skills already on main are addressed: both are model-invoked, so the exemption does not reach them, and the ruling states no grandfather clause. The ADR records that their trigger coverage is owed once the mechanics exist and that retro-measure-vs-grandfather is open, carried on the build ticket rather than decided at this desk. This departs from the criterion as written — disclosed as class 1 under ## Deviations below.
• Written where a stateless authoring session reads it — an ADR in .decisions/.
• No implementation lands here. Build ticket filed against the eval surface (linked in the progress comment on the issue).

Deviations

Section added in repair round 1, answering the deviation-disclosure FAIL (review comment 5234901884). The class-1 entry below belongs to the initial build, not to this round; this round touched no file — PR body only, ADR text and number unchanged.

• Class 1 — scope narrowing. Said: A fabrika skill can pass every ship gate and never fire — triggering is unmeasured (/report: 100% precision, ~0% recall) #4750's AC3 asks that the two skills already on main (/report and /adr) be either retro-measured before the next skill ships or explicitly grandfathered — one of two named outcomes. Did: the ADR settles only the half the governing ruling settles — both skills are model-invoked, so the disable-model-invocation exemption does not reach them; the ruling states no grandfather clause; their trigger coverage is therefore owed once fabrika eval layer: the execution harness, the incident corpus, and the ruled bar #4649's mechanics exist — and then records retro-measure-vs-grandfather as an open question rather than picking either named outcome. Why: the governing 2026-08-10 ruling does not decide the timing or the mechanism. Answering it at the coder's desk would manufacture founder doctrine inside an ADR, and a stateless later reader obeys an ADR as ruled — a worse artifact than a recorded open item. The residue is also genuinely downstream of eval mechanics that do not exist yet, so there is nothing to retro-measure against today. Disposition: follow-up Record the trigger-coverage convention + the user-only exemption in the eval mechanics docs (ruled on #4750) #5239 (open, status:needs-triage) carries the question against the eval surface; a one-line founder amendment on A fabrika skill can pass every ship gate and never fire — triggering is unmeasured (/report: 100% precision, ~0% recall) #4750 can settle it without editing this ADR.
• Class 2 — governing-ADR departure. None. Step 4a's contradiction sweep found no accepted ADR ruling on any of the four questions, and this ADR neither narrows nor widens one; it is the first on this subject.
• Class 3 — known defect left unfixed. None. No adjacent or sibling defect was seen and skipped.
• Class 4 — declined guidance. None. No reviewer suggestion, reviewer-appended acceptance criterion, or triage instruction was declined; the only review finding to date is the one this round addresses.
• Class 5 — guard or gate bypassed. None. No --no-verify push, no skipped or disabled hook, no suppressed lint or type error, no widened allowlist. pnpm lint:worktree was a clean skip because the diff changes no biome-handled file — a no-op by scope, not a bypass.
• Class 6 — pre-existing test or fixture changed. None. The diff is one added file, +122/−0; no existing assertion was modified, weakened, or deleted.
• Class 7 — out-of-scope change. None. Exactly one file changed, .decisions/0249-skill-trigger-coverage-lives-in-the-eval-set.md, which is precisely what A fabrika skill can pass every ship gate and never fire — triggering is unmeasured (/report: 100% precision, ~0% recall) #4750 asks for. Nothing under claude-plugins/**, no CODEOWNERS or control-plane-pattern change.

Notes for the gate

• Docs-only diff. pnpm lint:worktree is a clean skip (no biome-handled files changed).
• ADR number re-checked against origin/main (holds through 0247) and every open PR (0248 in docs(decisions): 0248 — the authoring session mints a contract's implementation ticket at handoff (#4749) #5235, 0237 in docs(decisions): 0237 — agent brevity binds messages, not board artifacts (#4694) #4703, 0235 in docs(decisions): ADR 0235 — a fence carries zero expansions; everything dynamic lives inside the script #4614) immediately before commit.
• Fences respected: no edit under claude-plugins/fabrika/skills/{wayfinding,prototyping,graduate,handoff}/, none under plan-epic/**, no CODEOWNERS or control-plane-pattern change.
• §CP by content (ADR 0164 / 0135). .decisions/** carries no CODEOWNERS row, but pipeline-cli guard-content-probe classify returns guard-touching [guard-vocabulary-match] on the ADR body, so this PR is in the blocking set by content: it needs a @kamp-us/control-plane approval at the then-current head before ship-it enqueues it.
• Closing keywords. Exactly one plain-prose closing keyword in this body: Fixes #4750 on line 1. No other fix/close/resolve variant precedes an issue reference — in particular no closing keyword is attached to Record the trigger-coverage convention + the user-only exemption in the eval mechanics docs (ruled on #4750) #5239 anywhere in this body, so that follow-up stays open to carry the class-1 residue above.

[github-actions]

No preview deploy

• No preview deploy for this PR — its diff touches no deploy-relevant path, so no preview stack was minted and e2e is not applicable. _(a1076e1)

• web — Stage pr-5238 torn down.

[usirin]

review-doc: advisory — blocking-set PR (§CP — approval-gated)

PR #5238 is §CP by CONTENT, not by path. .decisions/** carries no CODEOWNERS row (read live), so this is not path-§CP; the shared pipeline-cli guard-content-probe classify verb, re-run at this head against the ADR body sourced from the head ref, returns guard-touching (§CP, ADR 0164) [guard-vocabulary-match] on .decisions/0249-skill-trigger-coverage-lives-in-the-eval-set.md. My verdict is advisory only: it does not authorize a merge. Under the §CP hard gate (ADR 0135), a @kamp-us/control-plane member approves this at its current head and ship-it then enqueues it (ADR 0048 single merge authority) — there is no human hand-merge in the §CP path.

Reviewed-head: @ a1076e1

Re-gate round 2 — the head is UNCHANGED from the round-1 FAIL (comment 5234901884). The repair pushed no commit; it edited the PR body only. I re-derived every row from scratch rather than inheriting round 1's grades: the prior FAIL was neither treated as still standing nor as automatically discharged. Every file under review was sourced from the PR head through a per-run read-only ref (asserted equal to a1076e10…) — never a working-tree switch, never the launched checkout's working copy. Diff shape confirmed first-hand off the REST files endpoint: 1 file, added, +122/−0.

Class routing (re-derived, not inherited). pipeline-cli class-probe classify --namespaces over the live changed-file set: 1 changed file(s) → has-docs ⇒ the required namespace set is review-doc only (1 namespace). No has-code, no has-skills, no review-design — the probe named no other gate, so no sibling namespace is left empty at this head.

Graded against the 2026-08-10 ruling (comment 5234575962), read first-hand — not the superseded 2026-08-02 split.

Acceptance criteria (#4750)

• [PASS] AC1 — the ruling is recorded, yes-or-no. ADR title + ## Decision item 1 (L60–62): "§8 keeps its three parts, unchanged. No fourth ship-gate part," and claude-plugins/fabrika/docs/skill-conventions.md is untouched by the diff (verified: the diff is exactly one added file). §8's self-prohibition on restating eval mechanics stays intact.
• [PASS] AC2 — floors / co-gating / the single home. The "if yes" antecedent is false, so the box is satisfied by naming the single home: ## Decision item 2 (L63–66) puts trigger coverage in the skill's eval set under fabrika eval layer: the execution harness, the incident corpus, and the ruled bar #4649's mechanics, and ## Consequences (L104) keeps precision in the bar as one of three inherited constraints. No floors invented — correct, since the ruling named none; fabricating one would have been the defect.
• [PASS] AC3 — the two skills already on main, with the departure now disclosed (see the deviation row). The criterion offered a binary the governing ruling never produced ("retro-measured before the next skill ships, or explicitly grandfathered"). ADR L106–110 settles the load-bearing half definitively — /report and /adr are both model-invoked, so the user-only exemption does not reach them; the ruling states no grandfather clause (verified by reading comment 5234575962 first-hand: it is silent on the two shipped skills); their trigger coverage is therefore owed once fabrika eval layer: the execution harness, the incident corpus, and the ruled bar #4649's mechanics exist — and it records retro-measure-vs-grandfather as "not settled here," carried on Record the trigger-coverage convention + the user-only exemption in the eval mechanics docs (ruled on #4750) #5239 (re-verified live: open, status:needs-triage). Deferring remains right on the merits, and the round-1 defect — that the reasoning lived only in checklist prose and a progress comment — is now cured in the body, where a §CP approver and a cold repair round both see it.
• [PASS] AC4 — written where a stateless authoring session reads it. An ADR in .decisions/, plus follow-up build ticket Record the trigger-coverage convention + the user-only exemption in the eval mechanics docs (ruled on #4750) #5239.
• [PASS] AC5 — no implementation lands here. One added ADR file, +122/−0, zero deletions; the convention is routed to the eval surface via Record the trigger-coverage convention + the user-only exemption in the eval mechanics docs (ruled on #4750) #5239.

Doc hygiene

• [PASS] House-format — frontmatter carries id / title / status / date / tags; the body carries ## Context / ## Decision / ## Consequences plus the house **What this decides:** lead.
• [PASS] Index row + status match — no .decisions/index.md is committed (ADR 0126, discovery is ambient); frontmatter status: accepted is the source the ambient map renders.
• [PASS] Links resolve — exactly one relative link, ../claude-plugins/fabrika/docs/skill-conventions.md, confirmed to exist as an object at this head (read-only existence check against the head ref, not against a working copy). Everything else is an absolute GitHub issue/comment URL. Standard markdown, no wikilinks.
• [PASS] No leaked local/home paths — pipeline-cli leak-guard scan over the ADR sourced from head: scope 1 file / 1 doc surface; clean. An independent pattern scan for user-home, absolute-machine, file:// and worktree paths plus wikilink syntax, over both the ADR and the live PR body, returns nothing.
• [PASS] Supersession noted + cross-linked — the ADR supersedes an issue-comment ruling, not a prior ADR, so no .decisions/ status line is owed. ### What this supersedes (L76–87) names the 2026-08-02 split, links both comments, states the outcome in one unambiguous sentence ("That split is superseded: nothing lands in §8"), and adds the navigational instruction to read the issue newest-comment-first. It preserves the evidence the split rested on rather than discarding it along with the split — the right separation.
• [PASS] Status sanity — accepted, coherent with a settled ruling. date: 2026-08-09 matches the house convention on the adjacent ADR 0247 even though the ruling comment carries a 2026-08-10 UTC timestamp; the body cites the ruling date explicitly, so nothing is ambiguous.
• [PASS] Single Diátaxis mode — single-mode: explanation throughout. No how-to steps, no reference tables, no tutorial passage intruding.
• [PASS] Clear, concise prose — active voice, concrete, evidence-carrying. The bold-lead bullets in ## Consequences are established ADR house vocabulary, not slop; no AI-tell density.

ADR contradiction sweep (Step 4a) — re-run, not inherited

• [PASS] The four questions (does §8 gain a fourth part · where the trigger number lives · are user-only skills exempt · are the two shipped skills grandfathered). An independent case-insensitive search across all 246 .decisions/*.md on a freshly-read origin/main for trigger coverage / disable-model-invocation / description optimizer / trigger-eval / triggering returns 7 files, and every hit was opened and read: all are the unrelated sense of "trigger" (0073 non-overlapping skill triggering, 0075's triggering instance, 0132's merge_group workflow trigger, and four DO/workflow-trigger uses). No accepted ADR rules on any of the four questions, so 0249 is the first, and it neither narrows nor widens an existing ruling. The semantic pass was done by hand; no exit code was taken as a discharge.

ADR number collision (re-enumerated live at review time)

• [PASS] 0249 is free. origin/main holds through 0247 (0245, 0246, 0247 are the tail). All 9 other open PRs were enumerated (docs(fabrika): delete the dead line band from the sizing convention (#4701) #5240, fix(divan): render the incelemede badge through the design system #5237, fix(divan): size the upvote button through the button size contract #5236, feat(fabrika): author the grilling skill and derive its CLI contract (#5019) #5233, chore: release main #4833, docs(decisions): 0237 — agent brevity binds messages, not board artifacts (#4694) #4703, docs(decisions): ADR 0235 — a fence carries zero expansions; everything dynamic lives inside the script #4614, chore(deps): bump react-router from 7.18.1 to 8.3.0 #3941, plus this one); none carries .decisions/0249. docs(decisions): 0248 — the authoring session mints a contract's implementation ticket at handoff (#4749) #5235 is no longer open, so its 0248 cannot collide with 0249 in either direction. No duplicate at this head.

Closing keywords (re-run independently over the live body, each hit classified)

A case-insensitive (fix|close|resolve)[a-z]* +#[0-9]+ scan of the live PR body returns exactly 2 hits:

1. Line 1 — Fixes #4750 — plain prose, intended, and the only closing link. This PR records the ruling that closes A fabrika skill can pass every ship gate and never fire — triggering is unmeasured (/report: 100% precision, ~0% recall) #4750.
2. Line 48 — the same Fixes #4750, inside a code span, in the bullet that documents this very scan. It names the same target as hit 1, so even under the unsettled question of whether GitHub honors a backticked keyword (Seam checks pass a Part of #N PR that also arms a closing keyword for #N (silently closed #5183) #5234), the outcome is identical to hit 1 and no additional issue is implicated. No assumption is encoded here either way — this row reads the same whichever way Seam checks pass a Part of #N PR that also arms a closing keyword for #N (silently closed #5183) #5234 settles.

#5239 is NOT closed by this PR — confirmed. #5239 appears twice in the body (L34 in the class-1 deviation entry, L48 in the closing-keyword bullet) and neither occurrence is preceded by a fix/close/resolve variant. The repair's report that it caught and rewrote a drafted bare closing keyword on #5239 before it reached GitHub is verified against the live body: no such sentence survives. #5239 was re-read live at review time — open, status:needs-triage — so it remains the residue's home after this PR merges.

Deviation disclosure (§DEV) — the round-1 blocking row, re-judged on substance

• [PASS] deviation-disclosure — the FAIL is genuinely discharged, on substance and not merely because a heading now exists.
  • Section present. The canonical Tier-M scan (shared/scripts/dev-tier-m.sh) at this head: ## Deviations section present; 0 suppression/skip line(s), 0 removed-assertion line(s). The PR still owes the section (it carries Fixes #4750 with a graded AC list, so no ADR 0075/0184 issueless carve-out fires, and it is write-code-authored), so [N/A] remains unavailable — correctly.
  • The class-1 entry discloses the real departure, in §DEV's four-part shape. It names what the spec Said (A fabrika skill can pass every ship gate and never fire — triggering is unmeasured (/report: 100% precision, ~0% recall) #4750's AC3 and its two named outcomes), what the implementation Did (settles only the half the 2026-08-10 ruling settles; records retro-measure-vs-grandfather as open), Why (the governing ruling decides neither timing nor mechanism, so settling it at the authoring desk would manufacture founder doctrine inside an artifact a stateless reader obeys as ruled — and the residue is downstream of mechanics that do not yet exist), and its Disposition (follow-up Record the trigger-coverage convention + the user-only exemption in the eval mechanics docs (ruled on #4750) #5239, verified open; a one-line founder amendment on A fabrika skill can pass every ship gate and never fire — triggering is unmeasured (/report: 100% precision, ~0% recall) #4750 can settle it without editing the ADR). That is exactly the judgment round 1 said was missing from the body — not a restatement of the heading.
  • All seven classes walked with a real disposition each, not a bare shrug. I re-checked each None. against the artifact rather than accepting it: class 2 — confirmed by my own 246-ADR sweep above; class 4 — the only guidance on this PR is round 1's repair instruction ("add the section, do not touch the ADR text or its number"), and it was followed exactly (ADR bytes and number unchanged at this head; 0 inline review comments, 0 native reviews outstanding); class 5 — Tier-M scan clean, and the pnpm lint:worktree skip is honestly characterised as a no-op by scope, since the diff changes no biome-handled file; class 6 — structurally impossible on a +122/−0 single-added-file diff, confirmed against the REST file stats; class 7 — exactly one changed path, and it is precisely what A fabrika skill can pass every ship gate and never fire — triggering is unmeasured (/report: 100% precision, ~0% recall) #4750 asks for.
  • Checklist and section agree. The AC3 checkbox (body L26) now flags the departure inline — "This departs from the criterion as written — disclosed as class 1 under ## Deviations below." — so a reader who only skims the checklist is routed to the disclosure instead of reading a bare [x].
  • Round provenance is honest. The section's preamble states it was added in repair round 1 answering this FAIL, and that the class-1 entry belongs to the initial build, not to this round. §DEV's "repair appends, never replaces" rule is satisfied vacuously (there were no prior entries to preserve), and the preamble does the running log's work better than a bare round tag would have — it dates the departure, not just the edit.
  • Per §DEV's phrasing rule, this PASS means nothing undisclosed that this gate could see — never "no deviations exist." Class 3 and the artifact-free half of class 5 are Tier-D and unfalsifiable from here.

Also re-confirmed against the live body, since the repair claimed them: the §CP-by-content routing note (L47) is present, and its claim matches my own independent re-derivation exactly — no CODEOWNERS row for .decisions/**; guard-content-probe returns guard-touching [guard-vocabulary-match]; control-plane approval at head, then ship-it enqueue.

Read the PR head (§HEAD): all files under review were sourced from a1076e1069d3a51b558e363e5af727072d6fe81e through a per-run read-only ref — never a working-tree switch, and never the launched checkout's working copy.

Every acceptance criterion and every hygiene check passes, and the round-1 deviation-disclosure FAIL is discharged. This gate still does not authorize a merge: a @kamp-us/control-plane member must approve at the then-current head, and ship-it enqueues it (ADR 0135 approve-then-enqueue; ADR 0048 single merge authority). review-doc never merges. Merging will auto-close #4750 via the line-1 Fixes #4750; #5239 stays open.

Verdict-written: re-gate round 2 (body-only repair, same head)

Verdict-written: 2026-08-10T01:37:21Z