Skip to content

feat(fabrika): author the glossary skill and derive its CLI contract (#4711) - #5323

Merged
usirin merged 1 commit into
mainfrom
umut/fabrika-glossary
Aug 10, 2026
Merged

usirin merged 1 commit into
mainfrom
umut/fabrika-glossary

Conversation

@usirin

@usirin usirin commented Aug 10, 2026

Copy link
Copy Markdown
Member

Authors /glossary through /skill-creator — the register-maintenance twin of /adr. It owns both
registers: .glossary/TERMS.md (the domain nouns) and .glossary/LANGUAGE.md (the architecture
vocabulary). Per the founder ruling recorded on #4711, term-meaning judgement is a distinct judgement
kind and this skill stays separate from the pattern-doc sibling.

Fixes #4711

What is in the diff

  • claude-plugins/fabrika/skills/glossary/SKILL.md — 202 lines / ~2,065 words
  • claude-plugins/fabrika/skills/glossary/contract.md — six derived verbs
  • claude-plugins/fabrika/skills/glossary/evals/ — 5 cases, 29 assertions, all populated, plus the
    committed fixture registers and decision records the contract's examples reproduce against

Verbs specified: init, drift, lookup, sections, add, check.
Implementation ticket: #5322. No verb is implemented here — the spec is the deliverable.

Three questions deliberately became nothing rather than verbs

Each already has an authority, and a second answer could contradict it on a merge-gating question —
the test that dropped adr classify on the pilot:

Question Who already answers it Evidence
machine-local paths in the register leak-guard scan .github/workflows/leak-guard.yml, pull_request and merge_group; TERMS.md is a doc surface by suffix
dead internal links doc-links lychee --offline over every git-tracked *.md, pull_request + push: main, fail-closed on zero scope
control-plane class the merge gate never predicted author-side

Exit seats 5 and 6 are held empty for the first two rather than left unallocated, so a later
author cannot re-seat a different meaning on a number the base owns.

A drift verb is derived, and the distinction is evidence-backed rather than taste:
glossary-drift carries only schedule and workflow_dispatch triggers, is absent from
ci-required.yml, and its own header states it is off the blocking path by construction (ADR 0128
rejected extending the gate). Nothing is enforced, so nothing is contradicted, and ADR 0238 asks
fabrika to implement its own.

v1 read for its scars, called nowhere (ADR 0238)

Each is designed out and cited in a Grounding block:

  • a git log whose exit status was never checked, so a read failure laundered into a confident
    "never committed" — and a shallow clone (actions/checkout's default depth) reproduced it on a
    register that is committed, sending a caller to regenerate a populated file
  • empty stdout on a clean sweep, byte-identical to a run that died early
  • a diff defaulted to the literal pathspec apps packages, so any other layout read as "no drift"
    forever — --paths now defaults to the whole tree, and a --paths matching zero tracked files
    is exit 7, because fixing the default while leaving the flag able to manufacture the same false
    negative would be half a fix
  • a tokenizer of /\b[a-z][a-z-]+\b/g, excluding uppercase and all non-ASCII — so every Turkish
    product noun the glossary exists for was structurally invisible
  • substring-tolerant suppression in both directions, which inverts its own stated recall bias against
    a large register (measured precision ~10%); suppression here is equality on the normalized key
  • a write whose response was trusted instead of re-read

The parenthetical trap is encoded rather than inherited: a parenthetical in a row is a
disambiguating qualifier, not an alias — Database (tag) and tag are different terms — because
reusing v1's alias-splitting produced three false duplicates when it was tried (#4206).

Review — the calibration record (ADR 0270)

skill-reviewer ran on the authored skill before this PR opened, and was handed, as calibration:
claude-plugins/fabrika/docs/skill-conventions.md, claude-plugins/fabrika/docs/cli-interface-convention.md,
and the landed sibling claude-plugins/fabrika/skills/adr/ (its SKILL.md and its contract.md).
Recorded here at the hand-off per ADR 0270, which landed on main during this session.

Three reviewers ran, and they overlapped on almost nothing:

Pass Findings
skill-reviewer 9 blocking / 6 non-blocking — verdict Needs Major Revision, contract passing 2 of 7 completeness points
mechanical exit-matrix audit 20 defects
premise verification 3 critical, 3 high
narrow fix verification 7 new defects created by the fixes themselves

All were addressed. The ones worth naming, because each would have shipped broken:

  • bootstrap was a first-run dead end — declared the day-one path with no verb able to seed a
    register or name a section. That is why glossary init exists (FOUNDER RULING: fabrika is consumed as a plugin from day one — external consumability is a shipping criterion, and everything shipping-related belongs to M44 #4776 makes working in a foreign
    repo a release criterion).
  • HELD-AMBIGUOUS ordered a write no verb could make and the skill's own allowlist banned the
    only other target. It now states the question in the report and directs the model to fire the
    report skill — composition through the model, per skill-conventions §3.
  • Two DELIBERATE_GAP exports is a TypeScript error — GAP_EXPORT matches one exact name. Only
    seat 5 carries it; 6 is held by prose.
  • The stopword list the contract said to import is not exported, and the only exported term
    function is the ASCII-only tokenizer the Grounding condemns. The contract now requires exporting
    the set and explicitly forbids importing tokenize.
  • Three examples resolved against the live register and printed wrong cells (tag is declared
    at TERMS.md:86). Every example now pins a committed fixture.
  • "Products" was assigned to both registers in one sentence — and that was eval-2's graded seam.

Evals

5 cases, two arms, spawned in the same turn. 28/29 with-skill vs 23/29 baseline (benchmark:
97.1% vs 79.1%, delta +0.18).

Discriminating: 7 rows = 3 distinct assertions — and the honest split matters more than the
headline. 1 substance discriminator each way, plus one closed-set terminal assertion repeated
across all five evals, which alone produced 5 of the 7 rows. A headline "7 discriminating" overstates
behavioural lift by roughly 2×.

  • The baseline reached the right judgement in all five and never once used a closed token — it
    minted "BLOCKED", "No register edits", "Resolved the collision as a homonym". Closed vocabulary is
    the measured contribution.
  • The one substance win: without the skill, "hold" degraded into "park it somewhere else" — the
    baseline invented a non-canonical corner of the second register for a contested term. The skill arm
    wrote nothing anywhere.
  • The one with-skill failure is the sharpest finding in the suite: the skill answered "may I
    rewrite a declared row I have not read?"
    yes in eval-1 and no in eval-4, citing Glossary control-plane row defines §CP by the retired hand-merge model #4727 both
    times. A read-first rule now states which duty outranks the other.

Cost: with-skill 60.3k tokens / 79s against baseline 55.7k / 82s — +8% tokens, −4%
wall-clock
, well below the +45–70% band prior authoring sessions measured. Diagnosis rather than a
silent report: the fixtures are transcript-served and the sandbox forbids executing verbs, so the
skill's real cost never materialises, and ~2k words against a ~56k-token baseline is ~3%. The
terminals confirm the skill was read.

Known eval defects, shipped annotated rather than silently repaired (full text in
evals/evals.json): eval-4's transcript is impossible under the contract's own equality-suppression
rule; four of five fixtures state the conclusion their assertions grade, so the substance
discriminators are deflated; eval-2 requires inventing a LANGUAGE section that eval-1 penalises; no
eval fires STOPPED-UNKNOWN or exercises --replace. Repairing after grading would falsify the
benchmark.

The terminal vocabulary is unchanged by the fix wave, so the graded coverage still covers what
ships. glossary init, --create-section, the read-first rule and the capability correction are
eval-blind as shipped.

Trigger optimizer

Not run. Nine consecutive authoring sessions measured it flat — recall 0% throughout, precision
100%, and the tool returning the original as best_description — across one-step skills, nine-verb
procedures and both construction modalities. Prior sessions record it as a ~20-minute compliance
measurement with a settled result. Stated as a deliberate omission, not a skipped step: this is the
one flow step I did not execute, and it is the reason the routing answer below is where the
triggering effort went.

Does a routing path reach this skill in deployment?

Partly, and the gap is already filed — cited, not re-filed.

A path does reach it: the landed sibling names it at claude-plugins/fabrika/skills/adr/SKILL.md:104
— a coined term routes to .glossary/TERMS.md "otherwise /glossary" — and adr's own eval grades
that hand-off. So /glossary has a live upstream caller expecting it by name.

Two problems, both open:

What I could not verify

  • The six verbs have never executed. Every behaviour here is specified, not observed.
  • The eval sandbox forbade running verbs, so no run exercised a real fabrika glossary invocation.
  • #4481's measured ~10% precision figure is quoted from that issue; I confirmed the figure and the
    tokenizer regex in v1's source, but did not re-measure precision.
  • Whether #4776's plugin-consumability ruling implies a discovery path for .glossary/ in a
    foreign repo is not settled anywhere I could find, and I did not decide it.

…4711)

Authors `/glossary` through /skill-creator — the register-maintenance twin of
/adr, owning both `.glossary/TERMS.md` (domain nouns) and `.glossary/LANGUAGE.md`
(architecture vocabulary).

Six derived verbs: init, drift, lookup, sections, add, check.

Three questions deliberately became NOTHING rather than verbs, each because an
existing gate is the authority and a second answer could contradict it on a
merge-gating question (ADR 0238, the `adr classify` test): machine-local paths
(leak-guard, branch-protection-required on changed markdown), dead internal links
(doc-links, lychee repo-wide), and control-plane class. A drift verb IS derived
because glossary-drift is schedule-only, exits 0 by construction and is absent
from ci-required — nothing is enforced, so nothing is contradicted.

v1 was read for its scars and called nowhere: the laundered `git log` failure that
reported BOOTSTRAP on a shallow clone, the empty stdout that made a clean sweep
byte-identical to a dead run, the `apps packages` pathspec that read as "no drift"
in any other layout, the ASCII-only tokenizer that made every Turkish product noun
invisible, and substring suppression that inverted its own recall bias. Each is
designed out and cited in a Grounding block.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@usirin usirin added this to the fabrika campaign milestone Aug 10, 2026
@usirin usirin added the fabrika fabrika campaign wave — kampus-pipeline v2 label Aug 10, 2026
@github-actions

github-actions Bot commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

No preview deploy

  • No preview deploy for this PR — its diff touches no deploy-relevant path, so no preview stack was minted and e2e is not applicable. (9b79421)
  • web — Stage pr-5323 torn down.

@usirin

usirin commented Aug 10, 2026

Copy link
Copy Markdown
Member Author

review-skill: PASS @ 9b79421 — merge-ready

Adopted orphan, gated fresh. No verdict stood on this PR in any namespace when I started. The head was pinned up front and read read-only through a per-run ref, with no working tree switched. Every claim below binds to 9b79421f, and I re-confirmed the head had not moved before posting.

Class and §CP — both re-derived live, neither inherited

  • Namespace set — class-probe classify --namespaces over the 13 changed files returns exactly review-skill. All 13 live under claude-plugins/fabrika/skills/glossary/; no code, doc, or UI surface is in reach. One namespace required, one emitted.
  • §CP — cp-classify classify returns not-control-plane [path-clear-no-content-source]: no path matched the live CONTROL_PLANE_RE, and no .decisions/** file is present, so the ADR-0164 content clause has nothing to decide. Confirmed against live CODEOWNERS independently — the skills row is scoped to claude-plugins/kampus-pipeline/skills/, and claude-plugins/fabrika/** matches no row at all.
  • Non-§CP on both axes, so this is a SHA-bound marker and this PASS auto-ships. I am the only check this PR gets, and I weighted the review accordingly.

CI — re-read at head, not inherited

Paginated at 9b79421f: declared total_count 46 == 46 flattened, every run completed — 39 success, 7 skipped, zero failing, zero pending. The authoring session's "36 passing, 2 still running" was a snapshot; those landed. Not gated on its number.

Acceptance criteria — #4711, all nine rows

The six original plus the three added by Amendment 1.

  • [PASS] Fresh session authors SKILL.md from the issue and repo alone. Author-asserted and not gate-observable from the diff; recorded as disclosed rather than graded. Nothing in the artifact contradicts it.
  • [PASS] skill-reviewer ran before the PR opened, findings addressed, PR records the pass. The body tabulates four passes (9 blocking / 6 non-blocking, a 20-defect exit-matrix audit, a 3-critical premise pass, and a 7-defect narrow verification of the fixes themselves). The record is what this row asks for and it is present and specific; the running is author-asserted.
  • [PASS] One PR carries both SKILL.md and contract.md with Fixes #4711. Both present.
  • [PASS] Contract readable without the transcript. Verb inventory table with a why-mechanical column per verb; per-verb invocation, flag table (type / required / default), output shape including the --json variant, exit-status table, error taxonomy with the literal stderr string and its kind, an explicit scope and zero-scope line, and literal examples. A downstream coder has what they need.
  • [PASS] No verbs implemented, no v1 script ported. All 13 files are under the skill directory; packages/ is untouched.
  • [PASS] Every brief field usable as written. The session flagged what it could not settle (the FOUNDER RULING: fabrika is consumed as a plugin from day one — external consumability is a shipping criterion, and everything shipping-related belongs to M44 #4776 foreign-repo discovery question) rather than guessing it.
  • [PASS] Names no pipeline-cli verb as a dependency, and no fence invokes anything under claude-plugins/kampus-pipeline/ or packages/pipeline-cli/. I extracted every fenced command line from both files: seven in SKILL.md, six in contract.md, and every one is fabrika. The single sibling call is fabrika adr resolve, a fabrika verb — specified in the landed adr/contract.md and actually implemented under packages/fabrika-cli/src/adr/, so the citation resolves.
  • [PASS] No relay-only wrapper verb, and no second answer to a gate-enforced question. Three questions were deliberately dropped rather than derived — machine-local paths (leak-guard), dead links (doc-links), control-plane class (the merge gate) — and exit seats 5 and 6 are held empty for the first two rather than left unallocated, so a later author cannot re-seat a different meaning on a number the base owns. That is the adr classify test applied correctly. The one place the PR derives a verb over an existing v1 tool is drift, and it earns it with evidence I checked myself: the glossary-drift workflow carries only schedule and workflow_dispatch, its own header states it is off the blocking path by construction (ADR 0128 rejected extending the gate), and it is absent from ci-required. Nothing enforced, so nothing contradicted.
  • [UNKNOWN — see below] Scars recorded for each v1 surface named in field 4. Partially discharged, and the remainder turns on an unruled question.

Rigor checklist

  • [PASS] Behavioral correctness. I traced the six steps as an executing agent. Every branch reaches exactly one of the five terminals: bootstrap routes to step 4's init, clean stops on NO-CHANGE, a contested meaning stops on HELD-AMBIGUOUS, an unmerged citation stops on HELD-UNMERGED-ADR, a non-zero verb exit stops on STOPPED-UNKNOWN, and a landed row ends RECORDED. No branch dangles and no guard is unreachable. I probed the one seam most likely to contradict itself — SKILL.md's claim that an absent LANGUAGE.md degrades under --register both rather than failing, against lookup's exit 11 — and the contract handles it explicitly in Shared conventions: "absent degrades, unreadable refuses", with 11 reserved for a file that is present and unreadable. The two agree. That clause is correctly identified in the contract as the most load-bearing one in the group.
  • [PASS] Trigger / description quality. The description names the register pair, the real invocation phrases, and three explicit negatives (not sözlük, not an architecture audit, not the pattern-doc lane). It is written to carry disambiguating weight, which matters here — see the shadowing note.
  • [PASS] Cross-skill conflict / shadowing. There is a real collision and it is disclosed, not hidden: v1's skill is also name: glossary, and the two contradict on scope — v1 declares it edits only TERMS.md and explicitly not LANGUAGE.md, while this one owns both. I confirmed that by reading v1's frontmatter on main. It is not introduced by this PR: it is the campaign-wide condition of rebuilding v1 skill-for-skill, it is filed and open at The .claude/skills symlink loads v1 skills regardless of the plugin toggle, so v1 and fabrika skills are model-invoked under one name today #4829, its one-change fix is filed and open at Retire the v1 skill roster in one change: both .claude symlinks plus the ADR 0077 suppression #5276, and the PR cites both rather than re-filing. Under the ruled sequencing this is a scheduled consequence, not a defect this PR created. Recorded, not charged.
  • [PASS] Gate-invariant preservation. Nothing gate-critical is in the diff's reach — no .claude, no .github, no gate skill, no formats contract, no marker matcher. The check ran and had nothing to weaken.
  • [PASS] Contract implementation ticket (ADR 0248). In scope — the diff adds a fabrika contract.md. Six fabrika glossary verbs are specified and none exists #5322 is named in the body, and I read it: open, titled for these six verbs, enumerating all six by name plus the four registration obligations and the one required change to a shipped module. It routes to this contract, not to something else. Fixes #4711 closes the brief and does not discharge this row; the separate ticket does.

The five self-disclosures — weighed against the artifact, not inherited

The author's reporting is candid to an unusual degree. I checked each rather than crediting it.

1. The sixth verb, init — sound remedy, correctly diagnosed. The dead end was real and the contract states it precisely: before init, the bootstrap path "terminates in add exit 13 with no section to name, which is a first-run dead end." A skill that declares day-one as a first-class state and then ships no verb able to seed a register is the #4776 violation the author names. init closes it properly rather than patching around it: it refuses --register both so two registers are never created by one ambiguous call, refuses to overwrite an existing register (12), and re-reads to verify what landed (9). The loop now closes end to end — init, then sections, then add --create-section for a genuinely new home. Adding the verb was right, and catching it before the PR opened is the pass working.

2. The 7 self-inflicted defects — resolved at head, and the access-scope claim now agrees with the verbs. This is the one I checked hardest, because "declared read-only, then added a verb that hits the network" is exactly the kind of contradiction that ships. At head the two sides state the same thing: the contract's Shared conventions says "No verb in this group reaches GitHub or the network", and SKILL.md's Capabilities block says the lone network reach is the sibling fabrika adr resolve, naming what it does (fetches a base ref, enumerates open pull requests) and why it must. Those are consistent, and the disclosure is affirmative rather than buried. I spot-verified five of the seven fixes and all held: DELIBERATE_GAP sits on seat 5 only, with an explicit "Do not invent DELIBERATE_GAP_2"; the contract requires exporting STOPWORDS and expressly forbids importing tokenize — and I checked that premise against main, where sweep.ts does carry a module-private const STOPWORDS and does export the ASCII-only tokenize, so the claim is true rather than assumed; every worked example now pins the committed fixture register via --dir instead of resolving against the live one; HELD-AMBIGUOUS now states the question and fires the report skill rather than ordering a write no verb could make; and the "Products" double-assignment is resolved into one rule — the domain row is TERMS.md, only the brand spelling is LANGUAGE.md.

One narrow note, non-blocking. Capabilities says "no issue writes", while a HELD-AMBIGUOUS run fires report, which does file. The skill draws that line explicitly at the composition seam — "This skill writes registers and nothing else, so it does not file the issue itself" — and reconciles it at the point of use, so it reads as a defensible framing rather than a contradiction. Worth a sharper phrasing next pass; it does not change what the skill does.

3. Trigger optimizer skipped — accepted as disclosed. Nine consecutive sessions measured it flat, with the tool returning the original as best_description. Not an acceptance criterion, and stated as a deliberate omission with its evidence rather than quietly dropped. That is the right way to skip a step.

4. The impossible eval-4 fixture — acceptable as shipped, and I can say why it differs from #5293. I verified the impossibility myself rather than taking it on report: the fixture's drift prints three candidates that the very next lookup reports as declared, and since suppression is equality on the normalized key, a contract-faithful drift must have printed clean. The transcript cannot happen. The disclosure is accurate and precisely stated.

On consistency with #5293, the two PRs look opposite but follow one rule: never restate a measured number over a shape it was not measured on. #5293 could repair its fixtures without re-grading because that reviewer verified none of its 27 assertions keyed on the corrected field names — the repair was grading-neutral, so the recorded score survived it. Repairing eval-4 here would not be grading-neutral: the transcript is the thing the model reads, so changing it invalidates the run that produced 28/29. Faced with a non-neutral repair, the author annotated instead of silently repairing and keeping the number. Same rule, opposite action, because the repairs differ in kind. Consistent — and I would have objected to the reverse.

Three things make the annotation adequate rather than a dodge: it ships inside evals.json, so it travels with the fixture instead of dying in the PR body; the author establishes the defect did not corrupt the specific claim made, since both arms landed the intended no-op and the row still measured vocabulary; and evals are explicitly outside this brief's scope — #4711 states it carries no eval mechanics, no bar, no corpus format, those belonging to the eval epic (#4649). Failing this PR over an eval artifact would enforce a criterion the brief expressly disclaims.

One follow-up I am recording rather than charging. Unlike the prototyping set, which has a re-grade ticket at #5305, glossary's eval debt lives only in the file. The campaign's own practice is to put it on the board. A re-grade ticket for this set — covering eval-4, the four answer-leaking fixtures, the eval-2 / eval-1 rubric conflict, and the four eval-blind surfaces (init, --create-section, the read-first rule, the capability correction) — should be filed. Non-blocking, and not mine to file.

5. The 28/29 vs 23/29 headline — stated accurately enough to land. I counted the assertions at head: 7 + 6 + 5 + 5 + 6 = 29 across five evals, so the denominator is real. The caveat is not a footnote — the roughly-2x overstatement appears in the PR body and as defect (6) in the shipped evals.json, and the headline never appears without it. The author also volunteers the sharpest finding against their own skill: the single with-skill failure, where the skill answered the same question two opposite ways across eval-1 and eval-4. The +8% token cost is diagnosed rather than banked as a win — transcript-served fixtures and a sandbox that forbids executing verbs mean the skill's real cost never materialised. That is a claim reported against interest. It lands as written.

UNKNOWN — recorded, and this verdict does not rest on it

Amendment 1's v1-surface scar row turns on #5290, which is genuinely unruled, so I am not deciding it.

Field 4 names four v1 surfaces. Two are discharged and I confirmed them: glossary-drift carries six scars, each cited in a Grounding block and each designed out; leak-guard is handled as the merge-gate authority rather than duplicated. The other two, pointer-guard and scratchpad, appear zero times in contract.md — silent omission, with no non-applicability line either way.

That is exactly the question #5290 asks: when a field-4 entry names no scar, does an explicit "no flaw here" line discharge the row, does silent omission, or does neither? It is open, type:decision, status:triaged, and its only comment is the triage note — nobody has analysed it, and #5290 itself records that this criterion has already been enforced inconsistently across sibling PRs (grilling passed, graduate failed). I report UNKNOWN. I am not passing it, failing it, or inferring it from how other contracts happen to read — an invented answer here becomes precedent for a decision the founder has not made.

Why the overall verdict does not rest on it. #5290's own stated deliverable is a recorded choice about the authoring-brief contract plus a remediation for the contracts already landed — explicitly not a per-PR repair. So this row is campaign-wide by the decision's own framing: blocking this one PR would not discharge it, and passing does not decide it. Whichever way #5290 lands, this contract takes the same remediation as its siblings. Every other criterion is independently satisfied.

Verdict

Nine acceptance-criteria rows: eight PASS, one UNKNOWN that the verdict does not rest on and that I am barred from deciding. Five rigor checks: all PASS. CI green at head with no pending run. Zero machine-local, home-rooted, or sibling-repo paths across all 13 files. Every falsifiable claim I spot-checked against main — the STOPWORDS and tokenize state, the glossary-drift triggers and its absence from ci-required, the cited fabrika modules, adr resolve's existence — held.

This is a well-built artifact whose author found and fixed a real design dead end before shipping, disclosed the defects their own fixes created, and reported the headline against their own interest. PASS — merge-ready.

One follow-up to file, not blocking: the eval re-grade ticket for this set, mirroring #5305.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fabrika fabrika campaign wave — kampus-pipeline v2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Authoring brief: /glossary — fabrika skill (wave 2)

1 participant