feat(fabrika): author the glossary skill and derive its CLI contract (#4711) - #5323
Conversation
…4711) Authors `/glossary` through /skill-creator — the register-maintenance twin of /adr, owning both `.glossary/TERMS.md` (domain nouns) and `.glossary/LANGUAGE.md` (architecture vocabulary). Six derived verbs: init, drift, lookup, sections, add, check. Three questions deliberately became NOTHING rather than verbs, each because an existing gate is the authority and a second answer could contradict it on a merge-gating question (ADR 0238, the `adr classify` test): machine-local paths (leak-guard, branch-protection-required on changed markdown), dead internal links (doc-links, lychee repo-wide), and control-plane class. A drift verb IS derived because glossary-drift is schedule-only, exits 0 by construction and is absent from ci-required — nothing is enforced, so nothing is contradicted. v1 was read for its scars and called nowhere: the laundered `git log` failure that reported BOOTSTRAP on a shallow clone, the empty stdout that made a clean sweep byte-identical to a dead run, the `apps packages` pathspec that read as "no drift" in any other layout, the ASCII-only tokenizer that made every Turkish product noun invisible, and substring suppression that inverted its own recall bias. Each is designed out and cited in a Grounding block. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No preview deploy
|
|
review-skill: PASS @ 9b79421 — merge-ready Adopted orphan, gated fresh. No verdict stood on this PR in any namespace when I started. The head was pinned up front and read read-only through a per-run ref, with no working tree switched. Every claim below binds to Class and §CP — both re-derived live, neither inherited
CI — re-read at head, not inheritedPaginated at Acceptance criteria — #4711, all nine rowsThe six original plus the three added by Amendment 1.
Rigor checklist
The five self-disclosures — weighed against the artifact, not inheritedThe author's reporting is candid to an unusual degree. I checked each rather than crediting it. 1. The sixth verb, 2. The 7 self-inflicted defects — resolved at head, and the access-scope claim now agrees with the verbs. This is the one I checked hardest, because "declared read-only, then added a verb that hits the network" is exactly the kind of contradiction that ships. At head the two sides state the same thing: the contract's Shared conventions says "No verb in this group reaches GitHub or the network", and One narrow note, non-blocking. Capabilities says "no issue writes", while a 3. Trigger optimizer skipped — accepted as disclosed. Nine consecutive sessions measured it flat, with the tool returning the original as 4. The impossible On consistency with #5293, the two PRs look opposite but follow one rule: never restate a measured number over a shape it was not measured on. #5293 could repair its fixtures without re-grading because that reviewer verified none of its 27 assertions keyed on the corrected field names — the repair was grading-neutral, so the recorded score survived it. Repairing Three things make the annotation adequate rather than a dodge: it ships inside One follow-up I am recording rather than charging. Unlike the prototyping set, which has a re-grade ticket at #5305, glossary's eval debt lives only in the file. The campaign's own practice is to put it on the board. A re-grade ticket for this set — covering 5. The 28/29 vs 23/29 headline — stated accurately enough to land. I counted the assertions at head: 7 + 6 + 5 + 5 + 6 = 29 across five evals, so the denominator is real. The caveat is not a footnote — the roughly-2x overstatement appears in the PR body and as defect (6) in the shipped UNKNOWN — recorded, and this verdict does not rest on itAmendment 1's v1-surface scar row turns on #5290, which is genuinely unruled, so I am not deciding it. Field 4 names four v1 surfaces. Two are discharged and I confirmed them: That is exactly the question #5290 asks: when a field-4 entry names no scar, does an explicit "no flaw here" line discharge the row, does silent omission, or does neither? It is open, Why the overall verdict does not rest on it. #5290's own stated deliverable is a recorded choice about the authoring-brief contract plus a remediation for the contracts already landed — explicitly not a per-PR repair. So this row is campaign-wide by the decision's own framing: blocking this one PR would not discharge it, and passing does not decide it. Whichever way #5290 lands, this contract takes the same remediation as its siblings. Every other criterion is independently satisfied. VerdictNine acceptance-criteria rows: eight PASS, one UNKNOWN that the verdict does not rest on and that I am barred from deciding. Five rigor checks: all PASS. CI green at head with no pending run. Zero machine-local, home-rooted, or sibling-repo paths across all 13 files. Every falsifiable claim I spot-checked against This is a well-built artifact whose author found and fixed a real design dead end before shipping, disclosed the defects their own fixes created, and reported the headline against their own interest. PASS — merge-ready. One follow-up to file, not blocking: the eval re-grade ticket for this set, mirroring #5305. |
Authors
/glossarythrough/skill-creator— the register-maintenance twin of/adr. It owns bothregisters:
.glossary/TERMS.md(the domain nouns) and.glossary/LANGUAGE.md(the architecturevocabulary). Per the founder ruling recorded on #4711, term-meaning judgement is a distinct judgement
kind and this skill stays separate from the pattern-doc sibling.
Fixes #4711
What is in the diff
claude-plugins/fabrika/skills/glossary/SKILL.md— 202 lines / ~2,065 wordsclaude-plugins/fabrika/skills/glossary/contract.md— six derived verbsclaude-plugins/fabrika/skills/glossary/evals/— 5 cases, 29 assertions, all populated, plus thecommitted fixture registers and decision records the contract's examples reproduce against
Verbs specified:
init,drift,lookup,sections,add,check.Implementation ticket: #5322. No verb is implemented here — the spec is the deliverable.
Three questions deliberately became nothing rather than verbs
Each already has an authority, and a second answer could contradict it on a merge-gating question —
the test that dropped
adr classifyon the pilot:leak-guard scan.github/workflows/leak-guard.yml,pull_requestandmerge_group;TERMS.mdis a doc surface by suffixdoc-links--offlineover every git-tracked*.md,pull_request+push: main, fail-closed on zero scopeExit seats
5and6are held empty for the first two rather than left unallocated, so a laterauthor cannot re-seat a different meaning on a number the base owns.
A drift verb is derived, and the distinction is evidence-backed rather than taste:
glossary-driftcarries onlyscheduleandworkflow_dispatchtriggers, is absent fromci-required.yml, and its own header states it is off the blocking path by construction (ADR 0128rejected extending the gate). Nothing is enforced, so nothing is contradicted, and ADR 0238 asks
fabrika to implement its own.
v1 read for its scars, called nowhere (ADR 0238)
Each is designed out and cited in a Grounding block:
git logwhose exit status was never checked, so a read failure laundered into a confident"never committed" — and a shallow clone (
actions/checkout's default depth) reproduced it on aregister that is committed, sending a caller to regenerate a populated file
apps packages, so any other layout read as "no drift"forever —
--pathsnow defaults to the whole tree, and a--pathsmatching zero tracked filesis exit
7, because fixing the default while leaving the flag able to manufacture the same falsenegative would be half a fix
/\b[a-z][a-z-]+\b/g, excluding uppercase and all non-ASCII — so every Turkishproduct noun the glossary exists for was structurally invisible
a large register (measured precision ~10%); suppression here is equality on the normalized key
The parenthetical trap is encoded rather than inherited: a parenthetical in a row is a
disambiguating qualifier, not an alias —
Database (tag)andtagare different terms — becausereusing v1's alias-splitting produced three false duplicates when it was tried (#4206).
Review — the calibration record (ADR 0270)
skill-reviewerran on the authored skill before this PR opened, and was handed, as calibration:claude-plugins/fabrika/docs/skill-conventions.md,claude-plugins/fabrika/docs/cli-interface-convention.md,and the landed sibling
claude-plugins/fabrika/skills/adr/(itsSKILL.mdand itscontract.md).Recorded here at the hand-off per ADR 0270, which landed on
mainduring this session.Three reviewers ran, and they overlapped on almost nothing:
skill-reviewerAll were addressed. The ones worth naming, because each would have shipped broken:
bootstrapwas a first-run dead end — declared the day-one path with no verb able to seed aregister or name a section. That is why
glossary initexists (FOUNDER RULING: fabrika is consumed as a plugin from day one — external consumability is a shipping criterion, and everything shipping-related belongs to M44 #4776 makes working in a foreignrepo a release criterion).
HELD-AMBIGUOUSordered a write no verb could make and the skill's own allowlist banned theonly other target. It now states the question in the report and directs the model to fire the
reportskill — composition through the model, per skill-conventions §3.DELIBERATE_GAPexports is a TypeScript error —GAP_EXPORTmatches one exact name. Onlyseat
5carries it;6is held by prose.function is the ASCII-only tokenizer the Grounding condemns. The contract now requires exporting
the set and explicitly forbids importing
tokenize.tagis declaredat
TERMS.md:86). Every example now pins a committed fixture.Evals
5 cases, two arms, spawned in the same turn. 28/29 with-skill vs 23/29 baseline (benchmark:
97.1% vs 79.1%, delta +0.18).
Discriminating: 7 rows = 3 distinct assertions — and the honest split matters more than the
headline. 1 substance discriminator each way, plus one closed-set terminal assertion repeated
across all five evals, which alone produced 5 of the 7 rows. A headline "7 discriminating" overstates
behavioural lift by roughly 2×.
minted "BLOCKED", "No register edits", "Resolved the collision as a homonym". Closed vocabulary is
the measured contribution.
baseline invented a non-canonical corner of the second register for a contested term. The skill arm
wrote nothing anywhere.
rewrite a declared row I have not read?" yes in eval-1 and no in eval-4, citing Glossary
control-planerow defines §CP by the retired hand-merge model #4727 bothtimes. A read-first rule now states which duty outranks the other.
Cost: with-skill 60.3k tokens / 79s against baseline 55.7k / 82s — +8% tokens, −4%
wall-clock, well below the +45–70% band prior authoring sessions measured. Diagnosis rather than a
silent report: the fixtures are transcript-served and the sandbox forbids executing verbs, so the
skill's real cost never materialises, and ~2k words against a ~56k-token baseline is ~3%. The
terminals confirm the skill was read.
Known eval defects, shipped annotated rather than silently repaired (full text in
evals/evals.json): eval-4's transcript is impossible under the contract's own equality-suppressionrule; four of five fixtures state the conclusion their assertions grade, so the substance
discriminators are deflated; eval-2 requires inventing a LANGUAGE section that eval-1 penalises; no
eval fires
STOPPED-UNKNOWNor exercises--replace. Repairing after grading would falsify thebenchmark.
The terminal vocabulary is unchanged by the fix wave, so the graded coverage still covers what
ships.
glossary init,--create-section, the read-first rule and the capability correction areeval-blind as shipped.
Trigger optimizer
Not run. Nine consecutive authoring sessions measured it flat — recall 0% throughout, precision
100%, and the tool returning the original as
best_description— across one-step skills, nine-verbprocedures and both construction modalities. Prior sessions record it as a ~20-minute compliance
measurement with a settled result. Stated as a deliberate omission, not a skipped step: this is the
one flow step I did not execute, and it is the reason the routing answer below is where the
triggering effort went.
Does a routing path reach this skill in deployment?
Partly, and the gap is already filed — cited, not re-filed.
A path does reach it: the landed sibling names it at
claude-plugins/fabrika/skills/adr/SKILL.md:104— a coined term routes to
.glossary/TERMS.md"otherwise/glossary" — andadr's own eval gradesthat hand-off. So
/glossaryhas a live upstream caller expecting it by name.Two problems, both open:
.claude/skillssymlink loads v1 skills regardless of the plugin toggle, so v1'sglossaryand this one are model-invoked under one name today. The two contradict each other onscope: v1 declares it never edits
LANGUAGE.md; this one owns both registers. Until that fork isruled the description is the only disambiguator, and it is written to be one.
CLAUDE.mdpins skill routing to a filesystem path, so a fabrika skill cannot inheritrouting.
CLAUDE.mdnames.glossary/as a doc surface and routes to no glossary skill at all.What I could not verify
fabrika glossaryinvocation.#4481's measured ~10% precision figure is quoted from that issue; I confirmed the figure and thetokenizer regex in v1's source, but did not re-measure precision.
#4776's plugin-consumability ruling implies a discovery path for.glossary/in aforeign repo is not settled anywhere I could find, and I did not decide it.