From 1e57deb90f0a26da7ad8aa3bc4f196e863ab02e0 Mon Sep 17 00:00:00 2001 From: Trevin Chow Date: Fri, 21 Aug 2026 23:08:25 -0700 Subject: [PATCH 1/5] docs(agent-plugins): attribute the 8000-byte skill cap to its real owner The 8000-byte SKILL.md ratchet was widely believed to be a requirement of the Agent Plugins standard. Verified against primary sources on 2026-08-21: it is not. Neither the Agent Plugins spec (1.0.0 and 1.1.0) nor the Agent Skills spec it defers to imposes any body size limit -- Agent Skills constrains only frontmatter and explicitly frames its "< 5000 tokens" / "under 500 lines" numbers as recommendations. The real 8000 is Codex's MAX_SKILL_PROMPT_BYTES (codex-rs/ext/skills/src/ render.rs:19), which applies only on the Agent Plugins path. Record a second, previously unmodeled bound alongside it: Claude Code auto-compaction re-attaches each invoked skill keeping only its first 5,000 tokens, within a 25,000-token combined budget, on every path regardless of manifest. At roughly 20KB that is looser than the existing ratchet, which therefore subsumes it -- so the ratchet stays at 8000 and no separate compaction gate is added. Both truncations keep the start of the file, which makes body ordering load-bearing: what must survive belongs above what may be cut. Also correct two stale upstream claims this verification falsified -- a comment asserting the constant was absent from the Codex source, and a doc citing the since-removed core-skills crate for it -- and capture the generalizable method as a new learning: settle externally-attributed constraints at the spec text and the implementing source, and treat negative claims about upstream as perishable. No behavior change; no test assertions or versions touched. Claude-Session: https://claude.ai/code/session_013DVEGTARyLzqxxzvzoVXEA --- CONCEPTS.md | 5 + ...ly-attributed-constraints-at-the-source.md | 109 ++++++++++++++++++ ...plugins-schema-is-a-host-routing-switch.md | 2 +- .../size-driven-skill-restructure.md | 2 + docs/specs/agent-plugins.md | 16 +++ tests/codex-skill-prompt-budget.test.ts | 11 ++ tests/real-plugin-conversion.test.ts | 19 +-- tests/skill-conventions.test.ts | 16 ++- 8 files changed, 166 insertions(+), 14 deletions(-) create mode 100644 docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md diff --git a/CONCEPTS.md b/CONCEPTS.md index 3dd354279..efacdef15 100644 --- a/CONCEPTS.md +++ b/CONCEPTS.md @@ -99,6 +99,11 @@ A bulk evidence artifact — verbatim quotes with source pointers, gathered by a ### Outcome spine The part of a Skill that must hold without any reference loaded: the result it produces and who consumes it next, the done condition, the safe failure direction, and the facts the agent cannot derive from the repository in front of it. Everything else in the skill is protocol or judgment attached to that spine, and a block that cannot name its spine is restated before it is edited. +### Host prompt budget +The ceiling a specific agent host places on how much of a Skill's body it will keep in the model-visible prompt, enforced by that host's own loader rather than by any plugin or skill specification. Each host sets its own and reaches it by a different route — one may truncate the body outright and only for packages declaring a particular manifest shape, another may re-attach a shortened copy of each invoked skill after summarizing a long conversation — so a body that survives intact on one host can silently lose its tail on another. + +Every known truncation keeps the beginning of the body and discards the rest, and none of them reports an error. That is what makes ordering load-bearing: what must survive belongs above what may be cut, and a stop class or boundary rule sitting below a long routing block can disappear while every mechanical check still passes. A repository that ratchets body size sets that ratchet from the tightest bound among the hosts it ships against — a scoped engineering constraint whose owner and scope are re-verified at the source, not a conformance requirement of any specification. Load stub and Phase-loaded kernel are the two shapes content takes when it moves out of the body to fit. + ### Load stub The inline remnant left in a Skill when load-bearing content moves to a reference file: a load instruction that names what the reference contains and the failure mode of skipping it, while keeping no detail an agent could improvise from — making the load structurally necessary rather than advisory. diff --git a/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md b/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md new file mode 100644 index 000000000..301f055bb --- /dev/null +++ b/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md @@ -0,0 +1,109 @@ +--- +title: "Verify an externally-attributed constraint against the spec text and the implementing source before encoding it" +date: 2026-08-21 +category: conventions +module: "skills (SKILL.md size ratchet) and tests/codex-skill-prompt-budget.test.ts" +problem_type: convention +component: tooling +severity: medium +applies_when: + - "A test, lint rule, or convention enforces a number attributed to an external standard, spec, or platform" + - "The attribution carries no file-and-line citation into the spec text or the implementing source" + - "A review comment or another agent asserts \"the spec requires X\" and you are about to design around X" + - "Tightening, loosening, or justifying such a gate" + - "A repo comment makes a negative claim about upstream (\"no such constant exists\")" +tags: [provenance, primary-sources, agent-plugins, agent-skills-spec, codex, verification, stale-claims] +--- + +# Verify an externally-attributed constraint against the spec text and the implementing source before encoding it + +## Context + +This repo enforces an 8,000-byte ceiling on every `SKILL.md` body. Around that ratchet a belief had settled: that 8,000 bytes is a requirement of the **Agent Plugins standard**. Contributing agents repeated it in review, and it leaked into repo comments as fact. + +On 2026-08-21 the claim was checked against primary sources and found false. The Agent Plugins spec has no size limit of any kind; the real 8,000 is one host's implementation constant. A second, differently-shaped bound on another host had gone entirely unmodeled. + +The full provenance — which bound comes from which host, what each one truncates, and the invalidation triggers — is owned by `docs/specs/agent-plugins.md` under "Skill body size: what actually constrains it". **This doc does not restate it.** What is captured here is the part that generalizes past skill sizing: how the question got settled, and the failure mode that let the wrong answer stand. + +This is the sibling of `antigravity-target-empirical-format-verification.md` ("verify a new target's plugin format against the CLI binary, not its docs"). Same principle, different artifact: there the docs were unreliable, here the *ecosystem consensus* was. + +## Guidance + +**1. Settle a claimed external constraint at the source, not through what people say about it.** + +Two kinds of primary source answer nearly every such question, and both are usually one command away: + +```bash +# the spec text — is the constraint even in there? +gh api repos/agentplugins/agent-plugins-spec/contents/spec/1.0.0.md --jq .content | base64 -d | grep -inE 'bytes|size|truncat|8000' +gh api repos/agentplugins/agent-plugins-spec/contents/spec/1.1.0.md --jq .content | base64 -d | grep -inE 'bytes|size|truncat|8000' + +# the implementing source — where does the number actually live? +gh api repos/openai/codex/contents/codex-rs/ext/skills/src/render.rs --jq .content | base64 -d | grep -n 'MAX_SKILL_PROMPT_BYTES\|CHAR_BUDGET' +``` + +Check **every published version** of a spec, not just the one being cited — a constraint absent from `1.0.0` and present in `1.1.0` is a different answer than "not in the spec." + +Web search is the wrong instrument here. It returned confident, unsourced restatements of the same folklore, and one result asserted that no such limit could be found anywhere. A third-party blog restated a second host's numbers correctly, but only that host's own documentation could confirm them. Secondary sources reproduce consensus; the question was whether the consensus was true. + +**2. Record provenance next to the enforcement, with a date.** + +A number in a test needs three things beside it: which component owns it, what scope it applies to, and when that was last verified. Without them the number is unfalsifiable in practice — nobody re-checks it, nobody scopes it, and nobody notices when the implementing host moves it. + +**3. Distinguish the bound from the standard, because the difference changes decisions.** + +A limit believed to come from a standard is treated as immovable and universal. A limit known to come from one host's renderer is a scoped engineering constraint you can reason about: it may apply on only one code path, another host may impose a different one, and the two can be compared. Here the practical consequences were concrete — the repo's ratchet turned out to be the *tighter* of two bounds and to subsume the other, which is a fact you can only state once both are attributed. + +**4. When several unrelated constants share a number, name all of them.** + +Three unrelated 8,000s circulate in this ecosystem. That is why the folklore is self-reinforcing: any contributor who greps one of them "confirms" the belief. Recording all three with their distinct scopes is what stops the next agent from re-deriving the wrong conclusion from a correct grep. + +**5. Treat a negative claim about upstream as perishable.** + +A repo comment asserted that the body constant "is not in the Codex source." That was true when written and silently became false when upstream added it. Negative claims about someone else's codebase age worse than positive ones — a positive claim breaks loudly when the path moves, while a negative one just quietly stops being true. + +So: give any claim about upstream a `file:line` and a verification date, and re-check it when you touch the surrounding rule. `tests/real-plugin-conversion.test.ts` and `docs/solutions/integrations/agent-plugins-schema-is-a-host-routing-switch.md` both carried stale upstream source paths found this way. + +## Why This Matters + +**A misattributed constraint cannot be questioned.** "The standard requires it" ends the conversation; "this host's renderer truncates there, verified on this date" invites the next useful question. The whole repo was tuned to one host's truncation while a second host's bound went unmodeled for months. It turned out to be looser, so the existing ratchet already covered it — that was luck, not design. + +**The attribution determines what compliance means.** Once the bound is known to apply only on a specific host path, shrinking a skill stops being a compliance chore with an invented authority behind it and becomes a precondition for a specific capability, with a reason a reviewer can weigh. + +**The folklore regenerates unless the disambiguation is written down.** Correcting one comment does not help if three same-numbered constants remain undistinguished. Recording the scopes is the durable fix. + +## When to Apply + +- A repo test, lint rule, or convention enforces a number, cap, or format attributed to an external standard or platform, and the attribution has no citation into spec text or implementing source. +- Someone asserts "the spec requires X" and you are about to design around X. +- You are about to move such a number. Re-verify before, not after. +- A stated provenance is old enough that upstream may have moved — especially a claim that something upstream does *not* exist. + +Do **not** apply the full dig to internal repo conventions with no external claim behind them, or to a number the tree already cites to a specific file and line with a recent verification date. The cost is a few commands; spend it where an external party owns the truth. + +## Examples + +**Before — the belief as it read in the tree:** + +> The 8KB body cap comes from the Agent Plugins spec. Skills must fit under it to be conformant. + +Nothing to check, nowhere to look, and wrong on three counts: not from that spec, not universal, and not the only bound. + +**After — the same rule with owner, scope, and date.** `docs/specs/agent-plugins.md` now carries a provenance table naming each bound's owning component, the code path it applies to, and what the truncation actually does, plus a re-verify trigger for each. `tests/codex-skill-prompt-budget.test.ts` carries the same provenance in its header, next to the constant it enforces. + +**Before — a perishable negative claim:** + +> the body constant is not in the Codex source (codex-rs/core-skills has no body-size constant) + +**After** — the comment names the constants that now exist, cites the file that defines them, states which code path each applies to, and stamps the verification date. + +**The operational consequence that outranks the byte count.** Both known truncations keep the **start** of the file. Neither reports an error. So body ordering is load-bearing: what must survive belongs above what may be cut, and a stop class or boundary rule must never sit below a long routing block. A skill can pass every mechanical gate, sit comfortably under the line, and still lose its stop conditions on a host that truncates or compacts. No test catches that — only the ordering discipline does. `docs/solutions/skill-design/size-driven-skill-restructure.md` carries this alongside the restructuring procedure. + +## Related + +- `docs/specs/agent-plugins.md` — owns the provenance table for both bounds and their re-verify triggers +- `docs/solutions/skill-design/size-driven-skill-restructure.md` — the restructuring procedure and the start-of-file ordering consequence +- `docs/solutions/integrations/agent-plugins-schema-is-a-host-routing-switch.md` — why the root manifest stays schema-less, which is what scopes one of the two bounds +- `docs/solutions/conventions/antigravity-target-empirical-format-verification.md` — the same principle applied to a target's plugin format +- Repo issues #1411, #1412 (the discovery) and PR #1479 (the only measured truncation-in-practice evidence) +- Upstream `openai/codex#37027` (introduced the bound), `openai/codex#37463` (open: proposes widening it) diff --git a/docs/solutions/integrations/agent-plugins-schema-is-a-host-routing-switch.md b/docs/solutions/integrations/agent-plugins-schema-is-a-host-routing-switch.md index 67db457d9..54d0f981d 100644 --- a/docs/solutions/integrations/agent-plugins-schema-is-a-host-routing-switch.md +++ b/docs/solutions/integrations/agent-plugins-schema-is-a-host-routing-switch.md @@ -84,7 +84,7 @@ plus a set-based `OVER_BUDGET` allowlist (26 names) that is shrink-only: no new Both hosts branch on the `$schema` prefix, not on the plugin's actual conformance: -- Codex `find_plugin_manifest_path` (codex-rs/utils/plugins/src/plugin_namespace.rs, openai/codex#37027) prefers root `plugin.json` over `.codex-plugin/plugin.json` when `$schema` starts with `https://agent-plugins.org/schemas/`, classifying the package as an Agent Plugin; `build_skill_injections` (codex-rs/core-skills/src/injection.rs) then applies `take_bytes_at_char_boundary(contents, MAX_SKILL_PROMPT_BYTES)`, with the constant (`8_000`) defined in openai/codex `codex-rs/core-skills/src/lib.rs`. Legacy manifests are exempt. +- Codex `find_plugin_manifest_path` (codex-rs/utils/plugins/src/plugin_namespace.rs, openai/codex#37027) prefers root `plugin.json` over `.codex-plugin/plugin.json` when `$schema` starts with `https://agent-plugins.org/schemas/`, classifying the package as an Agent Plugin; the skill body is then truncated to `MAX_SKILL_PROMPT_BYTES` (`8_000`) before injection. Legacy manifests are exempt. **Upstream source paths re-verified 2026-08-21:** the constant and the truncation both live in `codex-rs/ext/skills/src/render.rs` (constant at line 19; `truncate_main_prompt_contents` -> `truncate_utf8_to_bytes` -> `take_bytes_at_char_boundary`, called from `host_prompt.rs` and `extension.rs`). This doc previously cited `codex-rs/core-skills/src/lib.rs` and `injection.rs`, which were accurate when written: the `core-skills` crate no longer exists and `build_skill_injections` is gone. Note `render.rs` also defines a same-valued `DEFAULT_SKILL_METADATA_CHAR_BUDGET` that bounds the skills *listing*, not any body -- see `docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md` for why that collision keeps regenerating the "8KB is a spec requirement" folklore, and `docs/specs/agent-plugins.md` for the full provenance table. - omp `agent-plugin-format.ts` (~line 179) routes the same prefix to a provider whose `validateAgentSkillFrontmatter` (~124-159) allows only `{name, description, license, compatibility, metadata, allowed-tools}` with string `allowed-tools`. 28 skills carry `argument-hint`, 13 `disable-model-invocation`, and `ce-proof` and `ce-product-pulse` use a list `allowed-tools`. Removing the `$schema` puts both hosts on their lenient legacy paths (`.codex-plugin/plugin.json` for Codex; omp's legacy provider loads all 33), while `agy`, Grok, and Claude Code never depended on the field. The unconditional assertion encodes the real condition (the routing switch itself is harmful given this plugin's shape) instead of a predicate over skill contents that reviewers could keep refining. diff --git a/docs/solutions/skill-design/size-driven-skill-restructure.md b/docs/solutions/skill-design/size-driven-skill-restructure.md index 63f895662..362f505c2 100644 --- a/docs/solutions/skill-design/size-driven-skill-restructure.md +++ b/docs/solutions/skill-design/size-driven-skill-restructure.md @@ -46,6 +46,8 @@ Two things landed together: 2. **The 8KB cap.** `docs/specs/agent-plugins.md` keeps the root manifest schema-less indefinitely, so no shipping surface reaches the cap today and it stays a ratchet: the goal is a body small enough that a conformant Agent Plugins package could ever be emitted, and `tests/codex-skill-prompt-budget.test.ts` sweeps skills under it one at a time. What is no longer hypothetical is what the truncation does when that path is taken. Forced onto Codex 0.147's Agent Plugins path, `lfg`'s 28,520-byte body was cut at 8,000 bytes inside its routing-carrier section, so steps 1-10 were never injected (#1479). That cell still opened a PR — but by reconstructing the pipeline out of the installed child skills, because the eval harness had the whole plugin installed, which is a confound rather than a demonstration that the truncation is survivable. **So "the cap does not bite today" is a true statement about the shipping path and not an argument that a body over it is fine.** Say in the PR which of the two you measured, because it changes how you size the eval. 3. **The cap is a ceiling, not the target.** Every byte of a SKILL.md body is charged to the context window on every invocation, on every host, whether or not the host truncates. A skill that already fits under 8000 bytes still deserves the same pass: the smallest body that carries the outcome/done, boundaries, ordering invariant, stop classes, and point-of-use pointers, with everything else in references loaded when a step needs it. Trimming to just under the line is the easy path and is not the goal. Some skills genuinely resist extraction — a body whose every block must fire without a read; when that is the finding, keep the body and record in the PR what was tried and why each remaining block must stay always-loaded, so the next sweep does not re-derive it. Never shrink by dropping an invariant; relocate it. + **Where the number comes from, and the second bound.** 8000 is Codex's `MAX_SKILL_PROMPT_BYTES`, not an Agent Plugins requirement -- the spec has no size limit of any kind, and the Agent Skills spec it defers to constrains only frontmatter. A separate host bound applies on Claude Code regardless of manifest: auto-compaction re-attaches each invoked skill keeping only its **first 5,000 tokens**, within a 25,000-token combined budget filled from the most recently invoked, so older skills drop entirely ([docs](https://code.claude.com/docs/en/skills)). At roughly 20KB that is looser than the 8000-byte ratchet, so clearing the ratchet clears it too. The operational consequence for this procedure: **both truncations keep the start of the file**, so body ordering is load-bearing -- put what must survive above what may be cut, and never let a stop class or boundary rule sit below a long routing block. `docs/specs/agent-plugins.md` carries the full provenance table. + ## What worked, in order 1. **Read the whole skill and the incident evidence first**, and write the analysis before touching anything (what went wrong; skill vs. operator; correct-but-buried rules; smallest fixes). The analysis found the description problem, the missing envelope exclusion ("unrequested branch update / CI-resetting push"), and the missing upward clause — none of which a size-only pass would have produced. diff --git a/docs/specs/agent-plugins.md b/docs/specs/agent-plugins.md index 060f00f1c..6c26cd48c 100644 --- a/docs/specs/agent-plugins.md +++ b/docs/specs/agent-plugins.md @@ -16,6 +16,21 @@ Layout already matches the portable package shape: root manifest + `skills/= 0.147), - * which is why tests/codex-skill-prompt-budget.test.ts ratchets every skill - * under it as a standing goal — that test owns the size gate, not this one. + * not a universal body cap, and it is not from the Agent Plugins spec, + * which imposes no size limit of any kind. On the legacy (schema-less) + * path Codex reads full SKILL.md bodies from disk on demand and only + * budgets the injected skills metadata LIST (see + * tests/real-plugin-conversion.test.ts). It IS a real 8000-byte body + * truncation on the Agent Plugins path (Codex >= 0.147), which is why + * tests/codex-skill-prompt-budget.test.ts ratchets every skill under it as + * a standing goal -- that test owns the size gate, not this one, and its + * header carries the full provenance including Claude Code's separate + * 5,000-token auto-compaction bound. * * 4. PLATFORM-VARIABLE FALLBACK (AGENTS.md "Platform-Specific Variables in * Skills"): skill markdown using harness variables (${CLAUDE_*}, From d7f24ee36c9657fcb5e00af108ed5e168d18f21f Mon Sep 17 00:00:00 2001 From: Trevin Chow Date: Fri, 21 Aug 2026 23:18:22 -0700 Subject: [PATCH 2/5] Address PR review feedback (#1511) Separate the two halves of the Claude Code compaction bound. The per-skill 5,000-token cap is subsumed by the 8000-byte ratchet; the combined 25,000-token cap is not, and no per-file check can express it. At ~4 bytes/token an at-cap body is ~2,000 tokens, so ~12 fully compliant skills exhaust the combined budget and the oldest are then dropped entirely. tests/codex-skill-prompt-budget.test.ts sizes each SKILL.md independently, so a green run there is not evidence the aggregate holds. Documented as a known, deliberate gap rather than a covered case. Corrected at every site this PR introduced the claim: docs/specs/agent-plugins.md (including the re-verify trigger), the test header, size-driven-skill-restructure.md, verify-externally-attributed-constraints-at-the-source.md, and the CONCEPTS.md Host prompt budget entry. Claude-Session: https://claude.ai/code/session_013DVEGTARyLzqxxzvzoVXEA --- CONCEPTS.md | 2 +- ...ternally-attributed-constraints-at-the-source.md | 4 ++-- .../skill-design/size-driven-skill-restructure.md | 2 +- docs/specs/agent-plugins.md | 8 ++++++-- tests/codex-skill-prompt-budget.test.ts | 13 +++++++++---- 5 files changed, 19 insertions(+), 10 deletions(-) diff --git a/CONCEPTS.md b/CONCEPTS.md index efacdef15..fe136197e 100644 --- a/CONCEPTS.md +++ b/CONCEPTS.md @@ -102,7 +102,7 @@ The part of a Skill that must hold without any reference loaded: the result it p ### Host prompt budget The ceiling a specific agent host places on how much of a Skill's body it will keep in the model-visible prompt, enforced by that host's own loader rather than by any plugin or skill specification. Each host sets its own and reaches it by a different route — one may truncate the body outright and only for packages declaring a particular manifest shape, another may re-attach a shortened copy of each invoked skill after summarizing a long conversation — so a body that survives intact on one host can silently lose its tail on another. -Every known truncation keeps the beginning of the body and discards the rest, and none of them reports an error. That is what makes ordering load-bearing: what must survive belongs above what may be cut, and a stop class or boundary rule sitting below a long routing block can disappear while every mechanical check still passes. A repository that ratchets body size sets that ratchet from the tightest bound among the hosts it ships against — a scoped engineering constraint whose owner and scope are re-verified at the source, not a conformance requirement of any specification. Load stub and Phase-loaded kernel are the two shapes content takes when it moves out of the body to fit. +Every known truncation keeps the beginning of the body and discards the rest, and none of them reports an error. That is what makes ordering load-bearing: what must survive belongs above what may be cut, and a stop class or boundary rule sitting below a long routing block can disappear while every mechanical check still passes. A repository that ratchets body size sets that ratchet from the tightest bound among the hosts it ships against — a scoped engineering constraint whose owner and scope are re-verified at the source, not a conformance requirement of any specification. Such a ratchet bounds only the per-skill budgets: where a host also caps the *combined* size of everything invoked in one session, that aggregate is a separate invariant no per-file check can express, so a green per-file gate is not evidence it holds. Load stub and Phase-loaded kernel are the two shapes content takes when it moves out of the body to fit. ### Load stub The inline remnant left in a Skill when load-bearing content moves to a reference file: a load instruction that names what the reference contains and the failure mode of skipping it, while keeping no detail an agent could improvise from — making the load structurally necessary rather than advisory. diff --git a/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md b/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md index 301f055bb..205e1fd50 100644 --- a/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md +++ b/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md @@ -52,7 +52,7 @@ A number in a test needs three things beside it: which component owns it, what s **3. Distinguish the bound from the standard, because the difference changes decisions.** -A limit believed to come from a standard is treated as immovable and universal. A limit known to come from one host's renderer is a scoped engineering constraint you can reason about: it may apply on only one code path, another host may impose a different one, and the two can be compared. Here the practical consequences were concrete — the repo's ratchet turned out to be the *tighter* of two bounds and to subsume the other, which is a fact you can only state once both are attributed. +A limit believed to come from a standard is treated as immovable and universal. A limit known to come from one host's renderer is a scoped engineering constraint you can reason about: it may apply on only one code path, another host may impose a different one, and the two can be compared. Here the practical consequences were concrete: once both bounds were attributed, the repo's ratchet turned out to subsume one of them (the other host's *per-skill* cap) while leaving that host's *aggregate* cap unbounded — a distinction invisible while the number was believed to come from a spec. **4. When several unrelated constants share a number, name all of them.** @@ -66,7 +66,7 @@ So: give any claim about upstream a `file:line` and a verification date, and re- ## Why This Matters -**A misattributed constraint cannot be questioned.** "The standard requires it" ends the conversation; "this host's renderer truncates there, verified on this date" invites the next useful question. The whole repo was tuned to one host's truncation while a second host's bound went unmodeled for months. It turned out to be looser, so the existing ratchet already covered it — that was luck, not design. +**A misattributed constraint cannot be questioned.** "The standard requires it" ends the conversation; "this host's renderer truncates there, verified on this date" invites the next useful question. The whole repo was tuned to one host's truncation while a second host's bound went unmodeled for months. Attributing it split that bound in two: a per-skill cap the existing ratchet already covered by luck, and an aggregate cap across everything invoked in one session that no per-file check can express and nothing currently guards. The first was invisible because the number was unattributed; the second was invisible because nobody had asked what *shape* the bound was. **The attribution determines what compliance means.** Once the bound is known to apply only on a specific host path, shrinking a skill stops being a compliance chore with an invented authority behind it and becomes a precondition for a specific capability, with a reason a reviewer can weigh. diff --git a/docs/solutions/skill-design/size-driven-skill-restructure.md b/docs/solutions/skill-design/size-driven-skill-restructure.md index 362f505c2..512f4be54 100644 --- a/docs/solutions/skill-design/size-driven-skill-restructure.md +++ b/docs/solutions/skill-design/size-driven-skill-restructure.md @@ -46,7 +46,7 @@ Two things landed together: 2. **The 8KB cap.** `docs/specs/agent-plugins.md` keeps the root manifest schema-less indefinitely, so no shipping surface reaches the cap today and it stays a ratchet: the goal is a body small enough that a conformant Agent Plugins package could ever be emitted, and `tests/codex-skill-prompt-budget.test.ts` sweeps skills under it one at a time. What is no longer hypothetical is what the truncation does when that path is taken. Forced onto Codex 0.147's Agent Plugins path, `lfg`'s 28,520-byte body was cut at 8,000 bytes inside its routing-carrier section, so steps 1-10 were never injected (#1479). That cell still opened a PR — but by reconstructing the pipeline out of the installed child skills, because the eval harness had the whole plugin installed, which is a confound rather than a demonstration that the truncation is survivable. **So "the cap does not bite today" is a true statement about the shipping path and not an argument that a body over it is fine.** Say in the PR which of the two you measured, because it changes how you size the eval. 3. **The cap is a ceiling, not the target.** Every byte of a SKILL.md body is charged to the context window on every invocation, on every host, whether or not the host truncates. A skill that already fits under 8000 bytes still deserves the same pass: the smallest body that carries the outcome/done, boundaries, ordering invariant, stop classes, and point-of-use pointers, with everything else in references loaded when a step needs it. Trimming to just under the line is the easy path and is not the goal. Some skills genuinely resist extraction — a body whose every block must fire without a read; when that is the finding, keep the body and record in the PR what was tried and why each remaining block must stay always-loaded, so the next sweep does not re-derive it. Never shrink by dropping an invariant; relocate it. - **Where the number comes from, and the second bound.** 8000 is Codex's `MAX_SKILL_PROMPT_BYTES`, not an Agent Plugins requirement -- the spec has no size limit of any kind, and the Agent Skills spec it defers to constrains only frontmatter. A separate host bound applies on Claude Code regardless of manifest: auto-compaction re-attaches each invoked skill keeping only its **first 5,000 tokens**, within a 25,000-token combined budget filled from the most recently invoked, so older skills drop entirely ([docs](https://code.claude.com/docs/en/skills)). At roughly 20KB that is looser than the 8000-byte ratchet, so clearing the ratchet clears it too. The operational consequence for this procedure: **both truncations keep the start of the file**, so body ordering is load-bearing -- put what must survive above what may be cut, and never let a stop class or boundary rule sit below a long routing block. `docs/specs/agent-plugins.md` carries the full provenance table. + **Where the number comes from, and the second bound.** 8000 is Codex's `MAX_SKILL_PROMPT_BYTES`, not an Agent Plugins requirement -- the spec has no size limit of any kind, and the Agent Skills spec it defers to constrains only frontmatter. A separate host bound applies on Claude Code regardless of manifest: auto-compaction re-attaches each invoked skill keeping only its **first 5,000 tokens**, within a 25,000-token combined budget filled from the most recently invoked, so older skills drop entirely ([docs](https://code.claude.com/docs/en/skills)). Only the per-skill half is covered by the ratchet: at roughly 20KB it is looser than 8000 bytes, so clearing the ratchet clears it. The combined 25,000-token half is an aggregate across every skill invoked in one session -- about a dozen fully compliant skills exhaust it, and the oldest are then dropped entirely -- so no per-file ratchet bounds it and nothing currently does. The operational consequence for this procedure: **both truncations keep the start of the file**, so body ordering is load-bearing -- put what must survive above what may be cut, and never let a stop class or boundary rule sit below a long routing block. `docs/specs/agent-plugins.md` carries the full provenance table. ## What worked, in order diff --git a/docs/specs/agent-plugins.md b/docs/specs/agent-plugins.md index 6c26cd48c..2c02d08ff 100644 --- a/docs/specs/agent-plugins.md +++ b/docs/specs/agent-plugins.md @@ -27,7 +27,11 @@ So every byte bound this repo enforces comes from a **host implementation**, not | 8,000 bytes | Codex `MAX_SKILL_PROMPT_BYTES` (`codex-rs/ext/skills/src/render.rs`) | Agent Plugin skills only (a root `$schema` under the Agent Plugins prefix) | Body silently truncated at the byte boundary; tail sections never injected | | 5,000 tokens per skill / 25,000 combined | Claude Code auto-compaction ([docs](https://code.claude.com/docs/en/skills)) | Every skill on Claude Code, regardless of manifest or `$schema` | After a summary, each skill is re-attached keeping only its first 5,000 tokens; the combined budget fills from the most recently invoked, so older skills drop entirely | -The Claude Code bound is the one that applies on this repo's actual shipping path, since the root manifest stays schema-less. It is also the looser of the two: 5,000 tokens is roughly 20KB, so a body under Codex's 8,000-byte bound clears it with wide margin. That is why `tests/codex-skill-prompt-budget.test.ts` ratchets on 8,000 and there is no separate compaction gate -- the tighter bound subsumes it. Both truncations keep the **start** of the file, so ordering inside a body is load-bearing: whatever must survive belongs above whatever may be cut. +The Claude Code bound is the one that applies on this repo's actual shipping path, since the root manifest stays schema-less. **Its two halves need separating, because only one of them is covered.** The *per-skill* 5,000-token cap is subsumed by the byte ratchet: 5,000 tokens is roughly 20KB, so a body under 8,000 bytes clears it with wide margin, which is why `tests/codex-skill-prompt-budget.test.ts` ratchets on 8,000 with no separate per-skill compaction gate. + +The *combined* 25,000-token budget is **not** subsumed and is currently unguarded. It is an aggregate over every skill invoked in one session, so no per-file ratchet can bound it: at roughly 4 bytes per token an 8,000-byte body is about 2,000 tokens, and about a dozen fully compliant skills exhaust the budget. Past that, Claude Code fills from the most recently invoked and drops the oldest skills **entirely** after compaction. A session that chains several skills reaches this while every individual file passes the gate. `tests/codex-skill-prompt-budget.test.ts` checks each `SKILL.md` independently and cannot express this invariant; nothing else does either. Treat it as a known, deliberate gap rather than a covered case, and size a chained workflow with it in mind. + +Both truncations keep the **start** of the file, so ordering inside a body is load-bearing: whatever must survive belongs above whatever may be cut. **Do not attribute 8,000 to the standard.** At least three unrelated 8,000s circulate, which is why the folklore is durable: Codex's `MAX_SKILL_PROMPT_BYTES` (body truncation, the one that matters here), Codex's `DEFAULT_SKILL_METADATA_CHAR_BUDGET` (listing budget, same file, same number), and Claude Code's historical skill-listing fallback -- current Claude Code sizes that listing at 1% of the context window, tunable via `skillListingBudgetFraction` / `SLASH_COMMAND_TOOL_CHAR_BUDGET`, and caps each entry's combined `description` + `when_to_use` at 1,536 characters. None of those three is a body limit except the first. @@ -64,7 +68,7 @@ Agent Plugins discovers skills via the [Agent Skills](https://agentskills.io/spe - A strict Agent Plugins client we ship to needs conformance (then add an emitted conformant package for it — do not add `$schema` to the root) - omp adds a per-host override / lenient fallback for `$schema` packages - Codex changes `MAX_SKILL_PROMPT_BYTES` or applies it to legacy/host skills ([openai/codex#37463](https://github.com/openai/codex/issues/37463)) -- Claude Code changes the auto-compaction skill budget (5,000 tokens per skill / 25,000 combined) -- it would become the binding bound if it ever drops below Codex's 8,000 bytes +- Claude Code changes the auto-compaction skill budget (5,000 tokens per skill / 25,000 combined) -- the per-skill half would become binding if it ever drops below Codex's 8,000 bytes, and the combined half is already unguarded, so a drop there widens an existing gap rather than opening a new one - Agent Plugins leaves Working Draft / publishes a new schema version - Adding top-level fields to root `plugin.json` - A concrete Agent Plugins client is observed to skip or reject skills with Claude-only frontmatter (observed 2026-08-17: omp 17.3.5, #1411) diff --git a/tests/codex-skill-prompt-budget.test.ts b/tests/codex-skill-prompt-budget.test.ts index 9c5a60be9..10dd31cb8 100644 --- a/tests/codex-skill-prompt-budget.test.ts +++ b/tests/codex-skill-prompt-budget.test.ts @@ -22,10 +22,15 @@ import path from "node:path" * 500 lines") is explicitly a recommendation. A second, independent host bound exists and is * deliberately not gated separately here: Claude Code auto-compaction re-attaches each invoked * skill keeping only its first 5,000 tokens, within a 25,000-token combined budget filled from - * the most recently invoked (https://code.claude.com/docs/en/skills). 5,000 tokens is roughly - * 20KB, so this 8000-byte ratchet is the tighter bound and subsumes it -- revisit only if that - * ordering ever inverts. Both truncations keep the START of the file, which is why a body's - * ordering is load-bearing: what must survive goes above what may be cut. + * the most recently invoked (https://code.claude.com/docs/en/skills). Only its PER-SKILL half is + * covered here: 5,000 tokens is roughly 20KB, so this 8000-byte ratchet subsumes it -- revisit + * only if that ordering ever inverts. Its COMBINED 25,000-token half is an aggregate over every + * skill invoked in one session, which a per-file check cannot express: at ~4 bytes/token an + * 8000-byte body is ~2000 tokens, so ~12 fully compliant skills exhaust it and the oldest are + * then dropped entirely. That invariant is deliberately UNGUARDED -- this file sizes each + * SKILL.md independently and no other test covers the aggregate. Do not read a green run here as + * proof the compaction budget is safe. Both truncations keep the START of the file, which is why + * a body's ordering is load-bearing: what must survive goes above what may be cut. */ const CODEX_MAX_SKILL_PROMPT_BYTES = 8_000 const AGENT_PLUGINS_SCHEMA_PREFIX = "https://agent-plugins.org/schemas/" From fa165d745649c654793000c0f88044a4adfea179 Mon Sep 17 00:00:00 2001 From: Trevin Chow Date: Fri, 21 Aug 2026 23:20:02 -0700 Subject: [PATCH 3/5] docs(agent-plugins): record the combined-budget gap as an accepted risk The 25,000-token aggregate is a property of a session's invocation sequence, not of the repository, so no repo-side check can establish it with certainty. A guard would only ever cover a declared chain. Bodies are now small enough that a chained workflow keeps real headroom, and the start-preserving truncation is handled by ordering rather than size, so longer-horizon chaining is left to the host. Recorded so the gap is not reopened as backlog. Claude-Session: https://claude.ai/code/session_013DVEGTARyLzqxxzvzoVXEA --- docs/specs/agent-plugins.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/specs/agent-plugins.md b/docs/specs/agent-plugins.md index 2c02d08ff..63bd03821 100644 --- a/docs/specs/agent-plugins.md +++ b/docs/specs/agent-plugins.md @@ -29,7 +29,9 @@ So every byte bound this repo enforces comes from a **host implementation**, not The Claude Code bound is the one that applies on this repo's actual shipping path, since the root manifest stays schema-less. **Its two halves need separating, because only one of them is covered.** The *per-skill* 5,000-token cap is subsumed by the byte ratchet: 5,000 tokens is roughly 20KB, so a body under 8,000 bytes clears it with wide margin, which is why `tests/codex-skill-prompt-budget.test.ts` ratchets on 8,000 with no separate per-skill compaction gate. -The *combined* 25,000-token budget is **not** subsumed and is currently unguarded. It is an aggregate over every skill invoked in one session, so no per-file ratchet can bound it: at roughly 4 bytes per token an 8,000-byte body is about 2,000 tokens, and about a dozen fully compliant skills exhaust the budget. Past that, Claude Code fills from the most recently invoked and drops the oldest skills **entirely** after compaction. A session that chains several skills reaches this while every individual file passes the gate. `tests/codex-skill-prompt-budget.test.ts` checks each `SKILL.md` independently and cannot express this invariant; nothing else does either. Treat it as a known, deliberate gap rather than a covered case, and size a chained workflow with it in mind. +The *combined* 25,000-token budget is **not** subsumed and is currently unguarded. It is an aggregate over every skill invoked in one session, so no per-file ratchet can bound it: at roughly 4 bytes per token an 8,000-byte body is about 2,000 tokens, and about a dozen fully compliant skills exhaust the budget. Past that, Claude Code fills from the most recently invoked and drops the oldest skills **entirely** after compaction. A session that chains several skills reaches this while every individual file passes the gate. `tests/codex-skill-prompt-budget.test.ts` checks each `SKILL.md` independently and cannot express this invariant; nothing else does either. + +**Posture (decided 2026-08-21): this is an accepted risk, not a backlog item.** The aggregate is a property of a session's invocation sequence rather than of the repository, so no repo-side check can establish it with certainty; a guard is expressible only for a *declared* chain and would still say nothing about arbitrary sessions. The mitigation that does work is already standing: bodies are small (the sweep put nearly all of them under 8,000 bytes, so a chained workflow keeps real headroom) and the start-preserving truncation is handled by ordering, not by size. Longer-horizon chaining is left to the host, which is also where re-invoking a skill after compaction to restore it belongs. Do not open work to close this gap; size a chained workflow with it in mind and revisit only if the numbers above move. Both truncations keep the **start** of the file, so ordering inside a body is load-bearing: whatever must survive belongs above whatever may be cut. From a08ed61e41a4e0f9d6abb1f04719bd340f040f5a Mon Sep 17 00:00:00 2001 From: Trevin Chow Date: Fri, 21 Aug 2026 23:20:19 -0700 Subject: [PATCH 4/5] docs(agent-plugins): drop the accepted-risk paragraph Stating that the unguarded aggregate is accepted rather than backlog is implicit; the factual note that nothing guards it is what the review asked for and is enough on its own. Claude-Session: https://claude.ai/code/session_013DVEGTARyLzqxxzvzoVXEA --- docs/specs/agent-plugins.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/specs/agent-plugins.md b/docs/specs/agent-plugins.md index 63bd03821..5772c1ec8 100644 --- a/docs/specs/agent-plugins.md +++ b/docs/specs/agent-plugins.md @@ -29,9 +29,7 @@ So every byte bound this repo enforces comes from a **host implementation**, not The Claude Code bound is the one that applies on this repo's actual shipping path, since the root manifest stays schema-less. **Its two halves need separating, because only one of them is covered.** The *per-skill* 5,000-token cap is subsumed by the byte ratchet: 5,000 tokens is roughly 20KB, so a body under 8,000 bytes clears it with wide margin, which is why `tests/codex-skill-prompt-budget.test.ts` ratchets on 8,000 with no separate per-skill compaction gate. -The *combined* 25,000-token budget is **not** subsumed and is currently unguarded. It is an aggregate over every skill invoked in one session, so no per-file ratchet can bound it: at roughly 4 bytes per token an 8,000-byte body is about 2,000 tokens, and about a dozen fully compliant skills exhaust the budget. Past that, Claude Code fills from the most recently invoked and drops the oldest skills **entirely** after compaction. A session that chains several skills reaches this while every individual file passes the gate. `tests/codex-skill-prompt-budget.test.ts` checks each `SKILL.md` independently and cannot express this invariant; nothing else does either. - -**Posture (decided 2026-08-21): this is an accepted risk, not a backlog item.** The aggregate is a property of a session's invocation sequence rather than of the repository, so no repo-side check can establish it with certainty; a guard is expressible only for a *declared* chain and would still say nothing about arbitrary sessions. The mitigation that does work is already standing: bodies are small (the sweep put nearly all of them under 8,000 bytes, so a chained workflow keeps real headroom) and the start-preserving truncation is handled by ordering, not by size. Longer-horizon chaining is left to the host, which is also where re-invoking a skill after compaction to restore it belongs. Do not open work to close this gap; size a chained workflow with it in mind and revisit only if the numbers above move. +The *combined* 25,000-token budget is **not** subsumed and is currently unguarded. It is an aggregate over every skill invoked in one session, so no per-file ratchet can bound it: at roughly 4 bytes per token an 8,000-byte body is about 2,000 tokens, and about a dozen fully compliant skills exhaust the budget. Past that, Claude Code fills from the most recently invoked and drops the oldest skills **entirely** after compaction. A session that chains several skills reaches this while every individual file passes the gate. `tests/codex-skill-prompt-budget.test.ts` checks each `SKILL.md` independently and cannot express this invariant; nothing else does either. Size a chained workflow with it in mind. Both truncations keep the **start** of the file, so ordering inside a body is load-bearing: whatever must survive belongs above whatever may be cut. From a823da2750b7c1b4485cd13389908d7ed47f7fbd Mon Sep 17 00:00:00 2001 From: Trevin Chow Date: Fri, 21 Aug 2026 23:26:29 -0700 Subject: [PATCH 5/5] Address PR review feedback (#1511) State the per-skill compaction relationship as an approximation, not a guarantee. A byte count never proves a token count: bytes per token is content-dependent, so an 8000-byte body is ~2000 tokens at ordinary prose density but the ratchet cannot prove a 5,000-token bound. Breaching 5,000 tokens inside 8000 bytes would take ~1.6 bytes per token, which Markdown prose does not reach, so the margin is wide -- but it is a margin, and a token-dense body erodes it. Took the review's documentation branch rather than adding a token-count guard. Corrected at all four sites carrying the claim: the spec section, the test header, size-driven-skill-restructure.md, and the convention doc. Claude-Session: https://claude.ai/code/session_013DVEGTARyLzqxxzvzoVXEA --- ...erify-externally-attributed-constraints-at-the-source.md | 2 +- .../solutions/skill-design/size-driven-skill-restructure.md | 2 +- docs/specs/agent-plugins.md | 2 +- tests/codex-skill-prompt-budget.test.ts | 6 ++++-- 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md b/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md index 205e1fd50..21da53f59 100644 --- a/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md +++ b/docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md @@ -52,7 +52,7 @@ A number in a test needs three things beside it: which component owns it, what s **3. Distinguish the bound from the standard, because the difference changes decisions.** -A limit believed to come from a standard is treated as immovable and universal. A limit known to come from one host's renderer is a scoped engineering constraint you can reason about: it may apply on only one code path, another host may impose a different one, and the two can be compared. Here the practical consequences were concrete: once both bounds were attributed, the repo's ratchet turned out to subsume one of them (the other host's *per-skill* cap) while leaving that host's *aggregate* cap unbounded — a distinction invisible while the number was believed to come from a spec. +A limit believed to come from a standard is treated as immovable and universal. A limit known to come from one host's renderer is a scoped engineering constraint you can reason about: it may apply on only one code path, another host may impose a different one, and the two can be compared. Here the practical consequences were concrete: once both bounds were attributed, the repo's ratchet turned out to approximate one of them (the other host's *per-skill* cap, in different units, so with a margin rather than a proof) while leaving that host's *aggregate* cap unbounded — a distinction invisible while the number was believed to come from a spec. **4. When several unrelated constants share a number, name all of them.** diff --git a/docs/solutions/skill-design/size-driven-skill-restructure.md b/docs/solutions/skill-design/size-driven-skill-restructure.md index 512f4be54..28cff26b4 100644 --- a/docs/solutions/skill-design/size-driven-skill-restructure.md +++ b/docs/solutions/skill-design/size-driven-skill-restructure.md @@ -46,7 +46,7 @@ Two things landed together: 2. **The 8KB cap.** `docs/specs/agent-plugins.md` keeps the root manifest schema-less indefinitely, so no shipping surface reaches the cap today and it stays a ratchet: the goal is a body small enough that a conformant Agent Plugins package could ever be emitted, and `tests/codex-skill-prompt-budget.test.ts` sweeps skills under it one at a time. What is no longer hypothetical is what the truncation does when that path is taken. Forced onto Codex 0.147's Agent Plugins path, `lfg`'s 28,520-byte body was cut at 8,000 bytes inside its routing-carrier section, so steps 1-10 were never injected (#1479). That cell still opened a PR — but by reconstructing the pipeline out of the installed child skills, because the eval harness had the whole plugin installed, which is a confound rather than a demonstration that the truncation is survivable. **So "the cap does not bite today" is a true statement about the shipping path and not an argument that a body over it is fine.** Say in the PR which of the two you measured, because it changes how you size the eval. 3. **The cap is a ceiling, not the target.** Every byte of a SKILL.md body is charged to the context window on every invocation, on every host, whether or not the host truncates. A skill that already fits under 8000 bytes still deserves the same pass: the smallest body that carries the outcome/done, boundaries, ordering invariant, stop classes, and point-of-use pointers, with everything else in references loaded when a step needs it. Trimming to just under the line is the easy path and is not the goal. Some skills genuinely resist extraction — a body whose every block must fire without a read; when that is the finding, keep the body and record in the PR what was tried and why each remaining block must stay always-loaded, so the next sweep does not re-derive it. Never shrink by dropping an invariant; relocate it. - **Where the number comes from, and the second bound.** 8000 is Codex's `MAX_SKILL_PROMPT_BYTES`, not an Agent Plugins requirement -- the spec has no size limit of any kind, and the Agent Skills spec it defers to constrains only frontmatter. A separate host bound applies on Claude Code regardless of manifest: auto-compaction re-attaches each invoked skill keeping only its **first 5,000 tokens**, within a 25,000-token combined budget filled from the most recently invoked, so older skills drop entirely ([docs](https://code.claude.com/docs/en/skills)). Only the per-skill half is covered by the ratchet: at roughly 20KB it is looser than 8000 bytes, so clearing the ratchet clears it. The combined 25,000-token half is an aggregate across every skill invoked in one session -- about a dozen fully compliant skills exhaust it, and the oldest are then dropped entirely -- so no per-file ratchet bounds it and nothing currently does. The operational consequence for this procedure: **both truncations keep the start of the file**, so body ordering is load-bearing -- put what must survive above what may be cut, and never let a stop class or boundary rule sit below a long routing block. `docs/specs/agent-plugins.md` carries the full provenance table. + **Where the number comes from, and the second bound.** 8000 is Codex's `MAX_SKILL_PROMPT_BYTES`, not an Agent Plugins requirement -- the spec has no size limit of any kind, and the Agent Skills spec it defers to constrains only frontmatter. A separate host bound applies on Claude Code regardless of manifest: auto-compaction re-attaches each invoked skill keeping only its **first 5,000 tokens**, within a 25,000-token combined budget filled from the most recently invoked, so older skills drop entirely ([docs](https://code.claude.com/docs/en/skills)). Only the per-skill half is approximated by the ratchet, and a byte count never proves a token count: 8000 bytes is ~2000 tokens at ordinary prose density, so the margin is wide but is not a guarantee, and a token-dense body erodes it. The combined 25,000-token half is an aggregate across every skill invoked in one session -- about a dozen fully compliant skills exhaust it, and the oldest are then dropped entirely -- so no per-file ratchet bounds it and nothing currently does. The operational consequence for this procedure: **both truncations keep the start of the file**, so body ordering is load-bearing -- put what must survive above what may be cut, and never let a stop class or boundary rule sit below a long routing block. `docs/specs/agent-plugins.md` carries the full provenance table. ## What worked, in order diff --git a/docs/specs/agent-plugins.md b/docs/specs/agent-plugins.md index 5772c1ec8..69c9a8014 100644 --- a/docs/specs/agent-plugins.md +++ b/docs/specs/agent-plugins.md @@ -27,7 +27,7 @@ So every byte bound this repo enforces comes from a **host implementation**, not | 8,000 bytes | Codex `MAX_SKILL_PROMPT_BYTES` (`codex-rs/ext/skills/src/render.rs`) | Agent Plugin skills only (a root `$schema` under the Agent Plugins prefix) | Body silently truncated at the byte boundary; tail sections never injected | | 5,000 tokens per skill / 25,000 combined | Claude Code auto-compaction ([docs](https://code.claude.com/docs/en/skills)) | Every skill on Claude Code, regardless of manifest or `$schema` | After a summary, each skill is re-attached keeping only its first 5,000 tokens; the combined budget fills from the most recently invoked, so older skills drop entirely | -The Claude Code bound is the one that applies on this repo's actual shipping path, since the root manifest stays schema-less. **Its two halves need separating, because only one of them is covered.** The *per-skill* 5,000-token cap is subsumed by the byte ratchet: 5,000 tokens is roughly 20KB, so a body under 8,000 bytes clears it with wide margin, which is why `tests/codex-skill-prompt-budget.test.ts` ratchets on 8,000 with no separate per-skill compaction gate. +The Claude Code bound is the one that applies on this repo's actual shipping path, since the root manifest stays schema-less. **Its two halves need separating, because only one of them is covered.** The *per-skill* 5,000-token cap is covered by the byte ratchet only as an **approximation, not a guarantee**. Bytes per token is content-dependent, so no byte count proves a token count: 8,000 bytes is about 2,000 tokens at ordinary prose density, and breaching 5,000 tokens inside 8,000 bytes would take roughly 1.6 bytes per token, which Markdown prose does not reach. The margin is wide enough that `tests/codex-skill-prompt-budget.test.ts` ratchets on 8,000 with no separate per-skill token gate — but it is a margin, not a proof, and a token-dense body erodes it. The *combined* 25,000-token budget is **not** subsumed and is currently unguarded. It is an aggregate over every skill invoked in one session, so no per-file ratchet can bound it: at roughly 4 bytes per token an 8,000-byte body is about 2,000 tokens, and about a dozen fully compliant skills exhaust the budget. Past that, Claude Code fills from the most recently invoked and drops the oldest skills **entirely** after compaction. A session that chains several skills reaches this while every individual file passes the gate. `tests/codex-skill-prompt-budget.test.ts` checks each `SKILL.md` independently and cannot express this invariant; nothing else does either. Size a chained workflow with it in mind. diff --git a/tests/codex-skill-prompt-budget.test.ts b/tests/codex-skill-prompt-budget.test.ts index 10dd31cb8..c9074bb0c 100644 --- a/tests/codex-skill-prompt-budget.test.ts +++ b/tests/codex-skill-prompt-budget.test.ts @@ -23,8 +23,10 @@ import path from "node:path" * deliberately not gated separately here: Claude Code auto-compaction re-attaches each invoked * skill keeping only its first 5,000 tokens, within a 25,000-token combined budget filled from * the most recently invoked (https://code.claude.com/docs/en/skills). Only its PER-SKILL half is - * covered here: 5,000 tokens is roughly 20KB, so this 8000-byte ratchet subsumes it -- revisit - * only if that ordering ever inverts. Its COMBINED 25,000-token half is an aggregate over every + * approximated here, and a byte count never proves a token count: 8000 bytes is ~2000 tokens at + * ordinary prose density, and breaching 5,000 tokens within 8000 bytes would take ~1.6 bytes per + * token, which Markdown prose does not reach. Treat that as a wide margin, not a guarantee -- a + * token-dense body erodes it, and a green run here is not a token-bound proof. Its COMBINED 25,000-token half is an aggregate over every * skill invoked in one session, which a per-file check cannot express: at ~4 bytes/token an * 8000-byte body is ~2000 tokens, so ~12 fully compliant skills exhaust it and the oldest are * then dropped entirely. That invariant is deliberately UNGUARDED -- this file sizes each