Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CONCEPTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,11 @@ A bulk evidence artifact — verbatim quotes with source pointers, gathered by a
### Outcome spine
The part of a Skill that must hold without any reference loaded: the result it produces and who consumes it next, the done condition, the safe failure direction, and the facts the agent cannot derive from the repository in front of it. Everything else in the skill is protocol or judgment attached to that spine, and a block that cannot name its spine is restated before it is edited.

### Host prompt budget
The ceiling a specific agent host places on how much of a Skill's body it will keep in the model-visible prompt, enforced by that host's own loader rather than by any plugin or skill specification. Each host sets its own and reaches it by a different route — one may truncate the body outright and only for packages declaring a particular manifest shape, another may re-attach a shortened copy of each invoked skill after summarizing a long conversation — so a body that survives intact on one host can silently lose its tail on another.

Every known truncation keeps the beginning of the body and discards the rest, and none of them reports an error. That is what makes ordering load-bearing: what must survive belongs above what may be cut, and a stop class or boundary rule sitting below a long routing block can disappear while every mechanical check still passes. A repository that ratchets body size sets that ratchet from the tightest bound among the hosts it ships against — a scoped engineering constraint whose owner and scope are re-verified at the source, not a conformance requirement of any specification. Such a ratchet bounds only the per-skill budgets: where a host also caps the *combined* size of everything invoked in one session, that aggregate is a separate invariant no per-file check can express, so a green per-file gate is not evidence it holds. Load stub and Phase-loaded kernel are the two shapes content takes when it moves out of the body to fit.

### Load stub
The inline remnant left in a Skill when load-bearing content moves to a reference file: a load instruction that names what the reference contains and the failure mode of skipping it, while keeping no detail an agent could improvise from — making the load structurally necessary rather than advisory.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
---
title: "Verify an externally-attributed constraint against the spec text and the implementing source before encoding it"
date: 2026-08-21
category: conventions
module: "skills (SKILL.md size ratchet) and tests/codex-skill-prompt-budget.test.ts"
problem_type: convention
component: tooling
severity: medium
applies_when:
- "A test, lint rule, or convention enforces a number attributed to an external standard, spec, or platform"
- "The attribution carries no file-and-line citation into the spec text or the implementing source"
- "A review comment or another agent asserts \"the spec requires X\" and you are about to design around X"
- "Tightening, loosening, or justifying such a gate"
- "A repo comment makes a negative claim about upstream (\"no such constant exists\")"
tags: [provenance, primary-sources, agent-plugins, agent-skills-spec, codex, verification, stale-claims]
---

# Verify an externally-attributed constraint against the spec text and the implementing source before encoding it

## Context

This repo enforces an 8,000-byte ceiling on every `SKILL.md` body. Around that ratchet a belief had settled: that 8,000 bytes is a requirement of the **Agent Plugins standard**. Contributing agents repeated it in review, and it leaked into repo comments as fact.

On 2026-08-21 the claim was checked against primary sources and found false. The Agent Plugins spec has no size limit of any kind; the real 8,000 is one host's implementation constant. A second, differently-shaped bound on another host had gone entirely unmodeled.

The full provenance — which bound comes from which host, what each one truncates, and the invalidation triggers — is owned by `docs/specs/agent-plugins.md` under "Skill body size: what actually constrains it". **This doc does not restate it.** What is captured here is the part that generalizes past skill sizing: how the question got settled, and the failure mode that let the wrong answer stand.

This is the sibling of `antigravity-target-empirical-format-verification.md` ("verify a new target's plugin format against the CLI binary, not its docs"). Same principle, different artifact: there the docs were unreliable, here the *ecosystem consensus* was.

## Guidance

**1. Settle a claimed external constraint at the source, not through what people say about it.**

Two kinds of primary source answer nearly every such question, and both are usually one command away:

```bash
# the spec text — is the constraint even in there?
gh api repos/agentplugins/agent-plugins-spec/contents/spec/1.0.0.md --jq .content | base64 -d | grep -inE 'bytes|size|truncat|8000'
gh api repos/agentplugins/agent-plugins-spec/contents/spec/1.1.0.md --jq .content | base64 -d | grep -inE 'bytes|size|truncat|8000'

# the implementing source — where does the number actually live?
gh api repos/openai/codex/contents/codex-rs/ext/skills/src/render.rs --jq .content | base64 -d | grep -n 'MAX_SKILL_PROMPT_BYTES\|CHAR_BUDGET'
```

Check **every published version** of a spec, not just the one being cited — a constraint absent from `1.0.0` and present in `1.1.0` is a different answer than "not in the spec."

Web search is the wrong instrument here. It returned confident, unsourced restatements of the same folklore, and one result asserted that no such limit could be found anywhere. A third-party blog restated a second host's numbers correctly, but only that host's own documentation could confirm them. Secondary sources reproduce consensus; the question was whether the consensus was true.

**2. Record provenance next to the enforcement, with a date.**

A number in a test needs three things beside it: which component owns it, what scope it applies to, and when that was last verified. Without them the number is unfalsifiable in practice — nobody re-checks it, nobody scopes it, and nobody notices when the implementing host moves it.

**3. Distinguish the bound from the standard, because the difference changes decisions.**

A limit believed to come from a standard is treated as immovable and universal. A limit known to come from one host's renderer is a scoped engineering constraint you can reason about: it may apply on only one code path, another host may impose a different one, and the two can be compared. Here the practical consequences were concrete: once both bounds were attributed, the repo's ratchet turned out to approximate one of them (the other host's *per-skill* cap, in different units, so with a margin rather than a proof) while leaving that host's *aggregate* cap unbounded — a distinction invisible while the number was believed to come from a spec.

**4. When several unrelated constants share a number, name all of them.**

Three unrelated 8,000s circulate in this ecosystem. That is why the folklore is self-reinforcing: any contributor who greps one of them "confirms" the belief. Recording all three with their distinct scopes is what stops the next agent from re-deriving the wrong conclusion from a correct grep.

**5. Treat a negative claim about upstream as perishable.**

A repo comment asserted that the body constant "is not in the Codex source." That was true when written and silently became false when upstream added it. Negative claims about someone else's codebase age worse than positive ones — a positive claim breaks loudly when the path moves, while a negative one just quietly stops being true.

So: give any claim about upstream a `file:line` and a verification date, and re-check it when you touch the surrounding rule. `tests/real-plugin-conversion.test.ts` and `docs/solutions/integrations/agent-plugins-schema-is-a-host-routing-switch.md` both carried stale upstream source paths found this way.

## Why This Matters

**A misattributed constraint cannot be questioned.** "The standard requires it" ends the conversation; "this host's renderer truncates there, verified on this date" invites the next useful question. The whole repo was tuned to one host's truncation while a second host's bound went unmodeled for months. Attributing it split that bound in two: a per-skill cap the existing ratchet already covered by luck, and an aggregate cap across everything invoked in one session that no per-file check can express and nothing currently guards. The first was invisible because the number was unattributed; the second was invisible because nobody had asked what *shape* the bound was.

**The attribution determines what compliance means.** Once the bound is known to apply only on a specific host path, shrinking a skill stops being a compliance chore with an invented authority behind it and becomes a precondition for a specific capability, with a reason a reviewer can weigh.

**The folklore regenerates unless the disambiguation is written down.** Correcting one comment does not help if three same-numbered constants remain undistinguished. Recording the scopes is the durable fix.

## When to Apply

- A repo test, lint rule, or convention enforces a number, cap, or format attributed to an external standard or platform, and the attribution has no citation into spec text or implementing source.
- Someone asserts "the spec requires X" and you are about to design around X.
- You are about to move such a number. Re-verify before, not after.
- A stated provenance is old enough that upstream may have moved — especially a claim that something upstream does *not* exist.

Do **not** apply the full dig to internal repo conventions with no external claim behind them, or to a number the tree already cites to a specific file and line with a recent verification date. The cost is a few commands; spend it where an external party owns the truth.

## Examples

**Before — the belief as it read in the tree:**

> The 8KB body cap comes from the Agent Plugins spec. Skills must fit under it to be conformant.

Nothing to check, nowhere to look, and wrong on three counts: not from that spec, not universal, and not the only bound.

**After — the same rule with owner, scope, and date.** `docs/specs/agent-plugins.md` now carries a provenance table naming each bound's owning component, the code path it applies to, and what the truncation actually does, plus a re-verify trigger for each. `tests/codex-skill-prompt-budget.test.ts` carries the same provenance in its header, next to the constant it enforces.

**Before — a perishable negative claim:**

> the body constant is not in the Codex source (codex-rs/core-skills has no body-size constant)

**After** — the comment names the constants that now exist, cites the file that defines them, states which code path each applies to, and stamps the verification date.

**The operational consequence that outranks the byte count.** Both known truncations keep the **start** of the file. Neither reports an error. So body ordering is load-bearing: what must survive belongs above what may be cut, and a stop class or boundary rule must never sit below a long routing block. A skill can pass every mechanical gate, sit comfortably under the line, and still lose its stop conditions on a host that truncates or compacts. No test catches that — only the ordering discipline does. `docs/solutions/skill-design/size-driven-skill-restructure.md` carries this alongside the restructuring procedure.

## Related

- `docs/specs/agent-plugins.md` — owns the provenance table for both bounds and their re-verify triggers
- `docs/solutions/skill-design/size-driven-skill-restructure.md` — the restructuring procedure and the start-of-file ordering consequence
- `docs/solutions/integrations/agent-plugins-schema-is-a-host-routing-switch.md` — why the root manifest stays schema-less, which is what scopes one of the two bounds
- `docs/solutions/conventions/antigravity-target-empirical-format-verification.md` — the same principle applied to a target's plugin format
- Repo issues #1411, #1412 (the discovery) and PR #1479 (the only measured truncation-in-practice evidence)
- Upstream `openai/codex#37027` (introduced the bound), `openai/codex#37463` (open: proposes widening it)
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ plus a set-based `OVER_BUDGET` allowlist (26 names) that is shrink-only: no new

Both hosts branch on the `$schema` prefix, not on the plugin's actual conformance:

- Codex `find_plugin_manifest_path` (codex-rs/utils/plugins/src/plugin_namespace.rs, openai/codex#37027) prefers root `plugin.json` over `.codex-plugin/plugin.json` when `$schema` starts with `https://agent-plugins.org/schemas/`, classifying the package as an Agent Plugin; `build_skill_injections` (codex-rs/core-skills/src/injection.rs) then applies `take_bytes_at_char_boundary(contents, MAX_SKILL_PROMPT_BYTES)`, with the constant (`8_000`) defined in openai/codex `codex-rs/core-skills/src/lib.rs`. Legacy manifests are exempt.
- Codex `find_plugin_manifest_path` (codex-rs/utils/plugins/src/plugin_namespace.rs, openai/codex#37027) prefers root `plugin.json` over `.codex-plugin/plugin.json` when `$schema` starts with `https://agent-plugins.org/schemas/`, classifying the package as an Agent Plugin; the skill body is then truncated to `MAX_SKILL_PROMPT_BYTES` (`8_000`) before injection. Legacy manifests are exempt. **Upstream source paths re-verified 2026-08-21:** the constant and the truncation both live in `codex-rs/ext/skills/src/render.rs` (constant at line 19; `truncate_main_prompt_contents` -> `truncate_utf8_to_bytes` -> `take_bytes_at_char_boundary`, called from `host_prompt.rs` and `extension.rs`). This doc previously cited `codex-rs/core-skills/src/lib.rs` and `injection.rs`, which were accurate when written: the `core-skills` crate no longer exists and `build_skill_injections` is gone. Note `render.rs` also defines a same-valued `DEFAULT_SKILL_METADATA_CHAR_BUDGET` that bounds the skills *listing*, not any body -- see `docs/solutions/conventions/verify-externally-attributed-constraints-at-the-source.md` for why that collision keeps regenerating the "8KB is a spec requirement" folklore, and `docs/specs/agent-plugins.md` for the full provenance table.
- omp `agent-plugin-format.ts` (~line 179) routes the same prefix to a provider whose `validateAgentSkillFrontmatter` (~124-159) allows only `{name, description, license, compatibility, metadata, allowed-tools}` with string `allowed-tools`. 28 skills carry `argument-hint`, 13 `disable-model-invocation`, and `ce-proof` and `ce-product-pulse` use a list `allowed-tools`.

Removing the `$schema` puts both hosts on their lenient legacy paths (`.codex-plugin/plugin.json` for Codex; omp's legacy provider loads all 33), while `agy`, Grok, and Claude Code never depended on the field. The unconditional assertion encodes the real condition (the routing switch itself is harmful given this plugin's shape) instead of a predicate over skill contents that reviewers could keep refining.
Expand Down
2 changes: 2 additions & 0 deletions docs/solutions/skill-design/size-driven-skill-restructure.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ Two things landed together:
2. **The 8KB cap.** `docs/specs/agent-plugins.md` keeps the root manifest schema-less indefinitely, so no shipping surface reaches the cap today and it stays a ratchet: the goal is a body small enough that a conformant Agent Plugins package could ever be emitted, and `tests/codex-skill-prompt-budget.test.ts` sweeps skills under it one at a time. What is no longer hypothetical is what the truncation does when that path is taken. Forced onto Codex 0.147's Agent Plugins path, `lfg`'s 28,520-byte body was cut at 8,000 bytes inside its routing-carrier section, so steps 1-10 were never injected (#1479). That cell still opened a PR — but by reconstructing the pipeline out of the installed child skills, because the eval harness had the whole plugin installed, which is a confound rather than a demonstration that the truncation is survivable. **So "the cap does not bite today" is a true statement about the shipping path and not an argument that a body over it is fine.** Say in the PR which of the two you measured, because it changes how you size the eval.
3. **The cap is a ceiling, not the target.** Every byte of a SKILL.md body is charged to the context window on every invocation, on every host, whether or not the host truncates. A skill that already fits under 8000 bytes still deserves the same pass: the smallest body that carries the outcome/done, boundaries, ordering invariant, stop classes, and point-of-use pointers, with everything else in references loaded when a step needs it. Trimming to just under the line is the easy path and is not the goal. Some skills genuinely resist extraction — a body whose every block must fire without a read; when that is the finding, keep the body and record in the PR what was tried and why each remaining block must stay always-loaded, so the next sweep does not re-derive it. Never shrink by dropping an invariant; relocate it.

**Where the number comes from, and the second bound.** 8000 is Codex's `MAX_SKILL_PROMPT_BYTES`, not an Agent Plugins requirement -- the spec has no size limit of any kind, and the Agent Skills spec it defers to constrains only frontmatter. A separate host bound applies on Claude Code regardless of manifest: auto-compaction re-attaches each invoked skill keeping only its **first 5,000 tokens**, within a 25,000-token combined budget filled from the most recently invoked, so older skills drop entirely ([docs](https://code.claude.com/docs/en/skills)). Only the per-skill half is approximated by the ratchet, and a byte count never proves a token count: 8000 bytes is ~2000 tokens at ordinary prose density, so the margin is wide but is not a guarantee, and a token-dense body erodes it. The combined 25,000-token half is an aggregate across every skill invoked in one session -- about a dozen fully compliant skills exhaust it, and the oldest are then dropped entirely -- so no per-file ratchet bounds it and nothing currently does. The operational consequence for this procedure: **both truncations keep the start of the file**, so body ordering is load-bearing -- put what must survive above what may be cut, and never let a stop class or boundary rule sit below a long routing block. `docs/specs/agent-plugins.md` carries the full provenance table.

## What worked, in order

1. **Read the whole skill and the incident evidence first**, and write the analysis before touching anything (what went wrong; skill vs. operator; correct-but-buried rules; smallest fixes). The analysis found the description problem, the missing envelope exclusion ("unrequested branch update / CI-resetting push"), and the missing upward clause — none of which a size-only pass would have produced.
Expand Down
Loading