Skip to content

test(schema): find out why the openai-codex path accepts schemas that broke #239 and #241 #276

Description

@sgraczyk

Context

AGENTS.md → Release checks records a measurement and an open question:

It is not a schema gate. Measured on 2026-08-04 with pi and gpt-5.5: a session accepted, and called, tools whose schemas broke #239 (no additionalProperties), #241 (required omits a key) and one that used allOf, which the OpenAI API rejects by name. So pi's openai-codex path does not apply strict validation, and a green smoke check says nothing about whether a schema is strict-legal. Why this differs from the sessions that broke #239 and #241 is not known — pi may sanitize schemas now, or the provider may have changed.

The question has no issue. It sits in prose, and the sentence that follows it sets repo policy: test/schema.test.ts and its recorded fixture are treated as the only schema gate until the reason is understood.

This matters because the two candidate explanations lead to opposite conclusions:

  • pi sanitizes the schema before the request. Then a bad schema in this package is repaired downstream, the strict rules encoded in test/schema.test.ts are stricter than reality, and the manual smoke check is measuring pi, not us. It also means a future pi release that drops the sanitizer re-opens Tool schemas omit additionalProperties: false, breaking every session on OpenAI/Codex providers #239 with no warning here.
  • The provider changed what it accepts. Then the recorded test/strict-contract.json fixture is stale the moment that happens again, and bun run probe:strict is the only thing that would notice.

Three tool schemas reached users through this gap already (#239, #240, #241), so the cost of guessing is known.

Change

Find out which explanation holds, then write the answer where the claim lives.

  1. Read the openai-codex provider path in pi (packages/coding-agent) and find whether it transforms tool schemas before the request. If it does, name the function and the version that introduced it.
  2. If pi sanitizes: record what it repairs and what it does not, and say plainly which of test/schema.test.ts's rules protect against a real provider rejection and which are ours alone.
  3. If the provider changed: say so, and state that the fixture, not the smoke check, is what tracks it.
  4. Update the Release checks section in AGENTS.md with the answer. Replace the "is not known" sentence — do not leave both the question and the answer in place.

Acceptance criteria

  • AGENTS.md states the reason, with the evidence it rests on (a pi source reference, or a dated probe result).
  • The status of the manual openai-codex smoke check is explicit: still required, or dropped, and why.
  • If the finding changes what test/schema.test.ts should assert, that is stated. Changing the assertions is a separate issue.

Non-goals

  • Do not relax test/schema.test.ts in this issue. A finding that a rule is ours alone is not a reason to drop it in the same PR that discovers it — the rule may still be worth keeping.
  • Do not re-run bun run probe:strict and commit a new fixture here. That needs a real key and belongs to a schema change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions