`gpt-5.5` under-reports `usage.output_tokens` for grammar-constrained custom-tool generations

[ericstj]

Summary

On POST /v1/responses, when the model emits a custom tool call whose output is constrained by a
lark context-free grammar (tools[].type:"custom" + format:{type:"grammar", syntax:"lark"}),
gpt-5.5 sporadically reports a usage.output_tokens that is a small fraction of the tokens it
actually emitted. The deficit also propagates into usage.total_tokens, so affected calls are
under-billed. status is completed and reasoning_tokens is 0, so this is neither
truncation nor reasoning accounting.

• API: /v1/responses (non-streaming), observed on OpenAI direct (api.openai.com).
• Model: gpt-5.5-2026-04-23 (clean repro). Aggregate fleet analysis indicates the issue spans
  the gpt-5.x family on this code path (gpt-5.5 most severe, gpt-5.3-codex most frequent,
  gpt-5.4 mild); it does not occur on non-OpenAI models, which lack this tool path.
• Trigger: a single large grammar-constrained apply_patch generation (hundreds of lines).
• Rate: ~1% extreme (ratio < 0.5), ~5% including mild (ratio < 0.9) under forced large patches.

Evidence (direct api.openai.com, gpt-5.5-2026-04-23, status=completed, reasoning_tokens=0)

We measure: visible_ratio = (output_tokens − reasoning_tokens) / o200k_base_tokens(emitted tool text).
A correct call is ≈ 1.00 (the emitted tool text is essentially all of the visible output).

response id	reported output_tokens	actual emitted tokens (o200k_base)	visible_ratio	total_tokens
resp_0abcbea5ee8ed031016a3034c7ce1481988703b36797cfb5e8	814	6,943 (27,763 chars)	0.117	21,701
resp_04c402a0b422909a016a30398812488199a7820b9cf725a4c9	3,584	7,129 (29,282 chars)	0.503	21,887
resp_05a681efc8e84b32016a303d63af88819a867161c64e98d854 (control)	5,935	5,924	1.002	20,680

For resp_0abcbea5…: the single emitted apply_patch tool call alone is 6,943 o200k tokens, yet
output_tokens is reported as 814 and total_tokens (21,701 = input 20,887 + output 814) is short by
~6,100 tokens versus the content actually produced. The control call (req-id
req_0122fb83ab304749b2928d7f6ef2db6e) on the identical request shape counts correctly (ratio 1.002).

How to reproduce

A self-contained script is attached (openai_apply_patch_undercount_repro.py, stdlib + tiktoken).
Minimal recipe:

1. POST /v1/responses with model:"gpt-5.5", a single custom tool:

   {"type":"custom","name":"apply_patch",
    "format":{"type":"grammar","syntax":"lark","definition":"start: begin_patch hunk+ end_patch\n..."}}

   and tool_choice:{"type":"custom","name":"apply_patch"}, reasoning:{"effort":"medium"},
   stream:false.
2. Prompt that forces one large *** Add File patch (e.g. "write a 300+ line Python SQL engine as a
   single apply_patch hunk").
3. For each response, compare usage.output_tokens − reasoning_tokens to an o200k_base token count
   of the emitted tool-call input text. Repeat a few dozen times; a fraction come back with
   output_tokens far below the emitted-token count.

The script saves each affected response body as undercount_<request_id>.json and prints its
request id. It runs against OpenAI or Azure OpenAI (api-key or managed identity).

Ruled out (not the cause)

• Truncation — status == "completed"; the full grammar-valid patch is present in output.
• Reasoning accounting — reasoning_tokens == 0 on the affected calls.
• Client-side counting/deserialization — the under-count is in the raw HTTP response bytes
  off the wire; we read usage.output_tokens verbatim, and an independent o200k_base count of the
  emitted text is several times larger.
• Predicted outputs — prediction is not set on these requests.

Impact

• Billing: output_tokens and total_tokens are deflated, so affected calls are under-billed.
• Downstream tooling: any system relying on usage for budgeting, rate accounting, eval token
  attribution, or per-call cost is silently skewed on grammar-constrained tool generations.

Ask

Please review the shared JSON response payload(s) (e.g. resp_0abcbea5…, included) and the
attached repro script to confirm the issue. The affected response ids are included should they
help your investigation.

openai_apply_patch_undercount_repro.py
openai_undercount_raw.json
openai_undercount_raw_2.json