Skip to content

fix: OpenAI StructuredModeResolver should match versioned model names#991

Open
ipalaus wants to merge 1 commit intoprism-php:mainfrom
ipalaus:fix/structured-mode-versioned-models
Open

fix: OpenAI StructuredModeResolver should match versioned model names#991
ipalaus wants to merge 1 commit intoprism-php:mainfrom
ipalaus:fix/structured-mode-versioned-models

Conversation

@ipalaus
Copy link
Copy Markdown

@ipalaus ipalaus commented Mar 28, 2026

Summary

StructuredModeResolver::supportsStructuredMode() uses in_array with exact string matching. Versioned model names like gpt-5-mini-2025-08-07 don't match gpt-5-mini, causing them to silently fall into JSON mode (json_object) instead of Structured mode (json_schema + strict: true).

In JSON mode the schema is embedded as plain text in a system message and json_object format is used — no constrained decoding. This results in significantly more verbose model output compared to proper structured mode.

Approach

Uses a hybrid strategy:

  • Exact matching for gpt-4o and o3-mini families, where only specific versions support structured output (e.g., gpt-4o-2024-08-06 does, earlier versions don't)
  • Prefix matching for gpt-4.1+, gpt-4.5+, and gpt-5+ families, where structured output is a baseline feature across all versions

This avoids the need to update the allowlist every time OpenAI releases a new dated version of a gpt-5 family model, while remaining conservative for older model families with inconsistent support.

Changes

  • Split supportsStructuredMode() into exact matches (gpt-4o, o3) and prefix matches (gpt-4.1+, gpt-5+)
  • Add StructuredModeResolverTest with 32 test cases covering exact matches, prefix matches, JSON mode fallback, and unsupported model exceptions

Test plan

  • All existing tests pass (1495 passed, 8 skipped)
  • 32 new tests for model name resolution
  • gpt-5-mini-2025-08-07 → Structured mode (was JSON mode)
  • gpt-5-mini → Structured mode (unchanged)
  • gpt-4o-mini-2024-07-18 → Structured mode (unchanged, exact match preserved)
  • gpt-4-turbo → JSON mode (unchanged)
  • o1-mini → throws (unchanged)

StructuredModeResolver::supportsStructuredMode() uses in_array with
exact string matching. Versioned model names like gpt-5-mini-2025-08-07
don't match gpt-5-mini, causing them to silently fall into JSON mode
(json_object) instead of Structured mode (json_schema + strict).

In JSON mode, the schema is embedded as plain text in a system message
with no constrained decoding, resulting in significantly more verbose
model output.

Use a hybrid approach:
- Keep exact matching for gpt-4o and o3-mini families where only
  specific versions support structured output
- Use prefix matching for gpt-4.1+, gpt-4.5+, and gpt-5+ families
  where structured output is a baseline feature across all versions
@ipalaus ipalaus force-pushed the fix/structured-mode-versioned-models branch from cb0bcd6 to ff00f86 Compare March 28, 2026 02:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant