test: run examples end to end against the live API by jbeckwith-oai · Pull Request #374 · openai/openai-ruby

jbeckwith-oai · 2026-08-12T19:49:19Z

Summary

add a workflow_dispatch-only suite that runs 22 Ruby examples against the live OpenAI API
classify every example as covered or explicitly excluded, validate expected output, and upload JSON/Markdown result reports
update image examples and README usage to gpt-image-2

Fixes uncovered by the live suite

use response_id when resuming response streams
require the chat streaming examples to receive non-whitespace model content before reporting success

Impact

The live suite is opt-in and only runs through manual workflow dispatch. A dispatched run fails if an example fails or the inventory becomes incomplete. Inventory validation also runs in required CI without making live API requests.

Validation

22/22 runnable examples passed against the live API on Ruby 4.0
both strengthened chat streaming checks rerun successfully against the live API
723 tests, 3,350 assertions
2,622 files passed RuboCop
Sorbet and 1,212 RBS files validated successfully

openai-sdks · 2026-08-12T20:04:26Z

OkTest Summary

✅ 237/237 SDK tests passed in 10.587s for Ruby SDK PR #374.

Test results — 42 files

Test	Result	Time
`tests/chat-completions-complex-body.test.ts`	✅ Passed	208ms
`tests/chat-completions-create.test.ts`	✅ Passed	265ms
`tests/chat-completions-stream.test.ts`	✅ Passed	144ms
`tests/files-content-binary.test.ts`	✅ Passed	156ms
`tests/files-create-multipart.test.ts`	✅ Passed	148ms
`tests/files-list-pagination.test.ts`	✅ Passed	176ms
`tests/initialize-config.test.ts`	✅ Passed	196ms
`tests/instance-isolation.test.ts`	✅ Passed	215ms
`tests/models-list.test.ts`	✅ Passed	274ms
`tests/responses-background-lifecycle.test.ts`	✅ Passed	263ms
`tests/responses-body-method-errors.test.ts`	✅ Passed	423ms
`tests/responses-cancel-timeout.test.ts`	✅ Passed	220ms
`tests/responses-cancel.test.ts`	✅ Passed	273ms
`tests/responses-compact-retries.test.ts`	✅ Passed	283ms
`tests/responses-compact.test.ts`	✅ Passed	248ms
`tests/responses-create-advanced-stream.test.ts`	✅ Passed	159ms
`tests/responses-create-advanced.test.ts`	✅ Passed	237ms
`tests/responses-create-disconnect.test.ts`	✅ Passed	993ms
`tests/responses-create-errors.test.ts`	✅ Passed	286ms
`tests/responses-create-malformed-api-responses.test.ts`	✅ Passed	161ms
`tests/responses-create-retries.test.ts`	✅ Passed	328ms
`tests/responses-create-stream-failures.test.ts`	✅ Passed	159ms
`tests/responses-create-stream-timeout.test.ts`	✅ Passed	233ms
`tests/responses-create-stream-wire.test.ts`	✅ Passed	2.546s
`tests/responses-create-stream.test.ts`	✅ Passed	123ms
`tests/responses-create-terminal-states.test.ts`	✅ Passed	307ms
`tests/responses-create-timeout.test.ts`	✅ Passed	212ms
`tests/responses-create.test.ts`	✅ Passed	299ms
`tests/responses-delete.test.ts`	✅ Passed	262ms
`tests/responses-input-items-errors.test.ts`	✅ Passed	207ms
`tests/responses-input-items-list.test.ts`	✅ Passed	199ms
`tests/responses-input-items-options.test.ts`	✅ Passed	334ms
`tests/responses-input-tokens-count-timeout.test.ts`	✅ Passed	293ms
`tests/responses-input-tokens-count.test.ts`	✅ Passed	276ms
`tests/responses-malformed-inputs.test.ts`	✅ Passed	2.212s
`tests/responses-not-found-errors.test.ts`	✅ Passed	401ms
`tests/responses-parse.test.ts`	✅ Passed	239ms
`tests/responses-retrieve-retries.test.ts`	✅ Passed	277ms
`tests/responses-retrieve.test.ts`	✅ Passed	292ms
`tests/responses-stored-method-errors.test.ts`	✅ Passed	908ms
`tests/retry-behavior.test.ts`	✅ Passed	3.368s
`tests/sdk-error-shape.test.ts`	✅ Passed	394ms

View OkTest run #31864609136

_{SDK merge (4cd6dd422459) · head (759cb2b3f13a) · base (a089ee1788b9) · OkTest (7e7850017529)}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bb053cb411

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

HAYDEN-OAI

Found one output-validation gap in the new live example suite.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a73a6d4334

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jbeckwith-oai · 2026-08-14T20:19:36Z

@codex review

chatgpt-codex-connector · 2026-08-14T20:22:08Z

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 10986875fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: 10986875fe

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

jbeckwith-oai · 2026-08-14T20:27:15Z

@codex review

chatgpt-codex-connector · 2026-08-14T20:29:19Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: e478efee8a

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e478efee8a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jbeckwith-oai · 2026-08-14T20:37:35Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a31a68c17c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jbeckwith-oai · 2026-08-14T20:44:56Z

@codex review

chatgpt-codex-connector · 2026-08-14T20:45:49Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-14T20:47:41Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 27397027fa

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector · 2026-08-14T20:47:51Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 27397027fa

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

jbeckwith-oai · 2026-08-14T20:48:14Z

@codex review

chatgpt-codex-connector · 2026-08-14T20:50:40Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 27397027fa

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27397027fa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jbeckwith-oai · 2026-08-14T20:58:09Z

@codex review

chatgpt-codex-connector · 2026-08-14T21:00:36Z

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 0d613c6e14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-14T21:00:36Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 0d613c6e14

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

HAYDEN-OAI

Found one substantive runtime compatibility regression in the updated standalone examples; details are in the inline comment.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cca87f90d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jbeckwith-oai · 2026-08-14T21:38:28Z

@codex review

chatgpt-codex-connector · 2026-08-14T21:41:01Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: 0880940224

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0880940224

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jbeckwith-oai · 2026-08-14T21:44:34Z

@codex review

chatgpt-codex-connector · 2026-08-14T21:46:55Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: aea22f49c8

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aea22f49c8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

jbeckwith-oai · 2026-08-14T21:49:51Z

@codex review

chatgpt-codex-connector · 2026-08-14T21:51:29Z

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: a0b54ad9bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-14T21:52:15Z

Security review completed. No security issues were found in this pull request.

Reviewed commit: a0b54ad9bb

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c82334a527

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-advanced-security AI found potential problems Aug 12, 2026

View reviewed changes

Comment thread scripts/examples-e2e.rb Fixed

jbeckwith-oai marked this pull request as ready for review August 12, 2026 20:03

jbeckwith-oai requested a review from a team as a code owner August 12, 2026 20:03

jbeckwith-oai added the generator Touches generated SDK files label Aug 12, 2026

apcha-oai requested changes Aug 13, 2026

View reviewed changes

Comment thread lib/openai/models/completion_choice.rb Outdated

Comment thread rbi/openai/models/completion_choice.rbi Outdated

Comment thread sig/openai/models/completion_choice.rbs Outdated

Comment thread test/scripts/examples_e2e_test.rb Outdated

apcha-oai reviewed Aug 13, 2026

View reviewed changes

Comment thread .simplecov Outdated

chatgpt-codex-connector Bot reviewed Aug 13, 2026

View reviewed changes

Comment thread .github/workflows/examples-e2e.yml

Comment thread .github/workflows/examples-e2e.yml Outdated

Comment thread scripts/examples-e2e.rb Outdated

Comment thread Rakefile

HAYDEN-OAI reviewed Aug 13, 2026

View reviewed changes

Comment thread examples/e2e.yml Outdated

jbeckwith-oai force-pushed the codex/examples-e2e-coverage branch from 9636bf1 to a73a6d4 Compare August 14, 2026 20:06

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread scripts/examples-e2e.rb Outdated

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread .github/workflows/examples-e2e.yml

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread examples/e2e.yml Outdated

Comment thread scripts/examples-e2e.rb Outdated

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread examples/e2e.yml

Comment thread examples/e2e.yml

Comment thread .github/workflows/examples-e2e.yml

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread examples/e2e.yml

Comment thread examples/e2e.yml

Comment thread examples/e2e.yml

Comment thread examples/e2e.yml

Comment thread examples/e2e.yml

Comment thread examples/e2e.yml

Comment thread examples/e2e.yml

Comment thread examples/e2e.yml

HAYDEN-OAI reviewed Aug 14, 2026

View reviewed changes

Comment thread examples/responses/streaming_basic.rb Outdated

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread examples/picture.rb Outdated

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread examples/image_stream.rb

Comment thread examples/chat/streaming_basic.rb

Comment thread examples/chat/streaming_text.rb

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread examples/chat/streaming_structured_outputs.rb Outdated

jbeckwith-oai requested review from HAYDEN-OAI and apcha-oai August 15, 2026 03:22

jbeckwith-oai added 15 commits August 15, 2026 04:03

test examples against the live API

3d25094

test: make coverage regression deterministic

f4dc6d7

fix: fully escape example reports

0046642

test: simplify example end-to-end coverage

e81c58c

test: tighten example inventory validation

0580ee5

fix: address example E2E review feedback

7b3a8fc

fix: validate example output thresholds

00ba2d1

fix: protect live example workflow

88280ec

fix: tighten example e2e completion checks

198959e

fix: require complete example streams

f0b69ba

fix: reject incomplete example results

d70a151

fix: keep standalone examples runtime-clean

3f72c32

fix: verify generated image payload

26f8bb7

fix: require streamed terminal states

74e3457

fix: require structured stream completion

c82334a

jbeckwith-oai force-pushed the codex/examples-e2e-coverage branch from a0b54ad to c82334a Compare August 15, 2026 04:03

chatgpt-codex-connector Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread examples/streaming.rb

Comment thread examples/e2e.yml

fix: validate legacy streams and tool calls

759cb2b

Uh oh!

Conversation

jbeckwith-oai commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Fixes uncovered by the live suite

Impact

Validation

Uh oh!

Uh oh!

openai-sdks Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

OkTest Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HAYDEN-OAI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

jbeckwith-oai commented Aug 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 14, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Security Review

Uh oh!

Uh oh!

jbeckwith-oai commented Aug 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 14, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

jbeckwith-oai commented Aug 14, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbeckwith-oai commented Aug 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 14, 2026

Uh oh!

jbeckwith-oai commented Aug 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 14, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

jbeckwith-oai commented Aug 12, 2026 •

edited

Loading

openai-sdks Bot commented Aug 12, 2026 •

edited

Loading