fix(evals): Stabilize model-backed scenarios#1022
Draft
dcramer wants to merge 1 commit into
Draft
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
dcramer
force-pushed
the
codex/fix-eval-stream-abort
branch
from
July 23, 2026 19:38
ca25401 to
91c677e
Compare
dcramer
force-pushed
the
codex/fix-eval-stream-abort
branch
from
July 23, 2026 19:53
91c677e to
69631d6
Compare
dcramer
marked this pull request as ready for review
July 23, 2026 19:55
dcramer
force-pushed
the
codex/fix-eval-stream-abort
branch
from
July 23, 2026 22:34
69631d6 to
644d744
Compare
dcramer
marked this pull request as draft
July 23, 2026 22:35
dcramer
marked this pull request as ready for review
July 23, 2026 22:46
dcramer
force-pushed
the
codex/fix-eval-stream-abort
branch
from
July 23, 2026 23:08
644d744 to
86cefef
Compare
dcramer
marked this pull request as draft
July 23, 2026 23:08
dcramer
marked this pull request as ready for review
July 23, 2026 23:34
dcramer
force-pushed
the
codex/fix-eval-stream-abort
branch
from
July 23, 2026 23:55
86cefef to
4c5f505
Compare
dcramer
marked this pull request as draft
July 23, 2026 23:55
dcramer
force-pushed
the
codex/fix-eval-stream-abort
branch
from
July 24, 2026 00:17
4c5f505 to
7b4a564
Compare
dcramer
marked this pull request as ready for review
July 24, 2026 00:21
dcramer
force-pushed
the
codex/fix-eval-stream-abort
branch
from
July 24, 2026 00:23
7b4a564 to
624b095
Compare
Centralize tool model configuration, make web source replays argument-faithful, bound stalled AI Gateway response bodies so existing provider retries can recover, and remove unrelated retrieval from the affected memory and subscription scenarios.
dcramer
force-pushed
the
codex/fix-eval-stream-abort
branch
from
July 24, 2026 00:24
624b095 to
8b501ec
Compare
dcramer
marked this pull request as draft
July 24, 2026 00:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The flaky model-backed evals now fail fast and retry at the correct boundaries instead of drifting into live-network, unrelated retrieval, or inconsistent routing paths.
botConfigvalues and passed through tool composition instead of read from ambient environment variables.max_charsvalue through shared production truncation logic. Strict harness coverage verifies two limits replay from the same fixture.The 60-second eval reply budget and normal shard parallelism remain unchanged.