Skip to content

fix(evals): Stabilize model-backed scenarios#1022

Draft
dcramer wants to merge 1 commit into
mainfrom
codex/fix-eval-stream-abort
Draft

fix(evals): Stabilize model-backed scenarios#1022
dcramer wants to merge 1 commit into
mainfrom
codex/fix-eval-stream-abort

Conversation

@dcramer

@dcramer dcramer commented Jul 23, 2026

Copy link
Copy Markdown
Member

The flaky model-backed evals now fail fast and retry at the correct boundaries instead of drifting into live-network, unrelated retrieval, or inconsistent routing paths.

  • Tool model settings are parsed once into typed botConfig values and passed through tool composition instead of read from ambient environment variables.
  • Web fetch replay stores one canonical max-size source per URL, then applies any model-selected max_chars value through shared production truncation logic. Strict harness coverage verifies two limits replay from the same fixture.
  • Eval AI Gateway requests use an origin-scoped 25-second body-idle timeout. A real Pi stream integration verifies the resulting assistant error enters Junior's existing provider retry path; non-AI eval HTTP is unchanged.
  • Memory provenance scenarios contain the facts required for their drafting/recap requests, and the resource-event fixture skill only handles synthetic provider discovery.
  • High-complexity or uncertain routing conservatively selects the handoff profile, while Slack delivery policy explicitly prevents presenting unsupported top-level channel posts as delivered.

The 60-second eval reply budget and normal shard parallelism remain unchanged.

@vercel

vercel Bot commented Jul 23, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
junior-docs Ready Ready Preview Jul 24, 2026 12:25am

Request Review

@dcramer dcramer changed the title fix(agent): Cancel stalled provider streams fix(evals): Stabilize provider-backed runs Jul 23, 2026
@dcramer
dcramer force-pushed the codex/fix-eval-stream-abort branch from ca25401 to 91c677e Compare July 23, 2026 19:38
@dcramer
dcramer force-pushed the codex/fix-eval-stream-abort branch from 91c677e to 69631d6 Compare July 23, 2026 19:53
@dcramer dcramer changed the title fix(evals): Stabilize provider-backed runs fix(config): Centralize tool model selection Jul 23, 2026
@dcramer
dcramer marked this pull request as ready for review July 23, 2026 19:55
@github-actions github-actions Bot added the risk: medium PR risk score: medium label Jul 23, 2026
@dcramer dcramer changed the title fix(config): Centralize tool model selection fix(evals): Stabilize model-backed scenarios Jul 23, 2026
@dcramer
dcramer force-pushed the codex/fix-eval-stream-abort branch from 69631d6 to 644d744 Compare July 23, 2026 22:34
@dcramer
dcramer marked this pull request as draft July 23, 2026 22:35
@dcramer
dcramer marked this pull request as ready for review July 23, 2026 22:46
@dcramer
dcramer force-pushed the codex/fix-eval-stream-abort branch from 644d744 to 86cefef Compare July 23, 2026 23:08
@dcramer
dcramer marked this pull request as draft July 23, 2026 23:08
@dcramer
dcramer marked this pull request as ready for review July 23, 2026 23:34
@dcramer
dcramer force-pushed the codex/fix-eval-stream-abort branch from 86cefef to 4c5f505 Compare July 23, 2026 23:55
@dcramer
dcramer marked this pull request as draft July 23, 2026 23:55
@dcramer
dcramer force-pushed the codex/fix-eval-stream-abort branch from 4c5f505 to 7b4a564 Compare July 24, 2026 00:17
@dcramer dcramer added the trigger-evals Run evals on a PR label Jul 24, 2026
@dcramer
dcramer marked this pull request as ready for review July 24, 2026 00:21
@dcramer dcramer added trigger-evals Run evals on a PR and removed trigger-evals Run evals on a PR labels Jul 24, 2026
@dcramer
dcramer force-pushed the codex/fix-eval-stream-abort branch from 7b4a564 to 624b095 Compare July 24, 2026 00:23
Centralize tool model configuration, make web source replays argument-faithful, bound stalled AI Gateway response bodies so existing provider retries can recover, and remove unrelated retrieval from the affected memory and subscription scenarios.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: medium PR risk score: medium trigger-evals Run evals on a PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant