Skip to content

test(agent-study): save the Sonnet 5.5 confirmatory run (T6 of #1071) and pin its bar - #1080

Merged
manojmallick merged 1 commit into
developfrom
feat/t6-sonnet-confirm-data
Nov 21, 2026
Merged

manojmallick merged 1 commit into
developfrom
feat/t6-sonnet-confirm-data

Conversation

@manojmallick

Copy link
Copy Markdown
Owner

Summary

Evidence

  • Bar met on all four clauses, one run per question so every run is a first ask: pooled cost -8.8% [-15.5, -2.0], file found 211 vs 207, uptake 93%. 5 repositories clearly save, 15 are indeterminate, none clearly costs more; chatmail__relay is above +15% on its point estimate (interval [-6.3, +43.5]).
  • Tokens up, cost down: total tokens per run +9.4% with SigMap, agent-reported cost -12.0%. A test pins that direction, so the table cannot be read as a token saving.
  • Two batches were rerun because the laptop slept mid-run (timeouts of 974 to 1,520 s against 420 s). With the discarded attempt in place of its rerun the bar is still met (-8.2% [-14.9, -1.7]), so the rerun does not decide the result.
  • agent-study-confirmatory-tools.test.js: 21 pass. A mutation sweep over a temp copy of the data file (deleted or duplicated row, flipped pass, changed cost, uptake, model, ask, failed run, swapped arm, token counts): 13 of 13 caught. The real file was not touched.
  • The tests that read the agent-study folder pass (advise, advise-emit, change-task-screen, prefetch-hook).

Not shown here

🤖 Generated with Claude Code

… and pin its bar

444 runs of the locate setup in Claude Code on the 20 held-out repositories, one run per question per arm (every run a first ask), as the unedited to-runs.mjs table. The pre-registered bar is met: cost -8.8% [-15.5, -2.0], file found 211 vs 207, uptake 93%; 5 repositories clearly save, 15 are indeterminate, none clearly costs more.

Total tokens per run were 9% higher with SigMap while the reported cost was 12% lower; a test pins that direction so the result is not read as a token saving. The README records the laptop-sleep rerun of two batches and its sensitivity (-8.2% with the discarded attempt, bar met).
@manojmallick
manojmallick merged commit 30d5a33 into develop Nov 21, 2026
6 checks passed
@manojmallick
manojmallick deleted the feat/t6-sonnet-confirm-data branch November 21, 2026 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant