Repository navigation
test(agent-study): save the Sonnet 5.5 confirmatory run (T6 of #1071) and pin its bar - #1080
Merged
Merged
Conversation
… and pin its bar 444 runs of the locate setup in Claude Code on the 20 held-out repositories, one run per question per arm (every run a first ask), as the unedited to-runs.mjs table. The pre-registered bar is met: cost -8.8% [-15.5, -2.0], file found 211 vs 207, uptake 93%; 5 repositories clearly save, 15 are indeterminate, none clearly costs more. Total tokens per run were 9% higher with SigMap while the reported cost was 12% lower; a test pins that direction so the result is not read as a token saving. The README records the laptop-sleep rerun of two batches and its sensitivity (-8.2% with the discarded attempt, bar met).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
locateis the only setup where no repository cost more #1071) asbenchmarks/agent-study/data/confirm-locate-claude-code-sonnet-5-5.ndjson(444 rows, the uneditedto-runs.mjsoutput, sha2563be79b9a…), kept apart fromdata/runs.ndjsonso the study's published counts do not move.agent-study-confirmatory-tools.test.js, re-derived with the unchangedconfirm-bar.mjs.locateis the only setup where no repository cost more #1071 (stays open: T1 to T4 and the maintainer's decision are not done).Evidence
chatmail__relayis above +15% on its point estimate (interval [-6.3, +43.5]).agent-study-confirmatory-tools.test.js: 21 pass. A mutation sweep over a temp copy of the data file (deleted or duplicated row, flipped pass, changed cost, uptake, model, ask, failed run, swapped arm, token counts): 13 of 13 caught. The real file was not touched.advise,advise-emit,change-task-screen,prefetch-hook).Not shown here
--bareunder an API key,locateonly, 20 repositories of up to about 2,400 files, "file found" and not answer quality. No data for Opus, the logged-in session or subscription allowance, or large repositories.locateis the only setup where no repository cost more #1071 (with the Sonnet-against-Haiku hint, which is exploratory and not in the repository) waits for the maintainer's yes.🤖 Generated with Claude Code