Prompt cache cost estimation #4
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
khromov
commented
Dec 12, 2025
There was a problem hiding this comment.
Pull request overview
This PR implements prompt cache simulation to estimate cost savings when using prompt caching across multi-step agent interactions. The implementation introduces a token cache tracking system that models how prompts would be cached and reused across conversation steps.
Key changes:
- New
TokenCacheclass to track cached tokens across conversation steps with a growing prefix model simulateCacheSavingsfunction that estimates costs when prompt caching is enabled- Updated pricing module to support cache creation and cache read costs separately
- Enhanced HTML report to display estimated cache savings
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| lib/token-cache.ts | New class implementing token cache tracking with cost calculation |
| lib/utils.ts | Added cache simulation logic and moved buildAgentPrompt from test-discovery |
| lib/utils.test.ts | Comprehensive test coverage for TokenCache and simulateCacheSavings |
| lib/pricing.ts | Refactored to use inline types and added cacheCreationInputTokenCost support |
| lib/pricing.test.ts | Updated tests with proper type usage |
| lib/report.ts | Added cacheSimulation field to metadata |
| lib/report-template.ts | Enhanced UI to display cache simulation results |
| index.ts | Integrated cache simulation into main flow with console output |
| results/result-2025-12-12-23-02-22-anthropic-claude-haiku-4.5.json | Example result file with cache simulation data |
| AGENTS.md | Updated test command from test:self to bun test |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When I started the PR I first thought we'd allow the user to select whether to enable prompt caching or not for a run but there is no central toggle for it in AI SDK so it's sort of a PITA to support it for all providers. Since prompt cache "logic" is fairly basic (system prompt + previous prompts are cached) we can quite easily make a "good enough" estimation that would support all providers where AI SDK has the cache insertion and cached token output costs.