docs(quality): settle the cost model against the final billing - #2967
Conversation
The first version was written from same-day readings; the settled meters more than doubled the day (ledger ~96 EUR at 45% -> 212.62 EUR final). Per-file moves from 25k to ~44k input tokens (~0.07 EUR) for a heavy mix, the incident's 135-file change measured 5.5-6M tokens per full review -- at the 6M ceiling's edge, not the comfortable 3.4M previously claimed -- and the per-push cost range is now production-measured (a tenth to all of a first review, set by context coupling). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GitHub Actions
Inactive
|
Keiko for Quality — run summary ✅ complete · head
Budget: 113438 tokens allotted |
GitHub Actions
Inactive
|
Warning Review limit reached
Next review available in: 32 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughChangesReview cost and spend documentation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/qa/review-cost-model.md`:
- Around line 162-164: Update the production-scale example in the review-cost
model documentation to reconcile the 36-file pull request with hits: 32 and
misses: 3: either identify the remaining non-reviewable file or change the
stated total to 35 reviewable files. Ensure the 91% coverage calculation uses 35
as its denominator.
- Around line 99-101: Update the cost-model statement in the paragraph around “A
push costs” to qualify the file-count claim: replace the absolute wording “not
by how many files it touches” with wording that says file count alone does not
determine cost, while preserving the surrounding explanation about coupling.
- Line 56: Update the cost calculation in the review-cost documentation so the
21-run average uses incident-only run-level spend, excluding the small-change
tail included in €212.62; if that data is unavailable, relabel €10.1 as a
daily-spend quotient rather than the average cost of one 135-file review.
- Around line 78-80: Update the heavy-change description in the cost model
documentation by replacing “cutting from about 135” with wording that states it
“truncates at about 135,” while preserving the surrounding threshold
explanation.
In `@docs/troubleshooting/model-review-spend.md`:
- Around line 71-72: Update the run-count guidance in the incident comment to
use conditional wording: replace the absolute “twenty is not enough” statement
with “twenty runs may not cover all completed reviews,” while preserving the
specific observations about twenty-one completed reviews and thirty-four runs on
one branch.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: dc29e814-5fff-4d90-ac98-8d5852b6b79b
📒 Files selected for processing (2)
docs/qa/review-cost-model.mddocs/troubleshooting/model-review-spend.md
📜 Review details
⏰ Context from checks skipped due to timeout. (12)
- GitHub Check: review
- GitHub Check: ui
- GitHub Check: Coverage shard (packages 2/3)
- GitHub Check: Coverage shard (packages 3/3)
- GitHub Check: Coverage suite (scripts)
- GitHub Check: Build, scan, SBOM, smoke
- GitHub Check: Core quality
- GitHub Check: Coverage shard (packages 1/3)
- GitHub Check: Coverage suite (keiko-ui)
- GitHub Check: Analyze (javascript-typescript)
- GitHub Check: Analyze (csharp)
- GitHub Check: Analyze (javascript-typescript)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{ts,tsx,js,jsx,mjs,cjs,json,md,yml,yaml}
📄 CodeRabbit inference engine (AGENTS.md)
Use English only in code, comments, identifiers, documentation, commit messages, issues, and pull requests.
Files:
docs/troubleshooting/model-review-spend.mddocs/qa/review-cost-model.md
docs/troubleshooting/**/*.md
📄 CodeRabbit inference engine (AGENTS.md)
Document operator failure modes using Symptom, Root Cause, Diagnostic Steps, and Resolution; keep the content redacted with no live endpoints or logs.
Files:
docs/troubleshooting/model-review-spend.md
🔇 Additional comments (1)
docs/qa/review-cost-model.md (1)
4-8: LGTM!Also applies to: 18-26, 33-46, 113-113
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41370ebb0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review round: the 21-run average no longer divides the whole day by the incident (the small-change tail and qualification runs are subtracted first), later-push cost names both of its terms (direct misses plus context invalidation), the 36-file example carries its reviewable count, 'cutting' becomes 'truncates', and the run-count guidance is scoped to the incident it records. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GitHub Actions
Inactive
GitHub Actions
Inactive
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6b65807c34
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The token_budget rationale in the workflow still carried the pre-settlement numbers (240 files at 25k, 'larger than any change this repository has produced') after the cost model moved to the settled ones. Same numbers in both places now: the 135-file incident measured 5.5-6M per full review, so a heavy change meets the ceiling around 135. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
![keiko-for-quality[bot]](https://avatars.githubusercontent.com/in/4290143?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ae06286ef8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GitHub Actions
Inactive
GitHub Actions
Inactive
|
1 participant
The cost model (#2962) was written the morning after the expensive day, from readings taken while that day was still billing. The settled meters have since more than doubled it: the credit ledger read ~€96 at 45% materialization; the day closed at €212.62 (verified twice — Cost Management meters and the portal's period figure, which also shows the €130.00 monthly credit at €0.00 remaining). Three planning figures move, and this change corrects them with their derivations written inline:
Also added from production data of 2026-08-03, because the "later push" column was the store's best case presented as the typical case: the measured per-push range runs from a tenth of a first review (32 hits / 3 misses on 35 reviewable) to all of it (
cache.context_invalidated: 12→ 19/19 misses on every push — the store working as designed once shared context moves). The troubleshooting entry's saturation warning is strengthened with the settled run counts (21 completed reviews of the incident change; 34 runs on a single branch in one day).Out of scope: the activation record (
docs/qa/keiko-for-quality.md) — its "twenty-one times" is settled-correct as history — and any workflow or config change.Verification:
npm run gates:sonarPASS on this diff;npx prettier --checkclean on both files; every changed number cross-checked against the sources named above. Docs-only — no code, no exports, no release impact.🤖 Generated with Claude Code