Skip to content

sessions: attribute PR spend per turn instead of per session#790

Merged
iamtoruk merged 4 commits into
mainfrom
feat/pr-turn-attribution
Jul 21, 2026
Merged

sessions: attribute PR spend per turn instead of per session#790
iamtoruk merged 4 commits into
mainfrom
feat/pr-turn-attribution

Conversation

@iamtoruk

@iamtoruk iamtoruk commented Jul 20, 2026

Copy link
Copy Markdown
Member

Problem

The "Spend by pull request" surfaces (CLI codeburn sessions --by-pr, the pullRequests payload, and the desktop PR tab) attributed spend by reference: a session counted its full cost toward every PR it referenced. One long orchestration session that touched many PRs rendered a wall of identical rows (same full-session cost, same call count on every PR), which is useless for seeing where money actually went.

Core change: per-turn attribution

Attribution is now per turn. During transcript parsing, PR references are captured per turn (prRefs on the cached turn shape), reusing the existing pr-link extraction; the session-level prLinks stays the union. A pure, unit-tested state machine (attributeSessionPrSpend) walks a session's turns in order, carrying currentPrSet = the PR set of the most recent turn that referenced any PR. Each turn's cost/calls are attributed to that set, split evenly across a multi-PR merge-sweep turn. Turns before the first reference form an unattributed bucket. Rows are now summable; the CLI and app footers report the attributed sum plus the unattributed remainder.

A session whose transcript expired before per-turn capture keeps its session-level prLinks but has no per-turn refs; it falls back to an even whole-session split, flagged approx (~ prefix + tooltip).

Range inclusion still keys a turn on its first call timestamp; that is the pre-existing boundary model for date filtering, unchanged by this PR.

Review fixes folded into this PR

  • Cache migration. The session-cache version bump now adopts the prior version's expired-source PR entries instead of abandoning them, and the claude scan preserves and surfaces PR-bearing orphans, so a PR-linked session whose transcript was later deleted still appears (as a legacy even-split) rather than vanishing. The durable daily cache is not touched; the recent-window daily backfill uses a primary-wins merge, so surfacing orphans cannot double-count history.
  • Date-range carry. The parser captures the PR set active at the start of the in-range turn slice and seeds the state machine with it, mirroring the existing branch carry-forward. A PR referenced before the window now owns its later, in-range, ref-less turns instead of the session collapsing to a whole-session approx split.
  • Top-N honesty. The payload caps rows at the top 20 but reports the attributed total for all PRs; it now also sends the count and cost of the capped-away PRs, and the app renders an "Other (N more PRs)" row so the visible table reconciles to the total.
  • Old-payload compatibility. A desktop build reading a payload from an older CLI (by-reference rows, no attributed/unattributed fields) keeps the old non-summable footer wording and never renders NaN.
  • Whole call counts. A multi-PR turn splits its calls by largest-remainder, so per-PR counts stay integral (a 1-call, 2-PR turn no longer shows as 2 calls).
  • Footer reconciliation. The CLI and app footers sum the rounded row values actually shown, so the visible column adds up to the stated total.
  • Session identity. Distinct-session counts key on project + sessionId, not sessionId alone.

Second-round review follow-ups

  • Mixed legacy/live rows reconcile. A PR row combining an expired-legacy contribution (even-split, no turn data) with a live per-turn contribution now emits an explicit "Legacy estimate (no per-turn detail)" category carrying the legacy share, so the expansion always sums to the row cost instead of silently dropping the legacy portion. A legacy-only row still shows no breakdown.
  • Per-file v5 adoption. The version-bump adoption validates each cached file individually, so one malformed entry is skipped rather than rejecting the whole prior cache and dropping every valid expired PR session with it.
  • Other row is a summary line. The desktop "Other (N more PRs)" line moved to the table footer as a muted, separated summary rather than a sorted row (its aggregate cost can exceed a visible row).
  • Expansion state is not stale. Row expansion resets whenever the PR set changes (period, provider, or a refresh that alters the list).
  • Bounded model lists. Payload models per row are capped to the top 4 by attributed cost with a name-ascending tie-break; categories get the same stable tie-break.

New surfaces: models and categories

Each PR contribution records the models of its calls and the turn's task category (split by the same share on a multi-PR turn); legacy even-split rows carry the model union but no category breakdown (none can be honestly assigned). Payload rows gain models (short names, cost-desc) and categories (label + cost, cost-desc, omitted when empty). The CLI --by-pr gains a Models column. The desktop table is now full-width with a Models column and click-to-expand rows showing a per-category cost breakdown with proportional bars, keyboard accessible, with the PR link still opening independently.

Validation

Unit and integration tests cover the state machine (interleaved parallel PRs, multi-ref split, pre-reference overhead, range-carry seed, legacy fallback, largest-remainder calls, project+sessionId dedup, models and categories), the prRefs round-trip through the real incremental-append path (continuation merge and straddle fallback), the version-bump adoption of an expired PR session, the payload round-trip, and the desktop expansion / models / old-payload cases. Root suite, app suite, and tsc are green.

Real-data drive on a frozen snapshot of a heavy orchestration session that shipped roughly thirty PRs, comparing the old and new engines from the same transcripts:

  • Old engine: every PR row showed the identical full-session cost and call count (the bug).
  • New engine, full-session range: distinct per-PR rows whose attributed sum plus the unattributed remainder equals the old distinct-session total to the cent, with the Models column populated.
  • New engine, a "today" range that starts mid-session: the old engine collapsed to a wall of identical rows again, while the new engine showed only the PRs actually carried into the window with their true split, proving the date-range carry fix.
  • The per-session sum invariant (attributed + unattributed = session total) held exactly, and category breakdowns matched the shape of the work (implementation PRs skewed to coding, a fix PR skewed to debugging).

reviewer added 4 commits July 21, 2026 00:30
The by-PR surfaces attributed a session's full cost to every PR it
referenced, so one orchestration session that touched many PRs rendered
identical full-session rows. Capture per-turn PR references during
transcript parsing and attribute each turn's cost to the PR set active at
that turn (split evenly across a multi-PR merge-sweep turn), carrying the
most recent PR set forward across turns that reference nothing. Turns
before the first reference form an unattributed bucket.

A session whose transcript expired before per-turn capture keeps its
session-level prLinks but has no per-turn refs; it falls back to an even
whole-session split, and any row carrying such a portion is flagged
approximate.

Rows are now summable: the CLI and app footers report the attributed sum
plus the unattributed remainder instead of a distinct-session total.
Bumps the session cache version so surviving transcripts re-parse and
populate the new per-turn field. Daily-cache versioning is untouched.
Addresses the review findings on the per-turn PR attribution and adds the
model and task-category surfaces.

Correctness:
- Cache migration: the 5 -> 6 session-cache bump now adopts the prior v5
  file's expired-source PR entries instead of abandoning them, and the claude
  scan preserves and surfaces PR-bearing orphans, so a session whose transcript
  was deleted still appears as a legacy even-split instead of vanishing. The
  daily cache is untouched.
- Date-range carry: the parser captures the PR set active at the start of the
  in-range turn slice and seeds the state machine with it, so a PR referenced
  before the window still owns its later, in-range, ref-less turns instead of
  the session falling back to a whole-session approx split.
- Calls are split across a multi-PR turn by largest-remainder, keeping per-PR
  counts whole (a 1-call, 2-PR turn no longer renders as 2 calls).
- The CLI and app footers reconcile to the rounded row values actually shown.
- Distinct sessions are keyed by project + sessionId, not sessionId alone.
- The app tolerates an older by-reference payload (no attributedCost): it keeps
  the old non-summable footer and never renders NaN.

Features:
- Each PR contribution records the models of its calls and the turn's task
  category (split by the same share on a multi-PR turn); legacy even-split rows
  carry the model union but no category breakdown.
- Payload rows gain models (short names, cost-desc) and categories (label + cost,
  cost-desc, omitted when empty); the payload gains otherPrCount/otherPrCost for
  the PRs beyond the sent top 20.
- CLI --by-pr gains a Models column. The desktop table is now full-width with a
  Models column and click-to-expand rows showing a per-category cost breakdown
  with proportional bars, keyboard accessible, with an "Other (N more PRs)" row
  when capped.

Tests: state-machine seed/models/categories/largest-remainder/dedup, the prRefs
round-trip through the real incremental-append path (continuation + straddle),
v5 adoption of an expired PR session, payload round-trip, and the desktop
expansion/models/old-payload cases.
Round-2 review follow-ups.

- Mixed legacy/live category breakdown: a PR row that combines an expired
  legacy contribution (even-split, no turn data) with a live per-turn
  contribution now emits an explicit "Legacy estimate (no per-turn detail)"
  category carrying the legacy share, so the expansion reconciles to the row
  cost instead of silently dropping it. A legacy-only row still shows no
  breakdown.
- v5 adoption is now per cached file: one malformed entry is skipped instead
  of rejecting the whole v5 cache and dropping every valid expired PR session.
- The desktop "Other (N more PRs)" line moved to the table footer as a muted,
  separated summary rather than a sorted row.
- Row expansion resets when the PR set changes (period/provider/data), so a
  stale expansion cannot linger on a row that is gone.
- Payload models per row are capped to the top 4 by attributed cost, with a
  name-ascending tie-break; categories get the same stable tie-break.

Tests: mixed live+legacy reconciliation, legacy-only omission, model cap and
tie-break, corrupt-plus-valid v5 adoption, and the desktop expansion-reset.
…ion; app: reset PR expansion on period switch

A v5 entry that was valid except for a malformed optional field (agentType,
failed) passed per-file validation and flowed downstream as a non-string.
The expansion reset keyed only on the row URL set, so a period switch that
returned the same PRs kept a stale expansion open over changed numbers.
@iamtoruk
iamtoruk merged commit 45e9312 into main Jul 21, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant