Skip to content

Track multi-provider spend and model usage - #2759

Open
Chipagosfinest wants to merge 53 commits into
steipete:mainfrom
Chipagosfinest:agent/cumulative-spend-converged
Open

Track multi-provider spend and model usage#2759
Chipagosfinest wants to merge 53 commits into
steipete:mainfrom
Chipagosfinest:agent/cumulative-spend-converged

Conversation

@Chipagosfinest

@Chipagosfinest Chipagosfinest commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What this changes

CodexBar Usage & Spend now works as a true multi-provider overview instead of stopping at Claude and Codex.

  • Every enabled or configured service stays visible, including providers without spend history yet.
  • The Overview leads with combined estimated spend, tracked tokens, and a clear reporting-provider count.
  • Compact native rows and smooth scrolling keep large provider rosters usable while Codex remains first.
  • Copy Stats and Copy Image produce a shareable local summary with honest partial-coverage disclosure.
  • Top models preserve meaningful variants such as Claude Fable, Opus, and Sonnet.

Unknown spend is never turned into $0. Estimates use ~, currencies remain separate, and incomplete coverage stays explicit.

Screenshots

All values shown here are synthetic.

Combined Overview

CodexBar Overview with combined estimated spend, token total, and six providers

Share card

CodexBar local share card with six connected services, partial spend coverage, and top models

Wide settings · Narrow settings · Proof notes

OpenRouter scope

OpenRouter stays inside the existing plugin boundary on JavaScriptCore and QuickJS. An ordinary API key continues to report credits and key quota.

Management /activity enrichment is deliberately deferred to a separate follow-up. Removing it restores the existing single optional-request timeout budget and avoids adding a provider-specific history payload or a broader async QuickJS host contract to this PR.

Verification

Exact head: 2194fc72c

  • Focused OpenRouter/plugin tests: 26 passing.
  • make check: localization, manifests, docs, SwiftFormat, and SwiftLint passed.
  • The prior exact head passed every GitHub CI lane; fresh exact-head CI is the merge gate for this narrowed commit.
  • Local screenshot proof covers the production Overview, share window, responsive Settings layouts, and Copy Stats flow with synthetic provider data.

This is the conflict-free successor to #2469 and includes the provider-runtime convergence from #2758. OpenRouter /activity is preserved separately for a clean follow-up after this lands.

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Aug 8, 2026
@clawsweeper

clawsweeper Bot commented Aug 8, 2026

Copy link
Copy Markdown

Codex review: needs real behavior proof before merge. Reviewed August 9, 2026, 5:12 PM ET / 21:12 UTC.

ClawSweeper review

What this changes

Expands CodexBar Usage & Spend into a multi-provider dashboard, menu overview, and local share card with 7-, 30-, and 365-day ranges, explicit partial coverage, and every tracked provider visible.

Merge readiness

Blocked until stronger real behavior proof is added - 5 items remain

Keep open for maintainer product sign-off and stronger current-head proof. Current main still caps an unset merged Overview at six providers and documents only 7/30-day spend history, so this PR’s central capability is not already implemented; however, its all-provider default changes existing unset setups and its late Codex RPC timeout-race change lacks direct runtime evidence.

Priority: P2
Reviewed head: 8bcaf4e74cac92719d0cabd753853fd991bc029e
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The UI implementation and focused coverage are strong, but direct proof is incomplete for the latest non-visual RPC timeout behavior and product acceptance remains pending.
Proof confidence 🦐 gold shrimp (3/6) Needs stronger real behavior proof before merge: Prepared screenshots convincingly demonstrate the synthetic Settings, menu, and share-card UI, but they do not demonstrate the current-head Codex RPC timeout-race change; add a redacted terminal trace or runtime log that forces the timeout and shows the post-fix classification. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Needs proof Needs stronger real behavior proof before merge: Prepared screenshots convincingly demonstrate the synthetic Settings, menu, and share-card UI, but they do not demonstrate the current-head Codex RPC timeout-race change; add a redacted terminal trace or runtime log that forces the timeout and shows the post-fix classification. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed 5 items Current main retains the six-provider default: The merged Overview limit is six, and an unset preference returns only that prefix of active providers; the proposed all-provider default is therefore a new behavior rather than an already-landed fix.
Current documentation defines the smaller spend surface: Current documentation describes 7- or 30-day history for a restricted token-cost provider set, confirming that the PR materially expands the shipped user capability.
Visible feature proof inspected: The supplied Overview, share-card, and wide/narrow Settings captures clearly show six tracked sources, partial spend disclosure, and responsive tracked-access layouts using synthetic data.
Findings None None.
Security None None.

How this fits together

CodexBar collects provider status and optional local cost-history snapshots, aggregates them by provider and currency, then presents the result in Settings, the merged menu-bar Overview, and a local share card. This PR changes the roster and coverage rules that feed all three user-facing summaries.

flowchart LR
A[Provider settings and accounts] --> B[Usage and cost snapshots]
B --> C[Spend aggregation]
C --> D[Coverage and partial-total rules]
D --> E[Settings dashboard]
D --> F[Merged menu overview]
D --> G[Local share card]
Loading

Decision needed

Question Recommendation
Should an unset merged Overview show every enabled provider by default, replacing the existing first-six behavior while preserving explicit saved subsets? Approve all enabled providers by default: Accept the PR’s model, relying on native scrolling for large rosters and retaining explicitly saved provider subsets.

Why: This changes the default information density and menu interaction for existing users without a stored selection; source and tests can verify mechanics but cannot choose the intended product default.

Before merge

  • Add real behavior proof - Needs stronger real behavior proof before merge: Prepared screenshots convincingly demonstrate the synthetic Settings, menu, and share-card UI, but they do not demonstrate the current-head Codex RPC timeout-race change; add a redacted terminal trace or runtime log that forces the timeout and shows the post-fix classification. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • Resolve merge risk (P1) - Users without an explicit saved Overview selection will move from a six-provider menu to every enabled provider; native scrolling mitigates long lists, but the default is an intentional compatibility and UX choice.
  • Resolve merge risk (P2) - The PR also changes Codex RPC timeout classification, yet the supplied screenshots do not prove after-fix behavior when a real subprocess times out.
  • Complete next step (P2) - A maintainer must choose the default Overview behavior, and the contributor must supply real runtime proof for the late RPC timeout change before merge.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Patch scope 72 files, +3,415 / −336 lines The feature spans provider aggregation, menu rendering, Settings, share output, localization, documentation, and regression coverage.
Focused test additions 4 new test files New coverage targets tracked sources, partial-coverage truth, integrated model aggregation, and Settings sharing.

Root-cause cluster

Relationship: canonical
Canonical: #2759
Summary: This PR is the explicitly described conflict-free successor to the older cumulative-spend branch.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

  1. Confirm the new default after runtime proof (recommended)
    Approve the all-provider default only after a redacted timeout-race trace establishes that the added Codex RPC behavior fails as a timeout rather than an EOF.
  2. Retain the existing default
    Keep the first-six unset behavior if menu density and legacy interaction should remain unchanged.

Technical review

Best possible solution:

Approve or narrow the all-provider default explicitly, then add a redacted production-boundary timeout trace showing that the Codex RPC EOF race now reports a timeout while retaining the existing visual proof.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a bug reproduction: this is a new dashboard and menu capability. The supplied captures provide high-confidence visual evidence for the UI portion, but not for the separate timeout-race behavior.

Is this the best way to solve the issue?

Unclear until product intent is confirmed. The aggregation and disclosure model is coherent, but changing the unset menu default needs explicit acceptance and the late RPC change needs direct runtime proof.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 4a83b87aff22.

Labels

Label justifications:

  • P2: This is a broad but non-emergency user-facing Usage & Spend enhancement requiring a bounded maintainer decision.
  • merge-risk: 🚨 compatibility: The unset merged Overview default changes from a first-six subset to all enabled providers.
  • merge-risk: 🚨 availability: The patch modifies Codex RPC timeout handling, which can affect fallback behavior when the subprocess stalls.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • feature: ✨ showcase: ClawSweeper spotlight: unusually compelling feature idea for maintainer attention. The feature turns fragmented provider history into a candid, local multi-provider view that preserves unknown and partial coverage instead of inventing totals.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Prepared screenshots convincingly demonstrate the synthetic Settings, menu, and share-card UI, but they do not demonstrate the current-head Codex RPC timeout-race change; add a redacted terminal trace or runtime log that forces the timeout and shows the post-fix classification. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • proof: 📸 screenshot: Contributor real behavior proof includes screenshot evidence. Prepared screenshots convincingly demonstrate the synthetic Settings, menu, and share-card UI, but they do not demonstrate the current-head Codex RPC timeout-race change; add a redacted terminal trace or runtime log that forces the timeout and shows the post-fix classification.

Evidence

What I checked:

  • Current main retains the six-provider default: The merged Overview limit is six, and an unset preference returns only that prefix of active providers; the proposed all-provider default is therefore a new behavior rather than an already-landed fix. (Sources/CodexBar/SettingsStore.swift:191, 0e46d1940c35)
  • Current documentation defines the smaller spend surface: Current documentation describes 7- or 30-day history for a restricted token-cost provider set, confirming that the PR materially expands the shipped user capability. (docs/providers.md:21, 0e46d1940c35)
  • Visible feature proof inspected: The supplied Overview, share-card, and wide/narrow Settings captures clearly show six tracked sources, partial spend disclosure, and responsive tracked-access layouts using synthetic data. (docs/screenshots/spend-dashboard-proof/README.md:1, 8bcaf4e74cac)
  • Timeout-race proof remains incomplete: The current head includes the Codex RPC timeout-race change, but the supplied proof is UI screenshots and does not show an injected or real timeout being classified as a timeout after the change. (Sources/CodexBarCore/UsageFetcher.swift:1015, 8bcaf4e74cac)
  • Current release provenance: The current source baseline is v0.49.0; it still contains the six-provider and 7/30-day behavior, so the PR is neither shipped nor obsolete. (Sources/CodexBar/SettingsStore+Defaults.swift:865, 0e46d1940c35)

Likely related people:

  • steipete: Available blame assigns the current merged-Overview limit and selection behavior to Peter Steinberger, and recent main history includes the provider-architecture refactors that this PR extends. (role: current implementation owner in available history; confidence: high; commits: 0e46d1940c35, 7deae2acc44e, 15a855a2132a; files: Sources/CodexBar/SettingsStore.swift, Sources/CodexBar/SettingsStore+Defaults.swift, Sources/CodexBar/PreferencesMenuBarPane.swift)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Add a redacted terminal or runtime-log trace that exercises the Codex RPC timeout race on the current head.
  • After adding the trace, update the PR body for automatic re-review or ask a maintainer to comment @clawsweeper re-review.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (53 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-09T07:27:17.407Z sha 2194fc7 :: found issues before merge. :: [P2] Separate token completeness from spend completeness
  • reviewed 2026-08-09T08:02:08.733Z sha 2194fc7 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-09T17:44:00.294Z sha e1c60a5 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-09T18:08:35.192Z sha a573a79 :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-09T18:15:36.293Z sha a573a79 :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-09T19:14:01.090Z sha 8bcaf4e :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-09T19:35:53.777Z sha 8bcaf4e :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-09T19:49:56.586Z sha 8bcaf4e :: needs real behavior proof before merge. :: none

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b03a03bd1a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Sources/CodexBar/SpendDashboardModel.swift
Comment thread Sources/CodexBar/UsageStore+TokenCost.swift Outdated
Comment thread Sources/CodexBar/StatusItemController+OverviewSpend.swift Outdated
@Chipagosfinest

Copy link
Copy Markdown
Contributor Author

Thanks for the sharp review. Two clarifications on the product and architecture decisions:

  • All connected providers in Overview is intentional for both unset fresh and unset existing installs; preserving the six-provider fallback would keep the exact omission this PR is meant to fix. The no-selection path is covered with 20 providers, explicit saved subsets remain scoped, and long rosters use native scrolling.
  • The OpenRouter bridge is intentionally a narrow first-party exception, gated to .openrouter, bounded and validated in Swift, Codable-compatible, fail-open for optional enrichment, and covered across JavaScriptCore/QuickJS. We considered a generic token-history contract, but without a second consumer it would prematurely freeze shared accounting semantics that differ here (completed UTC days, routed models, reasoning-token inclusion, management-key scope).

I also fixed the two concrete hosted-Xcode regressions exposed by the previous run at 65327296f8850f00b96ce5885431f244b716ae43. I will request a fresh review after the exact-head CI is terminal. Pete: the remaining requested decision is whether this narrowly tested first-party plugin exception is acceptable as the durable boundary.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Aug 8, 2026
@clawsweeper clawsweeper Bot removed the rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. label Aug 8, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 9, 2026
@Chipagosfinest

Copy link
Copy Markdown
Contributor Author

@steipete exact-head CI is green and every inline review thread is resolved. ClawSweeper's remaining concrete concern is valid on QuickJS, but its suggested JavaScript repair is not cross-engine:

  • ctx.http.get is asynchronous under JavaScriptCore.
  • QuickJS's hostHTTP bridge is intentionally blocking. In a focused local regression, wrapping /key and /activity in Promise.all still started /activity about 1.00s after /key and returned after about 2.03s when both timed out.
  • Making those calls truly concurrent on QuickJS would require a new generic async/batch host contract, which would widen this already-large PR and itself falls under VISION.md sign-off.

The honest maintainer choices are:

  1. Accept the bounded ~2s worst-case optional OpenRouter tail and approve the all-connected-provider default plus the narrow first-party openRouterActivityUsage payload.
  2. Split OpenRouter /activity enrichment into a follow-up while retaining the core multi-provider dashboard/share work here.
  3. Explicitly authorize a broader QuickJS host-API change.

I recommend 1 if the bounded provider-specific tail is acceptable; otherwise 2. I do not recommend hiding the engine limitation behind Promise.all or broadening this PR with option 3 without your direction.

Current head: b1a8742faee53e29fd25e4e212ee7ad9f4d685a1
CI: https://github.com/steipete/CodexBar/actions/runs/31298033582

@Chipagosfinest

Copy link
Copy Markdown
Contributor Author

Scoped this down in response to the compatibility P2: OpenRouter now keeps its existing credits and key-quota behavior, while /activity and its dedicated plumbing are deferred to a separate follow-up. This removes the second serial optional request from the synchronous QuickJS path and avoids widening this PR with a new host API.

Validation before push:

  • focused OpenRouter/plugin tests: 26 passing
  • make check: passing
  • local full runner timed out in CLIEntryTests even on isolated retry, so I am not claiming a clean local full-suite run; exact-head CI is the merge gate

@codex review
@clawsweeper re-review

@steipete the remaining owner decision is now only whether an unset Overview should show every connected provider instead of the legacy first six.

Exact head: 2194fc72c

@clawsweeper

clawsweeper Bot commented Aug 9, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot removed the rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. label Aug 9, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2194fc72c4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Sources/CodexBar/ShareStatsPayload.swift Outdated
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. merge-risk: 🚨 auth-provider 🚨 Merging this PR could break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. labels Aug 9, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. and removed proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 9, 2026
@clawsweeper clawsweeper Bot added merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. and removed merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. labels Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature: ✨ showcase ClawSweeper spotlight: unusually compelling feature idea for maintainer attention. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant