Skip to content

Add Google and xAI models.dev pricing lookup - #2377

Open
joeVenner wants to merge 7 commits into
steipete:mainfrom
joeVenner:codex/modelsdev-google-xai-models
Open

Add Google and xAI models.dev pricing lookup#2377
joeVenner wants to merge 7 commits into
steipete:mainfrom
joeVenner:codex/modelsdev-google-xai-models

Conversation

@joeVenner

@joeVenner joeVenner commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add an additive provider-aware models.dev lookup seam for Google/Gemini, Vertex AI, and xAI/Grok providers.
  • Map Gemini to google, Vertex AI to google-vertex with google fallback, and Grok to xai.
  • Wire those provider-scoped lookups into the Pi session runtime cost path and expose the supported providers through codexbar cost.

Proof

  • Previous-context check: PR Add GPT-5.6 Sol/Terra/Luna Codex pricing #2023 covered OpenAI GPT-5.6 pricing and Fix: Refresh pricing for unknown models #2018 covered unknown-model refresh behavior; neither mapped Google/xAI provider families through models.dev.
  • Absence check against current base returned no matches:
    git grep -n -E "gemini-3\.5-flash|gemini-3\.1-pro-preview|gemini-3\.1-flash-lite|grok-4\.5|grok-4\.3|grok-4\.20" origin/main -- Sources Tests docs
  • Source catalog check from https://models.dev/api.json showed:
    • google: gemini-3.5-flash / Gemini 3.5 Flash / input 1.5 / output 9 / cache_read 0.15 / context 1048576
    • google: gemini-3.1-pro-preview / Gemini 3.1 Pro Preview / input 2 / output 12 / cache_read 0.2 / context 1048576
    • google: gemini-3.1-flash-lite / Gemini 3.1 Flash Lite / input 0.25 / output 1.5 / cache_read 0.025 / context 1048576
    • xai: grok-4.5 / Grok 4.5 / input 2 / output 6 / cache_read 0.3 / context 500000
    • xai: grok-4.3 / Grok 4.3 / input 1.25 / output 2.5 / cache_read 0.2 / context 1000000
    • xai: grok-4.20-0309-reasoning / Grok 4.20 (Reasoning) / input 1.25 / output 2.5 / cache_read 0.2 / context 1000000
  • Runtime regression test writes cached models.dev fixture data with separate google, google-vertex, and xai rates, then verifies PiSessionCostScanner.loadDailyReport computes provider-scoped costs for Pi JSONL rows:
    • google / gemini-3.5-flash: 160 tokens, cost 0.0006015
    • google-vertex / gemini-3.1-pro-preview: 160 tokens, cost 0.0008171, proving Vertex uses the Vertex catalog before Google fallback
    • xai / grok-4.5: 160 tokens, cost 0.000503
  • Built CLI proof used an isolated temp home with a synthetic .pi/agent/sessions/2026-07-28T10-00-00-000Z_google-xai-proof.jsonl containing only assistant rows. No real account files, Keychain, or browser cookies were read. Command shape:
    HOME=[temp] CFFIXED_USER_HOME=[temp] XDG_CONFIG_HOME=[temp]/.config XDG_CACHE_HOME=[temp]/.cache .build/debug/CodexBarCLI cost --provider <provider> --format json --pretty --refresh
  • Built CLI output excerpts:
    { "provider" : "gemini", "source" : "local", "sessionTokens" : 160000, "sessionCostUSD" : 0.6015,
      "daily" : [{ "date" : "2026-07-28", "modelBreakdowns" : [{ "modelName" : "gemini-3.5-flash", "totalTokens" : 160000, "cost" : 0.6015 }] }] }
    { "provider" : "vertexai", "source" : "local", "sessionTokens" : 160000, "sessionCostUSD" : 0.802,
      "daily" : [{ "date" : "2026-07-28", "modelBreakdowns" : [{ "modelName" : "gemini-3.1-pro-preview", "totalTokens" : 160000, "cost" : 0.802 }] }] }
    { "provider" : "grok", "source" : "local", "sessionTokens" : 160000, "sessionCostUSD" : 0.503,
      "daily" : [{ "date" : "2026-07-28", "modelBreakdowns" : [{ "modelName" : "grok-4.5", "totalTokens" : 160000, "cost" : 0.503 }] }] }

Verification

  • CLANG_MODULE_CACHE_PATH=/private/tmp/codexbar-clang-cache swift test --disable-sandbox --scratch-path /Users/mosaab/Documents/Projects/CodexBar/.build --filter PiSessionCostScannerTests: 19 tests passed.
  • CLANG_MODULE_CACHE_PATH=/private/tmp/codexbar-clang-cache swift test --disable-sandbox --scratch-path /Users/mosaab/Documents/Projects/CodexBar/.build --filter 'provider lookup resolves current Google and xAI models': 1 test passed.
  • git diff --check: clean.
  • make check completed generated parser hash, package/docs/locales checks, SwiftFormat lint mode, and SwiftLint strict with 0 violations, then exited on the sandbox-only macOS plist cache write permission error after lint shutdown.

No live provider probes, browser-cookie imports, or Keychain reads were run.

@joeVenner

Copy link
Copy Markdown
Contributor Author

@clawsweeper review

@clawsweeper

clawsweeper Bot commented Jul 27, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal priority bug or improvement with limited blast radius. labels Jul 27, 2026
@clawsweeper

clawsweeper Bot commented Jul 27, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed August 12, 2026, 5:14 AM ET / 09:14 UTC.

ClawSweeper review

What this changes

Adds models.dev pricing lookup and local Pi-session cost reporting for Gemini, Vertex AI, and Grok, including support through the codexbar cost command.

Merge readiness

⚠️ Needs maintainer review before merge - 4 items remain

Keep open: current main and v0.49.2 still limit Pi-session cost reports to Codex and Claude, so this remains a distinct feature. The implementation and isolated CLI proof are credible, but the head is merge-dirty and needs an owner-selected landing order and a fresh rebase.

Priority: P2
Reviewed head: b8362254982bad4da719a2e257ac9bc789dbde22
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) Strong deterministic CLI proof and focused tests are present, but the merge-dirty head and failing checks require a current-main revalidation pass.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): The PR body shows after-fix built-CLI output from isolated synthetic Pi sessions for all three provider paths, which directly demonstrates the changed local cost behavior.
Patch quality 🦐 gold shrimp (3/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): The PR body shows after-fix built-CLI output from isolated synthetic Pi sessions for all three provider paths, which directly demonstrates the changed local cost behavior.
Evidence reviewed 5 items Current behavior remains missing: Current main returns an empty Pi-session report for every provider except Codex and Claude, so Gemini, Vertex AI, and Grok local-session costing has not landed. The v0.49.2 release contains this current two-provider implementation.
Provider-scoped implementation: The PR maps Google/Gemini, Google Vertex, and xAI Pi records to their own cost-report providers and resolves their models.dev prices, including Vertex-before-Google fallback.
Focused regression coverage: The PR adds fixture-based lookup assertions and a Pi JSONL regression test covering Google, Vertex, and xAI pricing calculations.
Findings None None.
Security None None.

How this fits together

CodexBar parses local Pi/OMP JSONL sessions into daily token-cost reports, using the cached models.dev catalog for provider/model rates. Those reports feed provider cost snapshots in the app and the codexbar cost CLI output.

flowchart LR
    A[Pi and OMP session logs] --> B[Session cost scanner]
    B --> C[Provider and model mapping]
    C --> D[models.dev pricing cache]
    D --> E[Daily cost report]
    E --> F[App and CLI cost output]
Loading

Decision needed

Question Recommendation
Should this Google/Vertex/xAI mapping be the first rebased landing in the overlapping provider-pricing series, ahead of #2374 and #2376? Land this mapping first: Use this proof-backed Google/Vertex/xAI change as the first rebased provider-pricing landing, then stack the other mappings onto its resolved main branch.

Why: The three open PRs modify the same cost-scanner, pricing, and capability-gate surfaces; the owner explicitly requested a chosen sequence, and the correct sequence cannot be inferred safely from source alone.

Before merge

  • Resolve merge risk (P1) - GitHub reports the head as merge-dirty; the required rebase overlaps current cost-pricing and provider-gatekeeper work, so preserving current-main behavior needs fresh validation.
  • Resolve merge risk (P1) - The lint and lint-build-test checks are failing on the submitted head; the supplied context does not establish their cause, so they should be rerun after the rebase.
  • Complete next step (P2) - A maintainer must select the landing order for the overlapping provider-pricing PRs; after that, the contributor can rebase this branch and refresh validation.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production versus test delta production +181/-29, tests +313/-93 The provider-cost surface grows substantially, with more changed test lines than production lines and focused fixture coverage for each new provider family.

Merge-risk options

Maintainer options:

  1. Decide the mitigation before merge
    Choose one provider-pricing PR as the first landing, rebase this mapping onto current main while retaining the recent cost-pricing behavior, then refresh focused tests, repository checks, and the isolated CLI transcript.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Technical review

Best possible solution:

Choose one provider-pricing PR as the first landing, rebase this mapping onto current main while retaining the recent cost-pricing behavior, then refresh focused tests, repository checks, and the isolated CLI transcript.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is an additive feature rather than a report of broken behavior; the PR instead provides an isolated built-CLI after-fix transcript with synthetic Pi sessions.

Is this the best way to solve the issue?

Yes, conditionally: provider-scoped catalog lookup is a narrow fit for the existing cost pipeline and its fixture tests cover the new routing, but it must be rebased and revalidated against current main.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a90dfed5c264.

Labels

Label justifications:

  • P2: This adds useful local cost estimates for optional provider sessions without evidence of an urgent existing-user outage.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body shows after-fix built-CLI output from isolated synthetic Pi sessions for all three provider paths, which directly demonstrates the changed local cost behavior.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body shows after-fix built-CLI output from isolated synthetic Pi sessions for all three provider paths, which directly demonstrates the changed local cost behavior.

Evidence

What I checked:

Likely related people:

  • Peter Steinberger: Current-main blame attributes the existing models.dev and Pi cost paths to Peter, and he authored the later cost-pricing fix on the conflicting file as well as the explicit sequencing request. (role: recent cost-pricing contributor and likely landing-sequence owner; confidence: high; commits: 330ae4384b18, 4befde4bbdf0; files: Sources/CodexBarCore/PiSessionCostScanner.swift, Sources/CodexBarCore/Vendored/CostUsage/CostUsagePricing.swift, Sources/CodexBarCore/CostUsageFetcher.swift)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Rebase onto current main after the owner selects the series order, preserving the recent cost-pricing fix.
  • Rerun the focused pricing/scanner tests and make check, then restore the failing CI checks on the rebased head.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (31 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-08T11:26:30.637Z sha 1eac100 :: needs changes before merge. :: [P1] Update the provider capability gate expectations
  • reviewed 2026-08-08T23:41:49.324Z sha 1eac100 :: needs changes before merge. :: [P1] Update the provider capability gate expectations
  • reviewed 2026-08-09T04:55:31.346Z sha 1eac100 :: found issues before merge. :: [P1] Update the provider capability gate
  • reviewed 2026-08-09T14:32:28.743Z sha 46c997f :: needs changes before merge. :: [P1] Update the provider capability gate expectations
  • reviewed 2026-08-09T20:12:22.152Z sha 46c997f :: needs changes before merge. :: [P1] Update provider-capability gate expectations
  • reviewed 2026-08-09T22:02:34.671Z sha 46c997f :: needs changes before merge. :: [P1] Update provider capability-gate expectations
  • reviewed 2026-08-09T22:50:38.143Z sha b836225 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-09T23:13:17.775Z sha b836225 :: needs maintainer review before merge. :: none

@joeVenner

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 28, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jul 28, 2026
@joeVenner

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 28, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 28, 2026
@joeVenner
joeVenner force-pushed the codex/modelsdev-google-xai-models branch from 2d19620 to b99340f Compare July 31, 2026 17:30
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 31, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. and removed merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 5, 2026
@joeVenner
joeVenner force-pushed the codex/modelsdev-google-xai-models branch from 73dfd89 to 3b034fa Compare August 5, 2026 16:05
@joeVenner

Copy link
Copy Markdown
Contributor Author

@steipete Rebased onto latest main and added a regression test confirming that a scoped Codex home still suppresses Pi-session merging. Ready for review.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Aug 6, 2026
@joeVenner
joeVenner force-pushed the codex/modelsdev-google-xai-models branch 2 times, most recently from 87ad75c to 1eac100 Compare August 6, 2026 20:36
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Aug 6, 2026
…DevCostUSD out of enum to satisfy lint limits
The hash commit was dropped during the rebase because the generated file conflicted with main. Regenerate it now that all pricing changes are applied.
ClawSweeper flagged that a custom codexHomePath should continue to suppress merging default Pi-session spend. Add a test verifying ambient scans merge Pi while scoped scans do not.
@joeVenner
joeVenner force-pushed the codex/modelsdev-google-xai-models branch from 1eac100 to 46c997f Compare August 9, 2026 14:28
@joeVenner

Copy link
Copy Markdown
Contributor Author

@steipete — this PR has been rebased onto latest main and is ready for review. Thanks!

After rebasing the Google/xAI models.dev pricing work, ProviderArchitectureGatekeeperTests failed because new provider-specific branches (Gemini, Grok, VertexAI) appeared in shared cost code and existing allowlist anchors shifted. Add '// Provider-specific by design:' markers at each provider-owned dispatch point, update the gatekeeper's hardcoded supportsTokenSnapshot set and shifted suppressed references, and remove obsolete CostUsageFetcher/PiSessionCostScanner allowlist entries whose anchors no longer match.

Closes provider-architecture gatekeeper failures on the Google/xAI rebase.
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants