Skip to content

fix: reconcile stale Grok Console quota after 429 - #1003

Merged
chenyme merged 1 commit into
mainfrom
Console
Aug 24, 2026
Merged

fix: reconcile stale Grok Console quota after 429#1003
chenyme merged 1 commit into
mainfrom
Console

Conversation

@chenyme

@chenyme chenyme commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Summary

  • Reconcile Grok Console 429 responses against the upstream /usage snapshot before treating an account as quota-exhausted.
  • Distinguish available, exhausted, refreshing, and inconclusive reconciliation results.
  • Apply a non-accumulating Retry-After cooldown when Console quota remains available.
  • Preserve the last valid snapshot and enqueue a bounded retry when the usage probe fails.
  • Refresh complete Console quota snapshots older than 6 hours in batches of 10 accounts per minute.
  • Use one distributed-lock namespace for immediate, queued, migration, and stale-quota refresh paths.

Why

A Console resource_exhausted 429 does not always mean the account has depleted its weekly quota. It may also represent a transient upstream rate limit.

Previously, routing could continue using an old quota snapshot while repeatedly applying account failure penalties. This caused accounts with remaining quota to enter increasingly long cooldowns, and switching accounts could reproduce the same failure across the pool.

Behavior

  • Confirmed remaining quota: temporarily cool the account down without increasing failure_count, then allow normal account rotation.
  • Confirmed exhausted quota: persist the refreshed quota window and exclude the account through quota routing.
  • Probe failure: preserve the previous snapshot and queue a retry.
  • Refresh already running on another replica: do not enqueue duplicate work or report a synchronization failure.
  • HTTP 402 behavior is unchanged and continues to use the existing hard-failure policy.

Tests

Added coverage for:

  • Console 429 with remaining quota
  • confirmed quota exhaustion
  • failed usage probes preserving the previous snapshot
  • retry scheduling after probe failures
  • distributed-lock contention without duplicate queuing
  • identical lock keys across immediate and queued refresh paths
  • bounded stale-snapshot catch-up
  • non-accumulating Console 429 cooldowns
  • unchanged hard-failure handling for HTTP 402

Verified with:

  • go test ./... -count=1
  • go vet ./...
  • go build ./...
  • git diff --check

@chenyme
chenyme merged commit 6463c6a into main Aug 24, 2026
9 checks passed
gnayhz added a commit to gnayhz/grok2api that referenced this pull request Aug 24, 2026
…, console quota reconcile, provider-prefix models, access-cookie auth, privacy diagnostics, doom-loop guards

Functional picks from upstream: chenyme#995 SuperGrok Plus paid-tier recognition, chenyme#998 shared weekly limits for Web Imagine, chenyme#1002 statsig refresh on stale-page media rejection, chenyme#1003 console quota reconciliation after 429 (transient 5xx/429 no longer cascade into durable cooldowns), chenyme#1001 provider prefixes preserved in public model names, chenyme#1000 scoped access-cookie fallback for stripped bearer auth, chenyme#983 privacy-safe request diagnostics with bounded retention, chenyme#994 output doom-loop guards with split content/reasoning thresholds, serde hardening (output_text.annotations, model backfill, Anthropic marker filtering).

Excluded: upstream quality-guard sidecar and its tooling/pages/config (superseded by the built-in quarantine/rotation guard), account-egress lease-block routing tests (superseded by scopeless pools). Kept our semantics where they diverge: strike-preserving clear-cooldown endpoint plus a new explicit /clear-cooldown-force, strict routing targets, RSC-clean confirmation gate, rotating fixed targets.
@chenyme
chenyme deleted the Console branch August 25, 2026 09:16
gnayhz pushed a commit to gnayhz/grok2api that referenced this pull request Aug 27, 2026
fix: reconcile stale Grok Console quota after 429
gnayhz added a commit to gnayhz/grok2api that referenced this pull request Aug 27, 2026
…, console quota reconcile, provider-prefix models, access-cookie auth, privacy diagnostics, doom-loop guards

Functional picks from upstream: chenyme#995 SuperGrok Plus paid-tier recognition, chenyme#998 shared weekly limits for Web Imagine, chenyme#1002 statsig refresh on stale-page media rejection, chenyme#1003 console quota reconciliation after 429 (transient 5xx/429 no longer cascade into durable cooldowns), chenyme#1001 provider prefixes preserved in public model names, chenyme#1000 scoped access-cookie fallback for stripped bearer auth, chenyme#983 privacy-safe request diagnostics with bounded retention, chenyme#994 output doom-loop guards with split content/reasoning thresholds, serde hardening (output_text.annotations, model backfill, Anthropic marker filtering).

Excluded: upstream quality-guard sidecar and its tooling/pages/config (superseded by the built-in quarantine/rotation guard), account-egress lease-block routing tests (superseded by scopeless pools). Kept our semantics where they diverge: strike-preserving clear-cooldown endpoint plus a new explicit /clear-cooldown-force, strict routing targets, RSC-clean confirmation gate, rotating fixed targets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Grok Console 配额同步失败后缓存永不过期,导致 resource_exhausted 429 换号也全失败

1 participant