Skip to content

fix(auth): floor quota cooldown at the escalating ladder - #5130

Open
warelik wants to merge 8 commits into
router-for-me:devfrom
warelik:fix/quota-backoff-hint-floor
Open

fix(auth): floor quota cooldown at the escalating ladder#5130
warelik wants to merge 8 commits into
router-for-me:devfrom
warelik:fix/quota-backoff-hint-floor

Conversation

@warelik

@warelik warelik commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Gemini and Antigravity can answer HTTP 429 for a fully exhausted daily quota with a RetryInfo hint of only ~479ms (observed: 479417207ns). Taking that hint verbatim returned the dead credential to the pool ~500ms later AND pinned BackoffLevel forever because every retry recomputed the ladder and then discarded it in favor of the sub-second hint.

This change floors the quota cooldown deadline at the escalating ladder calculation. A provider hint may still push the recovery deadline further out, but can never pull it in below the ladder step — except for the two cases the provider itself marks as non-exhaustion (see the follow-up rounds below).

Code changes

Before

next, backoffLevel = quotaCooldownAfterFailure(state.Quota, now)
if result.RetryAfter != nil {
    next = now.Add(*result.RetryAfter)
}

After

next, backoffLevel = quotaCooldownAfterFailure(state.Quota, now)
if result.RetryAfter != nil {
    // A provider hint can be sub-second even when the quota is exhausted for
    // the whole day, so never let it undercut the escalating quota ladder.
    if hinted := now.Add(*result.RetryAfter); hinted.After(next) {
        next = hinted
    }
}

Blast radius

quotaCooldownAfterFailure is package-private with exactly two call sites:

  • MarkResult in sdk/cliproxy/auth/conductor_cooldown.go
  • applyAuthFailureState in sdk/cliproxy/auth/conductor_cooldown.go

Both call sites are updated identically to respect the ladder floor.

Tests & Verification

Added two unit tests in sdk/cliproxy/auth/cooldown_backoff_test.go:

  • TestMarkResultSubSecondQuotaHintStillEscalates
  • TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates

Mandatory reverse bite-check failure

Reverting the fix produced the expected test failure:

=== RUN   TestMarkResultSubSecondQuotaHintStillEscalates
    cooldown_backoff_test.go:354: expected BackoffLevel 4 after hinted post-window failure, got 3
--- FAIL: TestMarkResultSubSecondQuotaHintStillEscalates (0.00s)
=== RUN   TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates
    cooldown_backoff_test.go:372: expected BackoffLevel 1 after the first hinted failure, got 0
--- FAIL: TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates (0.00s)
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.430s
FAIL

Passing tests with fix applied

=== RUN   TestMarkResultSubSecondQuotaHintStillEscalates
--- PASS: TestMarkResultSubSecondQuotaHintStillEscalates (0.00s)
=== RUN   TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates
--- PASS: TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates (0.00s)
PASS
ok  	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.348s

Follow-up review round 1 — zero-delay retries (4c854d7c)

Review pointed out that an unconditional floor also swallowed RetryAfter: 0, which upstream
uses to mean "retry immediately on the same credential". The gate now exempts a non-positive
hint (conductor_cooldown.go:829 in MarkResult, :1966 in applyAuthFailureState).

Tests: TestMarkResultZeroRetryAfterDoesNotApplyLadderFloor,
TestApplyAuthFailureStateZeroRetryAfterDoesNotApplyLadderFloor.

Reverse bite-check — dropping the *retryAfter <= 0 clause from both gates:

--- FAIL: TestMarkResultZeroRetryAfterDoesNotApplyLadderFloor (0.00s)
    cooldown_backoff_test.go:424: expected BackoffLevel to remain 0 for zero RetryAfter, got 1
--- FAIL: TestApplyAuthFailureStateZeroRetryAfterDoesNotApplyLadderFloor (0.00s)
    cooldown_backoff_test.go:439: expected BackoffLevel 0 for zero RetryAfter, got 1
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.319s
FAIL

Follow-up review round 2 — transient rate limits (fd030f7b)

Review then pointed out that the floor still applied to 429s the executor had already
classified as a short-lived rate limit rather than an exhausted quota, parking a usable
credential for a whole ladder step.

classifyAntigravity429 (internal/runtime/executor/antigravity_executor_credits.go:203) already
distinguishes RATE_LIMIT_EXCEEDED from QUOTA_EXHAUSTED, but the classification never reached
the conductor: newAntigravityStatusErr built a bare statusErr{code, msg, retryAfter} and only
status, text and hint were carried across.

  • statusErr gains transientRateLimit bool and func (e statusErr) TransientRateLimit() bool
    (internal/runtime/executor/openai_compat_executor.go:1031). Every other executor leaves it
    false, so their behaviour is unchanged.
  • newAntigravityStatusErr sets it for 429s only:
    err.transientRateLimit = classifyAntigravity429(body) == antigravity429RateLimited
    (antigravity_executor_credits.go:344). Deliberately narrow: QUOTA_EXHAUSTED, soft bodies
    and anything unclassified keep the floor and keep escalating.
  • isTransientRateLimitError (sdk/cliproxy/auth/conductor_cooldown.go:1459) recovers the flag
    with errors.As over an interface{ TransientRateLimit() bool }, mirroring how
    retryAfterFromError already recovers the hint. The execution paths set
    result.TransientRateLimit next to the existing result.RetryAfter assignment.

The flag lives on Result, not on auth.Error, and is threaded as the sixth parameter of
applyAuthFailureState: sdk/cliproxy/auth/errors_compat_test.go
TestErrorLegacyUnkeyedLiteralCompatibility constructs Error with an unkeyed literal, so any
new field on that struct breaks go vet. The seven pre-existing applyAuthFailureState call
sites in cooldown_backoff_test.go were updated mechanically with a trailing , false; no
assertion was changed.

Tests: TestMarkResultTransientRateLimitKeepsProviderHint,
TestApplyAuthFailureStateTransientRateLimitKeepsProviderHint (its second half feeds the same
sub-second hint through an exhausted-quota error and asserts the floor and BackoffLevel == 1
still apply), TestIsTransientRateLimitErrorDetectsWrappedProviderClassification,
TestNewAntigravityStatusErrMarksTransientRateLimit.

Reverse bite-check — reverting both gates to *retryAfter <= 0:

=== RUN   TestMarkResultTransientRateLimitKeepsProviderHint
    cooldown_backoff_test.go:482: expected BackoffLevel to stay 0 for a transient rate limit, got 1
--- FAIL: TestMarkResultTransientRateLimitKeepsProviderHint (0.00s)
=== RUN   TestApplyAuthFailureStateTransientRateLimitKeepsProviderHint
    cooldown_backoff_test.go:497: expected BackoffLevel to stay 0 for a transient rate limit, got 1
--- FAIL: TestApplyAuthFailureStateTransientRateLimitKeepsProviderHint (0.00s)
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.422s
FAIL

Reverse bite-check — discarding the executor classification:

=== RUN   TestNewAntigravityStatusErrMarksTransientRateLimit
    antigravity_executor_credits_test.go:249: expected a RATE_LIMIT_EXCEEDED 429 with a sub-second hint to be marked transient
--- FAIL: TestNewAntigravityStatusErrMarksTransientRateLimit (0.00s)
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.666s
FAIL

go build ./..., go vet ./sdk/cliproxy/... ./internal/runtime/executor/..., gofmt -l and
go test ./sdk/cliproxy/auth/... ./internal/runtime/executor/... are clean.

Follow-up review round 3 — streaming and token-count paths (4227378e)

Review found the classification was still only wired into the non-stream execution path.

  • executeStreamWithModelPool built every failure result from retryAfterFromError alone. All
    five results now set the flag next to the hint: sdk/cliproxy/auth/conductor_stream.go:275,
    :354, :374, :386, :401. The mid-stream result in wrapStreamResult (:131) is left
    alone on purpose — it sets no hint today, so the ladder still applies there and adding one
    would be a behavioural change beyond this review.
  • Antigravity CountTokens hand-rolled statusErr twice. Both now go through
    newAntigravityStatusErr (internal/runtime/executor/antigravity_executor_tokens.go:167 and
    :172), which does the same helps.ParseRetryDelay work and additionally applies
    classifyAntigravity429.

Tests: TestExecuteStreamKeepsProviderHintForTransientRateLimit (new file
sdk/cliproxy/auth/conductor_stream_classification_test.go, drives a real Manager.ExecuteStream
against a stub executor whose error reports 429 + 479417207ns + transient) and
TestAntigravityCountTokensClassifiesTransient429 (real CountTokens against an httptest
server serving a structured RATE_LIMIT_EXCEEDED 429).

Reverse bite-check — dropping the new line from the executor-error stream branch:

--- FAIL: TestExecuteStreamKeepsProviderHintForTransientRateLimit (0.00s)
    conductor_stream_classification_test.go:54: expected BackoffLevel to stay 0 for a transient rate limit, got 1
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.433s
FAIL

Reverse bite-check — restoring the hand-rolled token-count statusErr:

--- FAIL: TestAntigravityCountTokensClassifiesTransient429 (0.00s)
    antigravity_executor_credits_test.go:858: expected a RATE_LIMIT_EXCEEDED token-count 429 to be marked transient
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.666s
FAIL

Downstream counterpart: kaitranntt/CLIProxyAPIPlus#198

Review follow-up: short-cooldown 429s are transient

The Antigravity executor raises a synthetic 429 while an auth sits in a short cooldown. That cooldown is a local, self-imposed pause of at most a few minutes, but the error carried only a positive retryAfter hint and no classification, so isTransientRateLimitError() (sdk/cliproxy/auth/conductor_cooldown.go:1459) returned false and applyAuthFailureState() (sdk/cliproxy/auth/conductor_cooldown.go:914) treated it as an exhausted upstream quota, escalating BackoffLevel toward the 30 minute ceiling.

All three cooldown short-circuits now set transientRateLimit: trueinternal/runtime/executor/antigravity_executor_execute.go:36 (Execute), :268 (executeClaudeNonStream) and internal/runtime/executor/antigravity_executor_stream.go:35 (ExecuteStream) — because the classification is consumed independently on each path (conductor_execution.go:381, conductor_stream.go:275/354/374/386/401, conductor_home_execution.go:181).

Test: TestAntigravityShortCooldownErrorIsTransient in internal/runtime/executor/antigravity_executor_cooldown_transient_test.go.

Reverse bite-check — dropping transientRateLimit: true from the three literals:

--- FAIL: TestAntigravityShortCooldownErrorIsTransient (0.00s)
    --- FAIL: TestAntigravityShortCooldownErrorIsTransient/execute (0.00s)
        antigravity_executor_cooldown_transient_test.go:81: expected the synthetic short-cooldown 429 to be transient so the conductor rotates instead of escalating backoff
    --- FAIL: TestAntigravityShortCooldownErrorIsTransient/execute-claude (0.00s)
        antigravity_executor_cooldown_transient_test.go:81: expected the synthetic short-cooldown 429 to be transient so the conductor rotates instead of escalating backoff
    --- FAIL: TestAntigravityShortCooldownErrorIsTransient/execute-stream (0.00s)
        antigravity_executor_cooldown_transient_test.go:81: expected the synthetic short-cooldown 429 to be transient so the conductor rotates instead of escalating backoff
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.671s
FAIL

Review follow-up: ordinary Claude 429s are transient

Ordinary (non-unified) Claude 429s reached claudeRateLimitError wrapping a statusErr with no transientRateLimit flag (internal/runtime/executor/claude_executor_request.go:295-308), so isTransientRateLimitError() returned false and applyAuthFailureState() treated an ordinary model-level throttle as exhausted quota, escalating BackoffLevel toward the 30 minute ceiling. The ordinary path in classifyClaudeUpstreamError now sets transientRateLimit = true; the unified 5h/7d rejection path is untouched and stays on the quota ladder.

Tests: TestClassifyClaudeUpstreamError_OrdinaryRateLimitIsTransient and TestClassifyClaudeUpstreamError_UnifiedRejectionNotTransient (the negative pin for the unified path).

Reverse bite-check — dropping err.transientRateLimit = true:

--- FAIL: TestClassifyClaudeUpstreamError_OrdinaryRateLimitIsTransient (0.00s)
    claude_executor_beta_policy_test.go:306: ordinary Claude 429 = {"type":"error","error":{"type":"rate_limit_error","message":"Number of requests has exceeded your rate limit."}}, want a transient rate limit
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.695s
FAIL

Review follow-up: transient 429s without a hint bypass the quota ladder

The ladder bypass previously required a parseable retryAfter hint, so a transient 429 with no hint (e.g. an ordinary Claude throttle without reset headers) fell into quotaCooldownAfterFailure and advanced BackoffLevel anyway. Both copies of the logic — MarkResult's per-model state (sdk/cliproxy/auth/conductor_cooldown.go:829) and applyAuthFailureState's credential state (:1917) — now bypass the ladder for any transient 429, keeping the hint verbatim when present and falling back to nextTransientErrorRetryAfter (~60s) otherwise.

Test: TestManager_MarkResult_Transient429WithoutHintBypassesQuotaLadder.

Reverse bite-check — reverting conductor_cooldown.go:

--- FAIL: TestManager_MarkResult_Transient429WithoutHintBypassesQuotaLadder (0.00s)
    conductor_overrides_test.go:836: expected credential quota ladder to stay at level 0 for a transient 429 without hint, got 1
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.610s
FAIL

Review follow-up: transient fallback respects disabled cooldowns

With transientErrorCooldownSeconds < 0 the transient fallback returns a zero time, but both 429 branches still recorded Unavailable=true / Quota.Exceeded=true with an empty NextRecoverAt, which availabilityBlock reads as an indefinite park. The transient-429 handling in MarkResult (sdk/cliproxy/auth/conductor_cooldown.go:861) and applyAuthFailureState (:1952) now skips the marking when the transient fallback yields a zero time, preserving any pre-existing quota block.

Test: TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldown.

Reverse bite-check — reverting conductor_cooldown.go:

--- FAIL: TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldown (0.00s)
    conductor_overrides_test.go:899: expected the credential quota state to stay clear with transient cooldowns disabled
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.435s
FAIL

Review follow-up: disabled-cooldown skip restores availability fields

The transient-cooldown-off skip restored the status/quota fields but left auth.Unavailable=true (set at the top of applyAuthFailureState) and auth.NextRetryAfter untouched, so the credential stayed blocked anyway. The prior availability fields are now captured and restored (sdk/cliproxy/auth/conductor_cooldown.go:2027).

Test: TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldownAuthLevel (auth-level Result drives applyAuthFailureState).

Reverse bite-check — reverting conductor_cooldown.go:

--- FAIL: TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldownAuthLevel (0.00s)
    conductor_overrides_test.go:952: expected the credential to stay available with transient cooldowns disabled
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.426s
FAIL

Review follow-up: reasoned RATE_LIMIT_EXCEEDED without RetryInfo is transient

RATE_LIMIT_EXCEEDED without a RetryInfo detail downgrades to SoftRetry, and only the RateLimited category was marked transient, so a plain per-minute throttle was read as exhausted quota. newAntigravityStatusErr now marks the soft rate limit transient when the classification comes from the ErrorInfo reason (antigravity_executor_credits.go:350); the bare "too many requests" message heuristic still stays on the quota ladder.

Test: new case in TestNewAntigravityStatusErrMarksTransientRateLimit.

Reverse bite-check — reverting antigravity_executor_credits.go:

--- FAIL: TestNewAntigravityStatusErrMarksTransientRateLimit (0.00s)
    antigravity_executor_credits_test.go:290: expected a RATE_LIMIT_EXCEEDED 429 without RetryInfo to be marked transient
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.788s
FAIL

Tooling note

The jbcontext CLI is installed on this machine but its stored session cannot be decrypted (the OS keychain is not accessible), so jbcontext search could not run. The equivalent semantic search, review and blast-radius passes were performed with local code-intelligence tooling instead.

@github-actions
github-actions Bot changed the base branch from main to dev August 21, 2026 04:11
@github-actions

Copy link
Copy Markdown

This pull request targeted main.

The base branch has been automatically changed to dev.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 06c71872b9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4c854d7c0f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fd030f7bec

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Outdated
Comment thread sdk/cliproxy/auth/conductor_execution.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4227378efd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/runtime/executor/openai_compat_executor.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3955ad6fb6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4834e439a4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c898b95d6e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a9eee2b2cf

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go
Comment thread internal/runtime/executor/antigravity_executor_credits.go Outdated
@warelik

warelik commented Aug 23, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up for the mission review:

  • Positive or missing Retry-After on a transient 429 now uses the maximum of the provider hint, the quota backoff ladder, and a 10-second floor. The original provider hint remains on the result for client propagation.
  • Explicit zero or negative hints retain rotate-without-wait behavior.
  • Ordinary Claude model-level 429s are already classified as transient; unified 5-hour/7-day rejections remain quota-scoped.
  • Added coverage for per-model, credential-level, and streaming paths, including short and long hints.

Checks on fba80153: build, close-when-agents-md-changed, and ensure-no-translator-changes pass. Please review the latest commit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fba8015307

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Outdated
warelik added a commit to warelik/CLIProxyAPI that referenced this pull request Aug 23, 2026
Finding router-for-me#5130: ordinary throttling 429s ("Rate Limit Reached",
"Too Many Requests", "Resource exhausted") with no quota signal were
climbing the quota backoff ladder and putting live accounts into long bans.

- Add isQuotaExhaustedResultError and isTransientThrottlingResultError.
  Quota 429s are detected by "quota"/"insufficient_quota"/"quota_exceeded"
  or a long Retry-After (>= 1m).
- In applyAuthFailureState and the model-state 429 branch, transient 429s
  use recoverableFailureRetryAfter (or a short Retry-After) and mark
  Unavailable, but do not touch Quota.BackoffLevel or Quota.Exceeded.
- Credential-scoped 429s and explicit quota 429s still propagate as quota.
- Add TestApplyAuthFailureStateTransient429DoesNotEscalateQuotaBackoff.
- Update TestManager_DeepSeekCredentialFailuresRotateCredential so
  "Rate Limit Reached" is transient; add a "Quota Exceeded" case for quota.
- Update reports/4881-rebuild.md with the router-for-me#5130 finding and fix.
warelik added a commit to warelik/CLIProxyAPI that referenced this pull request Aug 23, 2026
Finding router-for-me#5130: ordinary throttling 429s ("Rate Limit Reached",
"Too Many Requests", "Resource exhausted") with no quota signal were
climbing the quota backoff ladder and putting live accounts into long bans.

- Add isQuotaExhaustedResultError and isTransientThrottlingResultError.
  Quota 429s are detected by "quota"/"insufficient_quota"/"quota_exceeded"
  or a long Retry-After (>= 1m).
- In applyAuthFailureState and the model-state 429 branch, transient 429s
  use recoverableFailureRetryAfter (or a short Retry-After) and mark
  Unavailable, but do not touch Quota.BackoffLevel or Quota.Exceeded.
- Credential-scoped 429s and explicit quota 429s still propagate as quota.
- Add TestApplyAuthFailureStateTransient429DoesNotEscalateQuotaBackoff.
- Update TestManager_DeepSeekCredentialFailuresRotateCredential so
  "Rate Limit Reached" is transient; add a "Quota Exceeded" case for quota.
- Update reports/4881-rebuild.md with the router-for-me#5130 finding and fix.
warelik added a commit to warelik/CLIProxyAPI that referenced this pull request Aug 23, 2026
Finding router-for-me#5130: ordinary throttling 429s ("Rate Limit Reached",
"Too Many Requests", "Resource exhausted") with no quota signal were
climbing the quota backoff ladder and putting live accounts into long bans.

- Add isQuotaExhaustedResultError and isTransientThrottlingResultError.
  Quota 429s are detected by "quota"/"insufficient_quota"/"quota_exceeded"
  or a long Retry-After (>= 1m).
- In applyAuthFailureState and the model-state 429 branch, transient 429s
  use recoverableFailureRetryAfter (or a short Retry-After) and mark
  Unavailable, but do not touch Quota.BackoffLevel or Quota.Exceeded.
- Credential-scoped 429s and explicit quota 429s still propagate as quota.
- Add TestApplyAuthFailureStateTransient429DoesNotEscalateQuotaBackoff.
- Update TestManager_DeepSeekCredentialFailuresRotateCredential so
  "Rate Limit Reached" is transient; add a "Quota Exceeded" case for quota.
- Update reports/4881-rebuild.md with the router-for-me#5130 finding and fix.
warelik added a commit to warelik/CLIProxyAPI that referenced this pull request Aug 23, 2026
Finding from router-for-me#4881 review: internal/pluginhost/executor_route.go:248
`discardStreamChunks` started a goroutine that ranged over the source
channel forever. If a plugin left the channel open after an empty terminal
frame, the drainer goroutine never exited, causing a cumulative leak in a
long-lived proxy.

- Pass the request context into `discardStreamChunks`.
- Add `streamDrainTimeout` (5s default); the drainer exits on context
cancellation, on timeout, or when the source channel closes.
- Reset the drain timeout on each received chunk so slow trailing chunks are
still drained, but the goroutine is always bounded.
- Add `TestDiscardStreamChunksExitsOnContextCancel` and
`TestDiscardStreamChunksExitsOnOpenUnclosedChannel` proving the goroutine
exits even when the source channel is never closed.
- Update reports/4881-rebuild.md with the take/no-take findings and router-for-me#5130 status.
warelik added a commit to warelik/CLIProxyAPI that referenced this pull request Aug 23, 2026
… and timeout

`sdk/cliproxy/auth/conductor_stream.go` had its own `discardStreamChunks`
variant that started a goroutine ranging over the source channel forever.
Without a context, a plugin that left the channel open leaked a goroutine
per stream.

- Add `ctx` parameter and `streamDrainTimeout` (5s default) to the auth
`discardStreamChunks`, mirroring the pluginhost fix.
- The drainer exits on context cancellation, on timeout, or when the source
channel closes; the timeout resets on each received chunk so slow trailing
chunks are still drained.
- Update all call sites in `conductor_stream.go` to pass the request context.
- Add `TestDiscardStreamChunksExitsOnContextCancel` and
`TestDiscardStreamChunksExitsOnOpenUnclosedChannel` in
`conductor_stream_test.go`.

Also removes the transient-429 changes from the router-for-me#4881 branch; that finding
belongs on router-for-me#5130 (fix/quota-backoff-hint-floor).
Rebase the router-for-me#5130 change onto current origin/dev. A provider Retry-After
hint may push the recovery deadline later, but must not undercut the
quota ladder or the 10s transient floor. The original hint is still
proxied to the client.

Transient 429s stay off the exhausted-quota ladder. Token-count errors
go through newAntigravityStatusErr so the same classification reaches
the conductor.
@warelik
warelik force-pushed the fix/quota-backoff-hint-floor branch from e9ac9fb to b72a9c8 Compare August 26, 2026 06:50

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b72a9c84b4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Outdated
A hintless classified 429 with transient cooldowns disabled was clearing
Unavailable and NextRetryAfter on the model even when a 401/403/404/5xx
window was already open. Restore the pre-result availability and retry
deadline, matching applyAuthFailureState.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 69f8e1baf3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_stream.go
wrapStreamResult dropped RetryAfter and TransientRateLimit from
post-bootstrap chunk errors, so a classified 429 after the first
payload advanced the exhausted-quota ladder.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 09eeca8028

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Outdated
Non-positive RetryAfter on a non-transient 429
undercut the quota ladder (Antigravity 0s).
Keep the bypass for classified transients only.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0dd09f8bc6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Outdated
Skip-path guards tested Exceeded, which stays true after
NextRecoverAt. An expired record then vetoed skip for a
hintless transient 429 with cooldowns disabled.

Check deadline activity at model and auth level.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d736dc686c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go
Zero-delay Retry-After on websocket_connection_limit_reached
is not a quota signal. After the bypass required
TransientRateLimit, this path climbed the exhausted-quota
ladder. Mark it transient so the conductor keeps the
zero-delay path; usage_limit 429s stay unchanged.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 54c809b590

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread sdk/cliproxy/auth/conductor_cooldown.go
oauth2.googleapis.com 429 is a token-endpoint throttle,
not model quota. Unclassified, it climbed the exhausted-quota
ladder. Mark it transient; Retry-After stays as-is.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 061aaa0de5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/runtime/executor/antigravity_executor_auth.go
Prepare-error Results dropped RetryAfter and
TransientRateLimit, so a classified refresh 429
during request preparation climbed the quota
ladder. Copy both fields like the execute path.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 75aef5f5a9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

next = time.Time{}
transientCooldownOff = true
} else {
next, backoffLevel = transientRateLimitCooldownAfterFailure(state.Quota, result.RetryAfter, now)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep transient cooldowns separate from quota windows

When a classified transient 429 takes this branch, the common code below still stores the deadline as Quota.Exceeded with reason quota. If a genuine exhausted-quota 429 arrives before this 10-second transient window expires, quotaCooldownAfterFailure treats that transient deadline as an existing quota window and reuses it instead of applying the one-minute ladder floor. The model path also calls SetModelQuotaExceeded, whose registry marker lasts five minutes and can make GetFirstAvailableModel report no available client for model: "auto" long after the transient cooldown expires. Store or tag transient cooldowns separately so exhausted-quota handling and registry quota accounting do not consume them.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant