bug: Home ignores Codex usage_limit_reached reset timing during scheduling

[sususu98]

Summary

In Home mode, Codex 429 usage_limit_reached responses include authoritative reset timing in error.resets_at / error.resets_in_seconds, and CLIProxyAPI already parses those fields into an executor RetryAfter. However, Home scheduling is updated later from the usage payload, whose failure shape contains only status_code and body. CLIProxyAPIHome reparses that body, but its parser does not understand the Codex reset fields.

As a result, the central Home scheduler loses the multi-day Codex reset deadline and can dispatch the exhausted credential again long before reset.

This is separate from, but currently amplified by, #57: production Home config also forces disable-cooling=true into Home's own coreManager, which drops even an explicit retry delay. Fixing #57 alone would still leave Codex usage-limit failures on the generic 1s exponential backoff because NewUsageResult currently returns RetryAfter=nil for this body.

Production evidence (anonymized)

Observed on:

• CLIProxyAPIHome v1.0.66 (fe004c5)
• CLIProxyAPI v7.2.116
• 2 healthy Home members and 3 healthy CPA nodes
• routing.strategy=round-robin
• request-retry=3
• effective Home runtime disable-cooling=true

One Codex OAuth credential returned the following failure class:

{
  "error": {
    "type": "usage_limit_reached",
    "message": "The usage limit has been reached",
    "resets_at": 1786160232,
    "resets_in_seconds": 326101
  }
}

The same credential was dispatched again and produced three such 429s:

Attempt	Time (UTC)	CPA node	Model	Reset hint
1	2026-08-04 06:46:51	node A	model A	resets_at=1786160230, resets_in_seconds=334218
2	2026-08-04 06:56:35	node B	model A	resets_at=1786160230, resets_in_seconds=333634
3	2026-08-04 09:02:07	node C	model B	resets_at=1786160232, resets_in_seconds=326101

The first two failures for the same model were only 9m44s apart, while their resets_at was 2026-08-08 03:37:10 UTC (about 3d20h50m after the first failure). This occurred across all three CPA nodes, so it was not isolated to one worker cache.

A later active quota snapshot confirmed the default Codex account-family window was exhausted (remaining=0) with the same multi-day reset window. The credential is now disabled as an operator workaround.

Code path

CLIProxyAPI preserves the reset timing locally

In CLIProxyAPI v7.2.116:

• internal/runtime/executor/codex_executor_terminal.go:283-293 constructs the Codex status error.
• internal/runtime/executor/codex_executor_terminal.go:364-402 recognizes usage_limit_reached, prefers a future error.resets_at, and falls back to positive error.resets_in_seconds.
• sdk/cliproxy/auth/conductor_home_execution.go:177-179 copies the executor retry delay into the local Result.RetryAfter.
• sdk/cliproxy/auth/conductor_cooldown.go:893-904 intentionally does not update local auth state for ephemeral Home dispatches; Home is the scheduler.

But the usage record contract does not carry the structured delay:

• sdk/cliproxy/usage/manager.go:58-62 defines failure metadata as only StatusCode and Body.
• internal/runtime/executor/helps/usage_helpers.go:304-317 serializes only those fields from the error.

Home reparses the body but misses Codex reset fields

In CLIProxyAPIHome current dev (cfae488):

• internal/home/usage_result.go:33-45 reads fail.status_code and fail.body, then calls NewUsageResult.
• internal/cliproxy/auth/result.go:977-1012 sets Result.RetryAfter from parseUsageRetryAfter.
• internal/cliproxy/auth/result.go:1015-1055 only recognizes Google RetryInfo / quotaResetDelay and textual Resets in <duration> messages.

The Codex message above contains no duration text, and error.resets_at / error.resets_in_seconds are not inspected, so RetryAfter is nil.

Current Home config drops cooldown entirely

Also on current dev:

• internal/config/home_mode.go:3-12 forces DisableCooling=true.
• internal/cluster/config_snapshot.go:188 applies that downstream projection to the config also passed to Home runtime.
• internal/cliproxy/auth/result.go therefore leaves the model retry deadline at zero and the scheduler considers it ready again.

That part is already tracked by #57. The missing Codex parser remains independently relevant after separating Home scheduler cooling from downstream CPA cooling.

Deterministic reproduction

A temporary Go overlay test was run against both the production release and current upstream dev without changing repository files:

body := `{"error":{"type":"usage_limit_reached","message":"The usage limit has been reached","resets_at":1786160232,"resets_in_seconds":326101}}`
result := NewUsageResult("codex-auth", "codex", "gpt-codex", 429, body)

// Actual on v1.0.66 and dev:
result.RetryAfter == nil

Commands completed three repetitions on each commit:

v1.0.66 fe004c5: go test -overlay=... ./internal/cliproxy/auth -run TestObserve... -count=3  # PASS
upstream/dev cfae488: go test -overlay=... ./internal/cliproxy/auth -run TestObserve... -count=3  # PASS

The observation test passes only when the current bad behavior (RetryAfter == nil) reproduces.

Expected behavior

For a Codex 429 whose error type is exactly usage_limit_reached:

1. Home should preserve an authoritative reset delay when replaying the usage result.
2. Prefer a future Unix error.resets_at; if it is absent or stale, fall back to a positive error.resets_in_seconds.
3. Block the affected credential/model scheduling state until that deadline instead of starting at generic 1s backoff.
4. Keep transient Codex rate_limit_error / capacity 429s on their existing retry policy.
5. Add focused tests for future resets_at, resets_in_seconds fallback, stale timestamps, non-429 responses, and non-usage_limit_reached 429s.

Related

• bug: Home applies downstream disable-cooling=true to its own scheduler, repeatedly dispatching failed credentials #57 — Home currently applies downstream disable-cooling=true to its own scheduler. This must be fixed for any parsed retry delay to take effect.
• Home 模式下 cooldown 问题总结 #14 — earlier Home-mode retry-delay loss report, fixed for Google/Antigravity response shapes; Codex uses a different response shape.
• CLIProxyAPI already has the desired Codex parsing behavior in parseCodexRetryAfter; Home can mirror the validated semantics or extend the usage-result contract in a backward-compatible way.

Update: actual surviving production log evidence

The following are exact surviving DB-backed Home usage-event log rows for the hard-quota failures. Credential/node identifiers were replaced with stable SHA-256 prefixes, private model aliases were mapped to model-A / model-B, and client/request identity was omitted. The timestamps, executor, status, latency, error type, and reset fields are unchanged.

2026-08-04T06:46:51.3626Z source=home_usage_event provider=codex executor=CodexExecutor credential=sha256:c52137b7a8 cpa_node=sha256:39d811451d model=model-A status=429 latency_ms=1045 error={"type":"usage_limit_reached","message":"The usage limit has been reached","plan_type":"pro","resets_at":1786160230,"resets_in_seconds":334218}
2026-08-04T06:56:35.640731Z source=home_usage_event provider=codex executor=CodexExecutor credential=sha256:c52137b7a8 cpa_node=sha256:00fb7d07ee model=model-A status=429 latency_ms=716 error={"type":"usage_limit_reached","message":"The usage limit has been reached","plan_type":"pro","resets_at":1786160230,"resets_in_seconds":333634}
2026-08-04T09:02:07.968816Z source=home_usage_event provider=codex executor=CodexExecutor credential=sha256:c52137b7a8 cpa_node=sha256:74932bfcb3 model=model-B status=429 latency_ms=3331 error={"type":"usage_limit_reached","message":"The usage limit has been reached","plan_type":"pro","resets_at":1786160232,"resets_in_seconds":326101}

The raw filesystem request-log files are no longer available: the Management index considered the first two remotely routable, but both downloads returned 404; the third event reported request_log_available=false. Read-only checks of the retired Home log directories also found no matching files. Therefore these persisted event rows are the complete surviving production log evidence; no request payload, token, credential label, email, or client identity is attached.

The dashboard's 93 failures are not 93 quota retries

For the matching 2026-07-29T00:00:00Z through 2026-08-05T00:00:00Z Codex window, all 93 failed usage events break down as:

Class	Count
Authentication failures (401, including one auth_unavailable)	77
Client context-window errors (400 context_too_large)	10
Canceled upstream requests	3
Hard Codex quota failures (429 usage_limit_reached)	3

So the scheduler incident contains three confirmed hard-quota redispatches, not 93. The 93 shown by the UI is the aggregate failure count for the selected Codex window.

Why exponential backoff did not protect this deployment

The exponential ladder does exist (1s, 2s, 4s, ... capped at 30m), but it was bypassed here:

1. Effective Home runtime config was disable-cooling=true.
2. nextQuotaRecoverAt returns a zero deadline immediately when cooling is disabled, before consulting either Result.RetryAfter or the exponential ladder.
3. The backoff level therefore remains at zero and no cooldown window is opened.
4. Home-mode CPA deliberately calls reportHomeResult instead of mutating local auth state, so there is no second worker-side cooldown protecting Home's decision.

This is the immediate #57 behavior. If #57 is fixed but the Codex parsing gap in this issue is not, exponential backoff would run, but it would still start at one second instead of honoring the provider's approximately 3-day-20-hour reset deadline.