fix(auth): honour Claude unified rate-limit resets and scope credential 429s

[chansearrington]

Problem

When a Claude credential hits its rate limit, Home discards the reset timing Anthropic sends and
substitutes its own exponential backoff, then re-dispatches the exhausted credential minutes later.

This is the same defect as #89 ("Home ignores Codex usage_limit_reached reset timing during
scheduling"), one provider over. #89 was fixed by adding a case "codex": to
parseUsageRetryHints and adding codex to an allowlist. That approach cannot work for Claude,
for a reason worth stating up front because it explains why this PR touches an extra file:

Codex's reset hint arrives in the response body (error.resets_at), which Home preserves.
Claude's arrives in response headers — and Home's usage sanitizer deletes response headers for
every provider before the scheduler ever sees them. Adding a case "claude": alone would produce
a correct-looking, fully-tested, permanently inert change.

There are three separate causes, all in Home. Line numbers are at v1.0.72.

1. The reset hint is discarded for every provider except two. quotaCooldownAfterFailure
(internal/cliproxy/auth/result.go:973) computes an exponential deadline and then returns early
unless the provider is one of two hardcoded names:

// internal/cliproxy/auth/result.go:984
provider := strings.ToLower(strings.TrimSpace(result.Provider))
if provider != "antigravity" && provider != "codex" {
    return deadline, nextLevel
}

So every Claude 429 falls back to the quotaBackoffBase → quotaBackoffMax ladder (1s → 30m),
regardless of the 5h/7d window Anthropic actually applied.

2. A credential-wide rejection is recorded as if it affected only one model. In the 429 branch
(result.go:252-273) the scope is hardcoded to quotaScopeModel. Anthropic's unified rate limit is
scoped to the credential. Home already understands a "credential" scope elsewhere
(cooldown.go:559), but nothing on the 429 path ever produces one — so sibling models on the same
exhausted credential stay dispatchable.

3. Home deletes the headers carrying the answer, before its own scheduler runs.
handleUsage (internal/respserver/push/usage.go) sanitizes the payload and reassigns the
variable before passing it on:

sanitizedPayload, errSanitize := cluster.SanitizeUsagePayloadSecrets(payload)   // :46
...
payload = sanitizedPayload                                                      // :50
...
env.Runtime.RecordUsagePayload(ctx, payload)                                    // :59

and inside that sanitizer (internal/cluster/quota_ingestion.go):

collect := func(result gjson.Result) {
    if provider != "codex" || !result.IsObject() { return }   // EXTRACTION is codex-only
    ...
}
collect(gjson.Get(payload, "quota_headers"))
collect(gjson.Get(payload, "response_headers"))

out, errDelete := sjson.Delete(payload, "quota_headers")      // :48
out, errDelete = sjson.Delete(out, "response_headers")        // :52  UNCONDITIONAL, every provider
...
out, errSet := sjson.Set(out, "quota_headers", filtered)      // :69  how codex values SURVIVE

The provider != "codex" guard gates extraction, not the delete. Codex's headers survive only
because they are allowlisted into filtered first and re-attached under quota_headers. Claude's
are destroyed.

This is the only such path: across internal/cluster and internal/respserver there are exactly
three sjson.Delete calls, the two above and one inside sanitizeUsageUpstreamRequestIDs that
touches only request-ID fields.

Production evidence

Measured on a live Home v1.0.72 instance, from its own database (non-secret columns only).

A Claude credential rejected at 11:15:56Z was given next_retry_after = 11:18:05Z — 2 minutes
9 seconds later, against Anthropic quota windows of 5h and 7d. That is the exponential ladder, not
the credential's reset.

Eight consecutive 429s against an already-exhausted quota inside 3.5 minutes:

09:21:50Z  09:21:59Z (+9s)   09:22:56Z (+57s)  09:23:28Z (+32s)
09:24:04Z (+35s)  09:24:27Z (+23s)  09:25:16Z (+49s)

51 Claude 429s over ~33 hours, later ones settling into near-exact 1-hour spacing consistent with
the ladder saturating at its 30m cap and being re-armed on each failure.

Confirming cause 3 independently: of 5,758 usage rows in that database, zero contain
response_headers and zero contain anthropic-ratelimit — while 5,758/5,758 have non-empty
payloads (positive control), and 51 genuine Claude 429s are present.

This is happening in production

Observed on a running v1.0.72 deployment (a small self-hosted cluster), 24h window — 208 log lines
matching 429|rate limit|cooldown|quota. Redacted sample:

[node] 04:21:51 [info] [claude_ratelimit.go:175] parsed Anthropic rate limit reset headers
[node] 04:21:51 [warn] [conductor_execution.go:1907] 429 | 302ms | upstream execution failed:
       provider=claude model=claude-sonnet-5 auth_file=<redacted>
       err={"type":"rate_limit_error","message":"This request would exceed your account's rate limit."}

Subsequent 429s for the same credential and model: 04:21:51 → 04:21:59 → 04:22:56 → 04:23:29 → 04:24:04 → 04:24:28 → 04:25:16 — gaps of 8s, 57s, 33s, 35s, 24s, 48s. It keeps re-attempting a
rate-limited credential within seconds rather than standing down until the advertised reset.

Those two adjacent lines are the whole bug. The first is emitted by the node, which parses
Anthropic's reset headers correctly and reports them upward. sanitizeUsageQuotaHeaders in Home
then deletes those headers before the scheduler ever sees them. The information the scheduler needs
is already being produced correctly, one hop away, and discarded on arrival.

(To be precise about what this log does and does not show: request arrival is driven by real
traffic, so these gaps corroborate the defect rather than measure the backoff ladder directly. The
decisive evidence that the ladder re-arms at ~1s is the mutation test under Testing.)

Changes

1. Gate on whether a reset hint exists, rather than on a provider allowlist (result.go):

if result.ResetAt == nil && result.RetryAfter == nil {
    return deadline, nextLevel
}

Deliberately provider-agnostic rather than "add claude to the allowlist". The allowlist and the
parseUsageRetryHints provider switch are two lists that must agree and nothing enforced that they
did — which is how #89 happened for Codex and how this happened for Claude. parseUsageRetryHints
still returns (nil, nil) from its default branch, so any provider that produces no hint reaches
exactly the same early return as before: antigravity and codex behaviour is unchanged. A parity
test keeps the two lists from drifting apart again.

2. Parse Anthropic's unified rate-limit headers (new claude_ratelimit.go). Reads
Anthropic-Ratelimit-Unified-Status and the -5h-/-7d-/-7d_oi- status and reset variants plus
Retry-After, and classifies a rejection as credential- or model-scoped. Case-insensitive lookup;
latest deadline wins across windows.

3. Fan a credential-scoped rejection across the credential's model states (result.go). When a
429 is classified credential-scoped, blockSiblingModelStatesUntil marks every known ModelState
on that credential unavailable until the reset. It writes through ModelStates rather than setting
aggregate fields directly, so the existing updateAggregatedAvailability derivation — already
called at the end of both the success and failure paths — produces Unavailable/NextRetryAfter/
Quota exactly as it always has. disableCooling suppresses the fan-out.

4. Let Claude's rate-limit headers survive sanitizing (quota_ingestion.go). The collect
closure becomes provider-aware: codex keeps isCodexQuotaHeaderKey unchanged, Claude gains a
matching isClaudeRateLimitHeaderKey allowlist, both under the existing
quotaHeaderValueMaxLength cap. Both deletes stay unconditional — the wholesale strip is the
function's security posture and is not weakened; only an explicit, named allowlist survives, and it
survives by the same quota_headers mechanism codex already uses.

5. Read the surviving headers (internal/home/usage_result.go). RecordUsagePayload previously
parsed only the response body. It now reads response_headers (array-valued, map[string][]string
shape) and quota_headers (flat map[string]string shape), merging them with response_headers
taking precedence, and returns nil when neither is usable so callers fall back to body-only
parsing exactly as before.

Scope

 internal/cliproxy/auth/claude_ratelimit.go             | 199 +++++++++++  (new, production)
 internal/cliproxy/auth/result.go                       | 129 ++++++---
 internal/cluster/quota_ingestion.go                    |  33 ++-
 internal/home/usage_result.go                          |  96 ++++++-
 internal/cliproxy/auth/claude_ratelimit_test.go        | 226 ++++++++++++  (new, test)
 internal/cliproxy/auth/claude_quota_scope_test.go      | 230 ++++++++++++  (new, test)
 internal/cluster/quota_ingestion_claude_test.go        | 177 ++++++++++  (new, test)
 internal/home/claude_usage_result_test.go              | 295 ++++++++++++  (new, test)
 internal/home/claude_usage_result_sanitize_path_test.go|  85 ++++++  (new, test)
 internal/home/usage_result_test.go                     |  13 +        (test helper only)
 10 files changed, 1462 insertions(+), 21 deletions(-)

Four production files, one of them new; the other six are tests. The one change to an existing
test file (usage_result_test.go, +13) adds a single exported test-only helper
(NewUsageResultTestRuntime) so the end-to-end test can live in package home_test;
internal/cluster imports internal/home in production code, so an in-package test importing
cluster would be a genuine import cycle. It is in a _test.go file and is not compiled into
the production binary — happy to restructure if you would rather not have the exported helper.

Deliberately unchanged: internal/cliproxy/auth/selector.go (see Known limitations); no SDK pin
bump; no route, request/response field, or config key added or changed; AGENTS.md untouched.

Codex cannot be affected by change 4. The only downstream consumer of quota_headers is
quotaSnapshotWriteFromUsagePayload (quota_ingestion.go:235), whose first statement is
if provider != "codex" || credentialID == "" { return QuotaSnapshotWrite{}, false } — a Claude
payload can never reach the codex quota-snapshot writer. There is a test asserting this rather than
assuming it.

Testing

Go 1.26.0, clean tree.

check	result
gofmt -l	clean, no output
go vet ./...	one pre-existing warning, internal/cluster/refresh.go:172:2: unreachable code — confirmed present on the base commit, unrelated to this change
go test -count=1 (4 affected packages)	all ok — internal/cluster, internal/cluster/management, internal/home, internal/cliproxy/auth
totals	1102 PASS / 0 FAIL / 15 SKIP (all skips are Postgres-only and pre-existing — no local Postgres)
CGO_ENABLED=1 go build ./cmd/home	succeeds

The four package suites were additionally re-run independently, offline (GOPROXY=off), from a
clean tree — all four ok.

Also verified against the v1.0.72 tag (a4fbe87), not only dev — and re-verified after the
final commit rather than only the first two. All three commits cherry-pick onto the tag with no
conflict; go build ./... and CGO_ENABLED=1 go build -o test-output ./cmd/home both succeed
there, and the same four package suites pass. This matters because v1.0.72 is the release we run,
so the fix is confirmed on the code actually in production, not only on dev.

The tests are not vacuous — each was confirmed to fail when the mechanism it covers is removed:

Each mechanism was individually reverted and the covering test re-run, to prove the test
fails for the right reason rather than passing by agreement with itself:

mechanism reverted	covering test	result with mechanism removed
the claude arm of the sanitizer's provider switch (headers stripped again, i.e. pre-fix behaviour)	TestRecordUsagePayloadHonoursClaudeHeadersThroughRealSanitizePath	FAIL — NextRetryAfter re-armed ~1s out instead of the 4h header reset
the Claude allowlist in sanitizeUsageQuotaHeaders	sanitizer-preserves-allowlist test	FAIL — quota_headers missing
the allowlist's rejection of non-listed keys	sanitizer-rejects-non-allowlisted test	FAIL at the sibling-survives assertion — which is the point: a bare "is Authorization absent?" check alone would have been vacuous

The first row is the one that matters most, and its failure message is the exact defect this PR
exists to fix:

--- FAIL: TestRecordUsagePayloadHonoursClaudeHeadersThroughRealSanitizePath
    ModelStates[claude-opus-4-1].NextRetryAfter = <~1 second from now>,
    want the header reset 2026-09-09T15:57:32Z (through the real sanitize path)

The regression test asserts the deadline equals a reset 4 hours out and explicitly asserts the
delay is not time.Second — it asserts the output is not what the unfixed code would have produced,
not merely that the new code agrees with itself.

One test is worth calling out, because its absence is why causes 1–2 could be fixed and still do
nothing: an end-to-end case that pushes a Claude 429 payload through
cluster.SanitizeUsagePayloadSecrets first and only then into RecordUsagePayload — the same
order handleUsage uses in production. Every pre-existing test calls RecordUsagePayload directly,
which is exactly the code path production does not take.

Heads-up on CI coverage: this repo's PR CI (pr-test-build.yml) runs go test only against
internal/cluster (line 42) and otherwise go builds ./cmd/home (line 45), which doesn't compile
_test.go files — so the new suites in internal/cliproxy/auth and internal/home won't execute
under this PR's CI, even though they were run locally and are green (see table above). Flagging this
so a green check here isn't mistaken for those two packages' tests having run; you may want to widen
the CI test path to cover them, but I've left the workflow file itself untouched in this PR.

Compatibility

• No behaviour change for antigravity or codex — verified by mutation-testing the gate, by
  the unchanged pre-existing suites, and structurally via the default branch returning no hint.
• No API surface change — no route, field, or config key added or changed; no Management API
  documentation impact.
• response_headers / quota_headers remain optional — a payload without them behaves exactly
  as before.
• The sanitizer still strips everything it stripped before. Only an explicitly named allowlist
  of non-secret Anthropic rate-limit headers survives, mirroring the existing codex allowlist. There
  is a negative test asserting a non-allowlisted header does not survive.

Known limitations (disclosed, not fixed here)

1. A model with no ModelState entry is not blocked by a credential-wide window.
   isAuthBlockedForModel (selector.go:340-386) falls through to return false, blockReasonNone, … when the queried model has no entry. Exposure is narrow: a model never dispatched under that
   credential has no state to block, so this bites only when a sibling model is tried for the first
   time during an active credential-wide window — that one request goes through, gets its own 429,
   and now correctly picks up its own reset hint. Closing it means changing selector.go, which
   differs between v1.0.72 and dev; staying out of that file is what lets one change serve both.
   This is a deliberate scope boundary rather than an oversight: result.go and usage_result.go
   stay byte-identical between the v1.0.72 tag and dev, which is what lets this fix apply
   unmodified to both, while selector.go does not, so reaching into it here would stop being a
   minimal, upstream-shaped diff. The gap is self-healing in one request — the sibling model's own
   429 populates the ModelState the credential-wide block was missing, so it is never left
   uncovered a second time.

2. The Anthropic header names are no longer unverified in the way an earlier note here said —
   confirmed by identity with a CPA parser that has itself been observed succeeding against a live
   Anthropic 429 — but the header value formats are still not independently confirmed here. CPA
   (the node binary; the version fetched for this comparison is tag v7.2.154, what the fleet's own
   nodes actually run) defines its Claude rate-limit header names in
   internal/runtime/executor/helps/claude_ratelimit.go. Fetched from that exact tag, it contains
   exactly 9 header literals. This PR's claudeRateLimitHeaderKeys allowlist contains exactly the
   same 9, byte-identical as sorted, case-normalised sets (diff exit 0), including the awkward
   7d_oi spelling — a negative control confirmed the comparison actually discriminates rather than
   trivially matching everything. That CPA parser is the same one behind the
   [claude_ratelimit.go:175] parsed Anthropic rate limit reset headers node log line already quoted
   above under Production evidence — i.e. it has itself been observed successfully parsing a real
   Anthropic 429, not just read as source. Separately, Go's http.CanonicalHeaderKey was run against
   all nine allowlist entries and canonicalizes every one to exactly the key the lookup expects —
   including the underscore in 7d_oi, where the non-letter 7 blocks Go's hyphen-triggered
   capitalization and the _ is passed through untouched. The mixed-case spelling Anthropic actually
   emits, Anthropic-RateLimit-Unified-7d_oi-Status (capital L), canonicalizes onto the same key,
   so the allowlist is casing-agnostic by construction — collect() canonicalizes before every
   lookup. (A positive control, FOO-bar-BAZ → Foo-Bar-Baz, confirms the function rewrites its
   input rather than returning it, so the identity-looking matches are real.)
   What remains genuinely unverified, narrower than before: the header value formats on the
   wire. We have not independently confirmed those here — they remain covered only by that same
   parser's own tolerance for unix-seconds / RFC3339 / HTTP-date, not by a response we captured
   ourselves. To be precise about the evidence: we did not capture an Anthropic response in the
   course of writing this PR; the live-429 confirmation is CPA's own parser succeeding, observed in
   the node's own log.

3. Overlap with fix(auth): fail closed on unbounded credential unavailability #92. That PR also rewrites quota/cooldown/availability logic in result.go and
   is still open. This change is scoped far more narrowly, but the two will conflict there and
   whichever lands second needs a rebase. Happy to rebase onto fix(auth): fail closed on unbounded credential unavailability #92 if you would rather take it
   first.

4. A Claude reset-driven cooldown still consumes one backoff level. On the first 429 with a
   fresh QuotaState, nextQuotaCooldown runs before the ResetAt override applies, so a later
   unrelated hint-less failure resumes from level 1 rather than 0 (2s instead of 1s). Bounded by
   quotaBackoffMax, cleared by any intervening success. Left as-is deliberately — ResetAt
   overrides the deadline, not the level.

Related: #89 (the Codex instance of this defect — left open, not closed by this PR, since this
change does not address the Codex case it describes).