Skip to content

fix: wait for console asset version overrides to converge - #323

Merged
joshavant merged 1 commit into
mainfrom
fix/production-asset-gate
Oct 23, 2026
Merged

joshavant merged 1 commit into
mainfrom
fix/production-asset-gate

Conversation

@joshavant

@joshavant joshavant commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The production console-asset gate fails immediately when a version override still selects another runtime, even if the preceding health check has converged. Its generic error also discards the failing asset path, HTTP status and observed version.

Retry only successful asset GETs that identify a different valid runtime, within the existing 30-second per-asset budget. Require the requested runtime and exact bytes before passing. HTTP errors, missing or malformed version proof, corrupt bytes and transport failures remain fatal. Add metadata-only failure diagnostics that flow into the existing rollback receipt; exclude response content and transport error text.

The failed scheduled run 39986461332 showed an earlier health override temporarily selecting blue, but its retained asset error cannot establish the incident's cause. This change covers the documented propagation behavior and makes a subsequent failure diagnosable.

Verification

  • node --test test/deploy-*.test.mjs test/smoke-readiness*.test.mjs: 72 passed.
  • PR CI passed all seven jobs; all reported PR checks succeeded for the reviewed head.
  • Regression coverage includes transient and persistent version mismatch, HTTP failures, missing or malformed version proof, corrupt and stalled bodies, diagnostic privacy, and controller rollback at zero normal traffic.
  • Read-only production check: health HTTP 200; all eight failed-run assets pass exact version, byte-length and SHA-256 checks through the serving blue version.
  • Independent Codex autoreview: scoped-clean at P0–P2 for commit d0b534c01290a851b9fd623b3c3eec17bcab1b61.

Deployment status and remaining risk

Merged as ebde1ab2f865dc5d4d90dc8b5f8a504dfacbfdef. Exact-commit main CI passed all seven jobs; CodeQL also passed.

The separately authorized production zero-smoke rehearsal passed for that merged SHA:

  • Eight assets verified through both version-pinned paths; mandatory OpenAI smoke and caller-visible usage verification passed.
  • Independent Cloudflare reads observed candidate d2aeb79f-a83f-4c15-bf08-5c90c289da83 at 0% and original version e6de8adf-5c99-494f-b5dd-a24a96a8d8f5 at 100%. The run's two deployment records show that split followed by the original version alone at 100%.
  • Final read-only reconciliation confirmed the original version, health HTTP 200, all eight assets, zero usage/dead-letter backlog, no active observer tails, and accepted secret lineage.

The uploaded candidate remains in version history and the immutable shared assets remain published. No full gradual rollout was performed. Zero configured normal traffic still permits synthetic/shared-state activity.

The original incident's exact cause remains unproven. This rehearsal needed one health readiness retry and no asset-version retries, so it does not reproduce the old asset failure; focused regression tests cover asset convergence and persistent failure. Application admin access was not needed for this investigation or rehearsal.

@clawsweeper

clawsweeper Bot commented Oct 23, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

@joshavant
joshavant merged commit ebde1ab into main Oct 23, 2026
14 checks passed
@joshavant
joshavant deleted the fix/production-asset-gate branch October 23, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant