Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -670,3 +670,40 @@ Relief shipped: superadmin `POST /api/admin/project-data/storage/:projectId/grou
(branch `claude/friendly-planck-qziggg`), which prunes grouped/FTS search rows delete-first.
Staging verification was skipped on Raphaël's explicit instruction for this emergency; the
at-cap behaviour is proven by the first bounded production call.

## Reconciliation — 2026-10-05 (weekly queue audit)

**Verdict: stays active, now recovering on its own.** The headline criterion (at or below
9,000,000,000 bytes) is still unmet, but the size is falling for the first time since the breaker
opened on 2026-09-27, and none of the 2026-09-30 next actions is still waiting on a human.

Measured 2026-10-05 (read-only production D1 `sam-prod`):

- `project_data_storage_telemetry`: **9,719,410,688 bytes** at 05:15Z (usage ratio 0.9719, status
`degraded`). That is 90.5% of the hard 10 GiB cap.
- Daily 16:40Z samples from `project_data_storage_telemetry_history`: 10.627 GB (10-01),
10.267 GB (10-02), 10.213 GB (10-03), 9.842 GB (10-04). Overnight 10-05 the hourly samples fall
by 12 to 15 MB an hour. At 250 to 370 MB a day, the 9.0 GB target is about two to three days out.
- `project_data_archive_circuit_breakers`: SAM's breaker is **`closed`** ("Closed from admin UI",
updated 2026-10-02 16:29Z).
- SAM archive migrations: 452 `published`, 47 `frozen`, and **0** `failed`, `poisoned` or in
flight. Publishes per day: 14 (10-02), 40 (10-03), 59 (10-04), 17 by 05:52Z on 10-05. The global
sweep (`archive_sharding_global_sweep`) last finished `succeeded`, with 0 budget stalls.

What changed since 2026-09-30:

- The root object hit the hard cap on 10-02 (see the incident section above). #2215 shipped the
superadmin grouped-FTS wall recovery and freed about 464 MB.
- The failed and poisoned migrations were abandoned from Admin → Storage and the breaker was
closed (10-02 16:28Z to 16:29Z). SAM archives resumed at 16:33Z.
- #2216 slowed the archive sweep to one hour on 10-03 02:50Z. #2220 restored the 18-minute
cadence the same day at 13:27Z, so the #2161 rollback question is settled: keep 18 minutes.

Still open:

1. The headline criterion. Re-measure around 2026-10-08. If the drain flattens before 9.0 GB, the
next lever is Slice C below, not another manual relief call.
2. Slice C (bounded root history indexing) is still unmerged: commit `7868bc894` on
`sam/implement-reliable-projectdata-archiving-tc49jm`, now 217 commits behind `main`.
3. Rebuild grouped FTS rows after the wall recovery:
`tasks/backlog/2026-10-02-rebuild-grouped-fts-after-wall-recovery.md`.
2 changes: 1 addition & 1 deletion tasks/archive/2026-06-15-codex-acp-midprompt-disconnect.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ The ACP peer closes JSON-RPC first: SAM receives `peer disconnected before respo
- [x] Keep unrecoverable failure explicit and terminal.
- [x] Add exact JSON-RPC recoverable race and terminal diagnostic/redaction coverage.
- [x] Phase 5 review round (go, security, constitution, test, docs-sync, task-completion): no CRITICAL/HIGH live bugs. Applied converged go+security fallback tightening (`crashRecovery.sessionID` only on `inProgress`), clarifying comments, and AC-aligned test additions (per-prerequisite terminal diagnostics for acpSessionId/agentType/all-three; captured-session LoadSession identity assertion; agentType partial-clear fallback; `crashRecoveryInProgress==false` on terminal path). Race + full vm-agent suites green.
- [x] Validate mid-prompt disconnect → `LoadSession` recovery on staging (real VM, new binary). Done 2026-07-11 with `claude-code` (agent-agnostic recovery path) because staging `openai-codex` OAuth is revoked (filed `tasks/backlog/2026-07-11-codex-staging-oauth-refresh-token-revoked.md`). Killed the agent process mid-generation; vm-agent logs show the exact `-32603 "peer disconnected before response"` routed to `deferring to crash recovery` → `attempting LoadSession with previous session` → `LoadSession succeeded`; task stayed `in_progress` with `errorMessage=null`, same `agentSessionId`, message count kept climbing (291→350). No terminal failure, no silent stall. Note: this run's goroutine ordering was Prompt-returns-first (live fields present); the specific cleanup-wins race is covered deterministically by unit tests.
- [x] Validate mid-prompt disconnect → `LoadSession` recovery on staging (real VM, new binary). Done 2026-07-11 with `claude-code` (agent-agnostic recovery path) because staging `openai-codex` OAuth is revoked (filed `tasks/archive/2026-07-11-codex-staging-oauth-refresh-token-revoked.md`). Killed the agent process mid-generation; vm-agent logs show the exact `-32603 "peer disconnected before response"` routed to `deferring to crash recovery` → `attempting LoadSession with previous session` → `LoadSession succeeded`; task stayed `in_progress` with `errorMessage=null`, same `agentSessionId`, message count kept climbing (291→350). No terminal failure, no silent stall. Note: this run's goroutine ordering was Prompt-returns-first (live fields present); the specific cleanup-wins race is covered deterministically by unit tests.
- [ ] Complete required reviews (done), CI (green), staging coordination (TURN 2 done), merge, and production monitoring.

### Updated acceptance criteria
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Codex (openai-codex) unusable on staging — OAuth refresh token revoked

> **Reconciliation 2026-10-05: archived as obsolete, not fixed by a dedicated change.** The symptom is gone: on 2026-10-03, fresh `openai-codex` VM and Instant turns completed on staging for the same smoke user (`provider_mode` `sam`) with no auth error, during the #2207/#2217 runtime-distribution staging runs (`scripts/diagnostics/acp-runtime-distribution.md:145-157,268-289`). Read-only staging D1 shows that user's `openai-codex` agent settings at `provider_mode='sam'` (updated 2026-10-03 15:01Z) and an active `oauth-token` credential created 2026-08-21. Moved from `tasks/backlog/` because `tasks/archive/2026-06-15-codex-acp-midprompt-disconnect.md` links to it.

## Problem

On staging (`sammy.party`), an `openai-codex` agent session for the smoke user
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,7 @@ selection and fresh-stock GPT-5.5 rollback now have distinct live evidence.
Timeout/interruption and unsupported-model attempts remain explicitly excluded
from successful continuation claims. No new provider login or token custody was
added. Final rollback and release disposition remain parent-owned.

---

_Reconciled 2026-10-05 (weekly queue reconciliation): shipped via PR #2206 (`989bf7bb6`, merged 2026-09-30T21:43Z), first successful production deploy run 36783707657 (2026-09-30T22:07Z). PR #2217 (`b79136805`) later added the schema-name field labels and the release. Production enablement came with deploy run 37137757826 (2026-10-03T16:41Z), and the `sam-api-prod` readback on 2026-10-05 shows `ACP_INTERACTION_FORMS_ENABLED=true`. Live form continuation is recorded for Codex only, on VM and Instant (`scripts/diagnostics/acp-runtime-distribution.md`). #2217 lists Claude forms as deterministic builder coverage only, yet forms are offered to any agent in conversation mode (`apps/api/src/services/acp-interaction-runtime-config.ts:15`). No boxes left unticked._
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,7 @@ Worker, shared schema, VM, or documentation contract changes are required.
- Draft PR and handoff only; coordinator owns integrated staging and activation.
- No VM, Worker route, shared schema, auth/token custody, form, or URL elicitation changes.
- Do not advertise forms or URL requests.

---

_Reconciled 2026-10-05 (weekly queue reconciliation): draft PR #2200 was closed unmerged (2026-09-30T22:32Z). Its reviewed head `e13a166b5` shipped inside integration PR #2202 (`86e6c5b75`, merged 2026-09-30T16:16Z). That commit added `AcpPermissionCard.tsx`, `useAcpPermissionInteractions.ts` and `apps/web/src/lib/api/acp-interactions.ts`, and deleted `packages/acp-client/src/components/PermissionDialog.tsx`. First successful production deploy run 36744720219 (2026-09-30T16:30Z). Permission creation was enabled in production by deploy run 37137757826 (2026-10-03T16:41Z, #2217 release), and the `sam-api-prod` readback on 2026-10-05 shows `ACP_INTERACTIONS_ENABLED=true`. #2200's only later commit, `fca210a90`, is test-only, and main's audit spec has the retry-control clearance assertions (`apps/web/tests/playwright/acp-permission-chat-audit.spec.ts:524-548`). No boxes left unticked._
Original file line number Diff line number Diff line change
Expand Up @@ -101,3 +101,7 @@ Slice B connects ACP `RequestPermission` to the shipped Cloudflare create/answer
- Ambiguous create acknowledgement does not override an answer already consumed by the runtime; the matching receipt remains duplicate-safe.
- Permission cancellation is owned by the exact prompt attempt. An old attempt's cancellation cannot cancel a newer attempt's permission.
- The real `acp-go-sdk@v0.13.5` connection remains usable after prompt deadline/cancel closes the matching permission.

---

_Reconciled 2026-10-05 (weekly queue reconciliation): draft PR #2201 was closed unmerged (2026-09-30T16:49Z). Its head `96df904f7` shipped inside integration PR #2202 (`86e6c5b75`, merged 2026-09-30T16:16Z), which added `packages/vm-agent/internal/acp/session_host_interactions.go`. No non-test `internal/acp` Go file still selects `Options[0]`. First successful production deploy run 36744720219 (2026-09-30T16:30Z). The parent's release lifted the keep-disabled and unmerged constraints. Production activation came with deploy run 37137757826 (2026-10-03T16:41Z), and the `sam-api-prod` readback on 2026-10-05 shows `ACP_INTERACTIONS_ENABLED=true`. No boxes left unticked._
Original file line number Diff line number Diff line change
Expand Up @@ -207,3 +207,7 @@ selection and fresh-stock GPT-5.5 rollback now have distinct live evidence.
Timeout/interruption and unsupported-model attempts remain explicitly excluded
from successful continuation claims. No new provider login or token custody was
added. Final rollback and release disposition remain parent-owned.

---

_Reconciled 2026-10-05 (weekly queue reconciliation): draft PR #2204 landed through PR #2217 (`b79136805`, merged 2026-10-03T15:59Z). Its head `8a5a1d1ad` is reachable from main, so GitHub marks #2204 merged at the same second, recording `61c796f4b`. First successful production deploy run 37136649002 (2026-10-03T16:23Z); the earlier run 37135306422 failed target validation. Activation: the production Environment overrides `ACP_INTERACTIONS_ENABLED`, `ACP_INTERACTION_FORMS_ENABLED` and `ACP_INTERACTION_URLS_ENABLED` were set to `true` at 16:41:02–05Z, followed by deploy run 37137757826 (workflow_dispatch, 16:41Z). A Cloudflare readback of `sam-api-prod` on 2026-10-05 shows all three `true`. Checked-in defaults remain `false` (`apps/api/wrangler.toml:392-394`, `packages/shared/src/acp-interactions.ts:39-41`). Acceptance evidence: `apps/api/tests/acp-interaction-runtime-config.test.ts:7-32` covers explicit opt-in, and `apps/api/tests/workers/acp-interaction-vertical-slice.test.ts:61` covers reading and answering a pending request after the flag is turned off. The criterion "URL elicitation remains unavailable" was superseded when the parent also enabled URLs in the #2217 release. The old workspace `01M3RBZ3RX4JN084KMNM21A5MT` snapshot is still `degraded/home-skipped` in production D1 and expires 2026-10-07T09:47:49Z; no owner review of it is recorded. No boxes left unticked._
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,9 @@ OpenAI released GPT-6.1 Sol on 2026-09-29. SAM's static OpenAI Codex picker, Ope
- [x] Confirm the provider deprecations check requires no catalog removals.
- [x] Keep the checked OpenCode static fallback synchronized with the live `models.dev` snapshot.
- [x] Run focused and full repository validation. Focused shared tests pass (84/84), the GPT-6.1 proxy vertical-slice test passes, and lint, typecheck, build, and file-size checks pass. The full test run reached 11,059/11,062 before three unrelated API failures under suite load; both affected files pass together in isolation (76/76).
- [ ] Complete specialist review, staging verification, CI, and best-effort CodeRabbit review.
- [ ] Merge and monitor the production deployment, then verify the live production catalog.
- [x] Complete specialist review, staging verification, CI, and best-effort CodeRabbit review.
- _Reconciled 2026-10-05:_ PR #2199 records task-completion-validator, constitution-validator and test-engineer PASS. Deploy Staging run 36688240865 succeeded at `fb9b447a6`. Every non-skipped PR check is SUCCESS. CodeRabbit raised one transport finding at 07:37 UTC; it was fixed in `fb9b447a6` and the thread resolved.
- [ ] Merge and monitor the production deployment, then verify the live production catalog. — not ticked: the merge (#2199 `762a97cf5`) and production deploy run 36697264202 are evidenced, but no production catalog readback is recorded. `/api/model-catalog/*` requires auth (an unauthenticated GET returns 401). The `openai-codex` catalog is served statically (`apps/api/src/services/model-catalog.ts:60-61`) from `packages/shared/src/model-catalog.ts:119`. The OpenCode catalog is dynamic (models.dev), so its live content is unverified.

## Acceptance criteria

Expand All @@ -45,3 +46,7 @@ OpenAI released GPT-6.1 Sol on 2026-09-29. SAM's static OpenAI Codex picker, Ope
- https://models.dev/api.json
- https://developers.cloudflare.com/ai-gateway/integrations/coding-agents/openai-codex/
- `.claude/rules/52-model-catalog-lifecycle.md`

---

_Reconciled 2026-10-05 (weekly queue reconciliation): shipped via PR #2199 (`762a97cf5`, merged 2026-09-30T09:10Z), first successful production deploy run 36697264202 (2026-09-30T09:37Z). One box stays unticked because no authenticated production catalog readback is recorded. Selecting the model in a Codex session failed at ACP startup until PR #2205 (`4e18af6b8`, deploy run 37188274268, 2026-10-04T08:14Z). `tasks/archive/2026-09-30-fix-gpt-6-1-sol-acp-startup.md` records a staging provider HTTP 400 for Codex with a ChatGPT account, and leaves a successful production completion to Raphaël's manual test._
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ The reviewed ACP runtime bridge in PR #2201 and project-chat permission UI in PR
- [x] Confirm normal chat remains functional and review mobile/desktop staging screenshots.
- [x] Delete only integration-owned staging workspaces/nodes immediately and prove zero owned VMs remain at rest.
- [x] Create and maintain a draft integration PR with exact deployed commit, resource IDs, fixtures, requests/results, bounded observability, cleanup, integration fixes, and remaining gaps.
- [ ] Send the final branch/head/PR/CI/staging evidence to parent task `01M3RT1PBZNM57EMC00B7ZXAEK`; do not merge or mark ready.
- [ ] Send the final branch/head/PR/CI/staging evidence to parent task `01M3RT1PBZNM57EMC00B7ZXAEK`; do not merge or mark ready. — not ticked: superseded. No record of a message to `01M3RT1PBZNM57EMC00B7ZXAEK` was found. The coordinating parent became `01M3SG06CFJYF7F6HVJXHTFTN1`. That parent reviewed final head `aecaf205f` (#2202 comment 5914878688, 2026-09-30T15:56Z) and released #2202 under Raphaël's authorization (merged 16:16Z).

## Acceptance criteria

Expand Down Expand Up @@ -103,3 +103,7 @@ Pinned deploy run `36718788997` then failed closed before publishing at the Work
- The two live runtimes prove successful delivery receipts; the deliberately lost/unconfirmed receipt, cancellation/deadline/Stop/process-loss, recreated-generation fencing, feature-off/version-skew, and stopped/stale-running no-wake cases remain deterministic evidence from the named suites rather than staged fault injection. This avoids mislabeling local coverage as deployed proof.
- The preload fixture exercised the pinned Codex ACP process path under SAM `bypassPermissions`/never-full-access configuration and emitted permission requests by design; it is not evidence that actual Codex emits them. No actual Codex account permission emission or actual Claude account emission was proven, so both remain explicit rollout gaps. Claude support here is limited to the reviewed adapter implementation and deterministic SDK fixture coverage.
- After deleting the VM and retained Instant workspace/node/profile through the public API, the authoritative D1 query returned zero nodes outside `deleted`/`failed`. All five temporary staging GitHub Environment overrides were then removed. Restoration deploy `36736535610` republishes the same exact candidate SHA with checked-in `ACP_INTERACTIONS_ENABLED=false`; production was never mutated.

---

_Reconciled 2026-10-05 (weekly queue reconciliation): shipped via PR #2202 (`86e6c5b75`, squash-merged 2026-09-30T16:16Z), first successful production deploy run 36744720219 (2026-09-30T16:30Z). The acceptance criterion that the PR "remains draft and unmerged" was superseded by the parent's release. Checked-in defaults remain `false`. Production permission creation was later enabled by an explicit Environment override (deploy run 37137757826, 2026-10-03T16:41Z). One box is left unticked as superseded because the parent handoff target changed (see that line)._
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,7 @@ selection and fresh-stock GPT-5.5 rollback now have distinct live evidence.
Timeout/interruption and unsupported-model attempts remain explicitly excluded
from successful continuation claims. No new provider login or token custody was
added. Final rollback and release disposition remain parent-owned.

---

_Reconciled 2026-10-05 (weekly queue reconciliation): draft PRs #2209 and #2210 were closed unmerged (2026-10-03T16:00Z). #2209's head `df1ef598a` is not on main. #2210 (head `b7334b6f8`) carried #2209's reviewed changes into PR #2217, merged as `b79136805` on 2026-10-03T15:59Z. That commit added `packages/vm-agent/internal/acp/auth_failure.go`, `session_host_loopback_diagnosis.go`, and the reason codes in `packages/shared/src/failure-classification.ts:115-160`. The creator-gated banners are at `apps/web/src/components/project-message-view/SessionStatusBanners.tsx:43-60`. First successful production deploy run 37136649002 (2026-10-03T16:23Z). Limit: the "parent-owned final runtime matrix" above was not run live. #2217's Limits section says the reason-code tests "include real SDK/HTTP/Worker/UI boundaries, not an assertion that every external account failure was reproduced live." No boxes left unticked._
Original file line number Diff line number Diff line change
Expand Up @@ -162,3 +162,7 @@ selection and fresh-stock GPT-5.5 rollback now have distinct live evidence.
Timeout/interruption and unsupported-model attempts remain explicitly excluded
from successful continuation claims. No new provider login or token custody was
added. Final rollback and release disposition remain parent-owned.

---

_Reconciled 2026-10-05 (weekly queue reconciliation): draft PR #2207 landed through PR #2217 (`b79136805`, merged 2026-10-03T15:59Z). Its head `bbfc13999` is reachable from main, so GitHub marks #2207 merged at the same second. First successful production deploy run 37136649002 (2026-10-03T16:23Z). The acceptance criterion "draft, dormant, unmerged, undeployed" was superseded by the parent's release. URLs were enabled in production by deploy run 37137757826 (2026-10-03T16:41Z), and the `sam-api-prod` readback on 2026-10-05 shows `ACP_INTERACTION_URLS_ENABLED=true`. The checked-in default remains `false` (`apps/api/wrangler.toml:394`). The live wrapper→Go→Worker→browser gap was closed for Codex on a VM (both completion orders) and on Instant (URL `62842f67-…`), per `scripts/diagnostics/acp-runtime-distribution.md`. Claude URL continuation was not run live. Post-timeout tool availability is open in Idea `01M414TM187NXNWF8AFNJZ6181`. No boxes left unticked._
Loading
Loading