fix(api): a slept, restorable session is not conclusive runtime death

[simple-agent-manager]

Summary

Production has been terminalizing restorable task sessions as failed since 2026-08-06 with
"Task runtime is conclusively gone after reconciliation grace (workspace_deleted)." — 31+ tasks,
still firing on 2026-08-17. Their session_snapshots rows were asleep, unexpired, and wakeable for
another seven days at the moment SAM declared them dead.

The chain is composed entirely of individually-correct steps:

1. An agent's ACP turn ends normally (end_turn); the task stays in_progress / awaiting_followup.
2. The session-sleep cron sleeps it after the idle interval — correct, and exactly what the
   "aggressively sleep idle sessions" policy asks for. workspaces.status becomes sleeping;
   session_snapshots.sleep_status='sleeping' with expires_at seven days out.
3. Five minutes later NodeLifecycle runs
   UPDATE workspaces SET status='deleted' ... WHERE status IN ('stopped','sleeping')
   (apps/api/src/durable-objects/node-lifecycle.ts:570-573) — rewriting the inconclusive
   sleeping marker into the conclusive deleted marker.
4. classifyTaskRuntimeLiveness reads workspaces.status, sees deleted, returns
   conclusive: true, and the stuck-task sweep writes failed.

This PR fixes only step 4. The sleep predicate in apps/api/src/services/session-sleep.ts is
deliberately untouched — a parallel task owns it, and sleeping an idle session is correct behaviour.

The core defect

The destroyer and the resumer disagree about what "gone" means.
loadRecoveryContext (apps/api/src/services/session-recovery.ts:85) — the code that actually wakes
a slept session — never reads workspaces.status. A workspace row reading deleted is still
fully wakeable. The classifier's only workspace signal is workspaces.status, and it never read
session_snapshots. .claude/rules/02-quality-gates.md already requires sleep/wake/restore/unknown
to be treated as inconclusive; the classifier violated that.

The fix

Teach the shared classifier the one thing it was missing: whether the session is currently asleep and
restorable. When the workspace row exists but is not running, and a live sleep record exists, the
verdict becomes inconclusive (workspace_<status>_snapshot_resumable) instead of conclusive death.

The resumability predicate mirrors the resumer, plus one bound the resumer lacks:

• a session_snapshots row scoped to the same project_id and workspace_id (rule 11)
• sleeping_at IS NOT NULL — a user delete destroys the snapshot row entirely
  (session-snapshot-persistence.ts:deleteSessionSnapshotState), so snapshot presence discriminates
  idle-sleep from user deletion without needing a new deleted_reason column
• sleep_status = 'sleeping' — asleep now, not a stale marker from a session that already woke
• expires_at parses and is in the future — the bounded escape (rule 47). An absent or
  unparseable expiry counts as NOT resumable, so no task can become immortal.

workspace_missing (row absent) stays conclusive, because loadRecoveryContext requires the
workspace row to exist (session-recovery.ts:91-94). The boundary is chosen to match the resumer exactly.

All three terminalization paths (stuck-task cron, processExpiredCleanups, checkWorkspaceIdleTimeouts)
funnel through this one classifier, so fixing it here fixes all three (DRY) rather than bolting separate
guards into each sweep. Both adapters are wired per rule 44.

Validation

• pnpm lint
• pnpm typecheck
• pnpm test
• Additional validation run (if applicable) — discrimination proof, see below
• If this PR changes candidate selection for a sweep/cron/alarm loop, expected candidate volume and worst-case per-candidate cost are stated

Control-loop I/O budget (rule 47)

This PR does not widen candidate selection — the candidate set is unchanged. It adds at most one
extra D1 point lookup per candidate, gated by needsSessionResumabilityProbe() so it fires only for
a workspace that would otherwise be declared conclusively dead (status !== 'running', not already in
INCONCLUSIVE_WORKSPACE_STATUSES, chat_session_id present, workspace read ok). Steady-state cost is
therefore zero for healthy candidates. The lookup is a point read on the uniquely-indexed
session_snapshots.chat_session_id. Worst case (every candidate about to be terminalized) is one indexed
point read per candidate, bounded by the sweep's existing candidate cap.

Bounded escape (rule 47): a preserved task is not immortal — preservation ends when
session_snapshots.expires_at passes, at which point the task terminalizes normally. Absent/unparseable
expiry → terminal. Covered by tests.

Staging Verification (REQUIRED for all code changes — merge-blocking)

• Staging deployment green — run 32025672280, SHA e1dd5e197 (the review-fixed code)
• Live app verified via Playwright — authenticated on the browser context via token-login
• Existing workflows confirmed working — dashboard, projects, settings, project detail
• New feature/fix verified on staging — full sleep→delete→classify chain reproduced live
• Infrastructure verification completed — N/A: no infra changes (no packages/cloud-init/, packages/vm-agent/, scripts/deploy/, DNS, or TLS paths touched)
• Mobile and desktop verification notes added for UI changes — N/A: no UI changes

Staging Verification Evidence

The incident chain, reproduced live end-to-end

Submitted a real task on staging (01M07RYZX0W1GPHK9CBE351CT2, project hono) and let it run the
exact production sequence on real infrastructure — a real Hetzner VM, the real sleep sweep, the real
NodeLifecycle TTL, and the real stuck-task cron. Verdicts read from
GET /api/admin/tasks/:id/reconciliation-diagnostics, which calls the exact function this PR
changes (getTaskRuntimeLiveness) read-only:

Time (UTC)	workspace	sleep_status	Deployed classifier verdict
11:56	running	–	conclusive=true task_acp_session_live
12:16:50	sleeping	sleeping	conclusive=false workspace_sleeping_resumable
12:22:13	deleted	sleeping	conclusive=false workspace_deleted_snapshot_resumable

12:22:13 is the exact moment the bug used to fire. NodeLifecycle rewrote sleeping →
deleted, and pre-fix the classifier would have returned conclusive=true / workspace_deleted,
causing the sweep to write failed. Instead the task was preserved, and stayed preserved across
~9 minutes of subsequent 5-minute cron sweeps.

Final diagnostics while the workspace was deleted:

{
  "status": "in_progress",
  "executionStep": "awaiting_followup",
  "decision": "preserve_inconclusive_runtime",
  "eligible": true,
  "liveness": {
    "live": false,
    "conclusive": false,
    "reason": "workspace_deleted_snapshot_resumable",
    "workspaceStatus": "deleted"
  }
}

eligible: true is the load-bearing part — the task was past the reconciliation grace threshold, so
the sweep genuinely evaluated it and chose preserve_inconclusive_runtime. It was not merely
skipped. error_message stayed NULL throughout. The running row is the owner-path control: a
healthy workspace is never diverted onto the resumability branch.

The bug's fingerprint is present on staging

Staging D1 independently shows the untreated shape: 15 snapshots with sleep_status='sleeping',
all 196 workspaces at status='deleted', and 11 tasks historically failed with
workspace_deleted — three of whose snapshots were available/unexpired at kill time (one with
recovery_attempts=0, expires_at six days out). Those are prior victims of exactly this defect.

Regression pass (Playwright, staging)

token-login → HTTP 200, then walked the core surfaces. All render real data (asserted on live
project names, not just chrome — an early run sampled at 3s and caught the pre-hydration shell, so
the check now waits for the projects list to populate):

Surface	HTTP	Data rendered	Horizontal overflow	New console errors
dashboard	200	hono, Potato, Test Project, serverspresentation	none	0
projects	200	hono, Potato, Test Project, serverspresentation	none	0
settings	200	hono, Potato, Test Project, serverspresentation	none	0
project detail	200	hono, Potato, Test Project, serverspresentation	none	0

Mobile (375×667) dashboard: no horizontal overflow. Total console errors across the run: 0.
Screenshots in .codex/tmp/playwright-screenshots/pr1844-*.png.

Cleanup

Node 01M07RZ3WTVJWJ81WFTF4MMFBS deleted immediately after verification; staging D1 confirms
zero non-deleted nodes, so no Hetzner capacity is held from the shared limit. The test task was
transitioned to cancelled (not failed), leaving no misleading lifecycle record.

UI Compliance Checklist (Required for UI changes)

N/A: no UI changes — this PR touches only apps/api/src/, apps/api/tests/, .claude/rules/, and tasks/.

End-to-End Verification (Required for multi-component changes)

• Data flow traced from user input to final outcome with code path citations
• Capability test exercises the complete happy path across system boundaries
• All spec/doc assumptions about existing behavior verified against code (not just "read the code")
• If any gap exists between automated test coverage and full E2E, manual verification steps documented below

Data Flow Trace

1. Agent's ACP turn ends normally (end_turn)
   -> packages/vm-agent ... -> recordTurnEnd -> session_state.activity = 'idle'
   -> task stays in_progress / execution_step='awaiting_followup'

2. Session-sleep cron finds the session eligible and sleeps it
   -> apps/api/src/scheduled/session-sleep.ts:221 runSessionSleepSweep()
   -> apps/api/src/services/session-sleep.ts:195 isActivitySafeForSleep() === true
   -> sleepWorkspaceSession(): workspaces.status='sleeping',
      session_snapshots.sleep_status='sleeping' + sleeping_at + expires_at (+7d)
   -> arms NodeLifecycle.scheduleWorkspaceDeletion (session-sleep.ts:610-621)
   [UNCHANGED BY THIS PR — parallel task owns this path]

3. 5 min later (WORKSPACE_STOPPED_TTL_MS) the NodeLifecycle alarm rewrites the marker
   -> apps/api/src/durable-objects/node-lifecycle.ts:570-573
      UPDATE workspaces SET status='deleted' ... WHERE status IN ('stopped','sleeping')
   [UNCHANGED BY THIS PR]

4. Stuck-task cron classifies the runtime  <-- THE STEP THIS PR FIXES
   -> apps/api/src/scheduled/stuck-tasks.ts:439 getTaskRuntimeLiveness()
   -> NEW: needsSessionResumabilityProbe() gates a single point lookup
   -> NEW: loadSessionResumabilitySnapshot(db, projectId, workspaceId, chatSessionId)
           (apps/api/src/services/task-runtime-liveness.ts:365)
   -> classifyTaskRuntimeLiveness() (task-runtime-liveness.ts:176-202)
      BEFORE: status='deleted' -> { conclusive:true, reason:'workspace_deleted' }
      AFTER:  live snapshot    -> { conclusive:false, reason:'workspace_deleted_snapshot_resumable' }
   -> stuck-tasks.ts:988 `conclusive && !live` no longer fires -> task preserved

4'. Same fix reaches the ProjectData DO idle sweeps via the second adapter
   -> apps/api/src/durable-objects/project-data/task-runtime-liveness.ts:57
      getLocalTaskRuntimeLiveness() (feeds processExpiredCleanups + checkWorkspaceIdleTimeouts)

5. The session remains restorable by the resumer, which was always willing to wake it
   -> apps/api/src/services/session-recovery.ts:85 loadRecoveryContext()
      (requires snapshot.workspaceId && projectId match && sleepingAt — never reads workspaces.status)

Untested Gaps

N/A: full flow covered by automated tests for the classifier boundary, with one nuance recorded for
reviewers: the cross-workspace SQL scoping predicate is defended at two layers — the
AND workspace_id = ? clause in loadSessionResumabilitySnapshot() and an in-memory
snapshot.workspaceId !== workspaceId guard in isSessionResumable() (rule 28 defence-in-depth). When
the SQL predicate is deleted, the cross-workspace test still passes because the in-memory guard catches
it; the cross-project test does go red. Both layers are intentional and both are exercised; the
in-memory guard is what the cross-workspace assertion discriminates on.

Discrimination proof (rule 58 requires this to be verified once)

Simulated pre-fix code by neutralizing the resumability branch and re-ran the suite:

× does not terminalize a slept session with a live snapshot (incident 8bd22a42)
× treats a degraded-but-restorable snapshot as resumable (incident da90b7c4)
× withholds a death verdict when the resumability probe failed
× does not declare a slept, restorable session conclusively dead
× preserves a degraded snapshot the recovery path would still restore
× applies the same protection to a stopped workspace
× preserves a slept, restorable session instead of terminalizing it
× withholds a death verdict when the snapshot read fails
Test Files 2 failed (2) | Tests 8 failed | 39 passed (47)

Exactly the 8 incident/preserve assertions go red; all 39 controls stay green — including
user-deleted-no-snapshot, expired snapshot, already-woke, workspace_missing, cross-project,
cross-workspace, and running-workspace. That proves the suite is not simply asserting
"terminalization is disabled".

The seven guards added during review were verified the same way — removing the
isRestorableSnapshot, recoveryAttempts, and in-memory projectId checks turns exactly those
seven cases red (7 failed | 51 passed) and nothing else.

Separately, deleting the AND project_id = ? scoping predicate turns
ignores a snapshot belonging to a different project red, proving that predicate discriminating
against a real SQL engine (better-sqlite3 + createSqliteD1 + createSchemaTables, rule 28 —
not a .where()-ignoring mock).

Real-schema verification (staging D1, read-only)

Local tests build tables from the drizzle definitions, which cannot catch drift against the
applied migrations. Verified directly against staging D1 per .claude/rules/32:

• PRAGMA table_info(session_snapshots) — every column the query reads exists and is TEXT
  (so Date.parse on ISO strings is correct), with expires_at NOT NULL
• the exact production query executed successfully with dummy binds
• idx_session_snapshots_chat_session_id is a UNIQUE index, confirming the lookup is a true
  point read and substantiating the I/O-budget claim above

Post-Mortem (Required for bug fix PRs)

What broke

Task-mode sessions were terminalized as failed with
"Task runtime is conclusively gone after reconciliation grace (workspace_deleted)." while their work
was intact and restorable. Users saw a red "Task failed" on work that had merely gone to sleep. 31+ tasks
since 2026-08-06, still firing on 2026-08-17 — including the SAM task that was originally writing this
very fix, which was killed mid-flight by the bug it was fixing.

Root cause

classifyTaskRuntimeLiveness decided recoverability from workspaces.status alone, while
loadRecoveryContext — the code that actually wakes a slept session — decides from
session_snapshots.sleeping_at and never reads workspaces.status at all. NodeLifecycle rewrites a
slept workspace's sleeping status to deleted five minutes after sleep
(node-lifecycle.ts:570-573, predicate status IN ('stopped','sleeping')), collapsing the one
inconclusive marker the classifier understood into a conclusive one. From that moment the classifier and
the resumer disagreed, and the classifier won.

Class of bug

Destroyer/resumer signal divergence — two subsystems answering "is this work still recoverable?" from
different records, with a third path (a TTL sweep) mutating only the one the destroyer reads. Not covered
by the existing rules, which addressed cleanup paths disagreeing with each other rather than cleanup
disagreeing with recovery.

Why it wasn't caught

• Every component was individually correct. The sleep cron is activity-aware and refuses to sleep a
  prompting session. Both DO idle sweeps route through the shared classifier and preserve unless it says
  dead. The classifier's logic is sound given its inputs. There was no single wrong line to find in review.
• The one shared classifier gave false confidence. .claude/rules/02 already required a single shared
  lifecycle classifier, and SAM had one. That guaranteed the cleanup paths agreed with each other — and
  said nothing about whether they agreed with the resumer.
• Sleep-then-classify was never tested as a sequence. Tests covered sleep, and covered classification,
  but no test slept a session and then asked the classifier about it. The existing workspace_deleted
  assertions all used no-snapshot fixtures, so they encoded the buggy verdict as expected behaviour.
• The symptom read as a different bug. "Agent was working and got reaped" points at the idle detector.
  The idle detector was innocent; only the production D1 audit trail (session_snapshots rows still
  sleeping/unexpired at kill time) disproved that framing.

Process fix included in this PR

• .claude/rules/58-terminal-verdicts-must-match-the-resumer.md (new): any verdict that work is
  unrecoverable must be derived from the record the resumer requires, must name that resumer function in a
  comment, must be bounded by the artifact's own retention (absent bound → terminal, so nothing becomes
  immortal), must withhold the verdict when the lookup fails, must keep the probe off the hot path, and
  must ship both an incident reproduction and a discriminating control proving terminalization still fires.
• .claude/rules/02-quality-gates.md amended to state that a shared lifecycle classifier is necessary
  but not sufficient, pointing at rule 58.

Post-mortem file

tasks/active/2026-08-17-fix-slept-session-classified-as-dead.md (moves to tasks/archive/ in this PR),
including the full production D1 audit trail for both reported kills.

Specialist Review Evidence (Required for agent-authored PRs)

• All local reviewers completed and findings addressed before merge
• If any reviewer did NOT complete: needs-human-review label added and merge deferred to human — N/A, all three completed

Reviewer	Status	Outcome
cloudflare-specialist	ADDRESSED	No CRITICAL/HIGH. D1 query, index coverage, I/O budget, error handling, DO concurrency all PASS. One MEDIUM fixed in e1dd5e1: the predicate mirrored only loadRecoveryContext, not the real authorizer claimSessionSnapshotRecovery.
test-engineer	ADDRESSED	No CRITICAL/HIGH. Three MEDIUMs + two LOWs fixed in e1dd5e1: non-discriminating degraded-snapshot tests, a weak running-workspace control, the cron adapter's untested probe-error path, missing in-memory projectId re-check, and unparseable-expiry only covered at the pure-function boundary.
task-completion-validator	ADDRESSED	Checks A/B/D/E/F PASS; C was open only on staging (now done). Independently reproduced the discrimination proof. Its MEDIUM (rule 58 referencing tasks/archive/ while the file sat in tasks/active/) fixed by archiving the task file in e1dd5e1.

What review changed (all fixed in-branch, nothing deferred)

The most valuable finding was that the first implementation was looser than the resumer, not
just tighter. loadRecoveryContext merely assembles context; the function that actually authorizes
a wake is claimSessionSnapshotRecovery
(apps/api/src/services/session-snapshot-recovery-lifecycle.ts), whose WHERE also requires a
restorable status/degradation pair and recovery_attempts < max. Mirroring only the first
function meant a snapshot whose wake attempts were exhausted would be preserved for the full
7-day TTL waiting on a wake that can never happen — the inverse failure to the original bug.

isSessionResumable now mirrors the claim exactly:

• shared isRestorableSnapshot helper, extracted so the SQL predicate and its in-memory twin
  cannot drift apart
• env-configurable SESSION_SNAPSHOT_RECOVERY_MAX_ATTEMPTS ceiling threaded through both
  adapters — a second bounded escape alongside expires_at
• in-memory projectId re-check, so both scoping predicates now have the SQL + in-memory pair
  rule 28 asks for

The predicate now lines up 1:1 with claimSessionSnapshotRecovery's own unavailability reasons
(snapshot_missing, snapshot_expired, snapshot_not_complete, recovery_attempts_exhausted).

Rule 58 gained a "find the whole resumer before you mirror it" section, since this failure mode
is the natural next mistake after the one the rule was written for.

Exceptions (If any)

• Scope: none
• Rationale: n/a
• Expiration: n/a

Agent Preflight (Required)

• Preflight completed before code changes

Classification

• external-api-change
• cross-component-change
• business-logic-change
• public-surface-change
• docs-sync-change
• security-sensitive-change
• ui-change
• infra-change

External References

N/A: no external API surface consulted — the change is entirely internal to SAM's own task-liveness
classifier. Grounding evidence came from production Cloudflare D1 (sam-prod, sam-observability-prod)
queried directly per .claude/rules/32-cf-api-debugging.md: task_status_events, session_snapshots,
and workspaces rows for the two reported kills, plus the 31-task workspace_deleted audit. In-repo
official documentation consulted: .claude/rules/02-quality-gates.md, .claude/rules/47,
.claude/rules/53, .claude/rules/57, .claude/rules/44, .claude/rules/28, .claude/rules/35.

Codebase Impact Analysis

• apps/api/src/services/task-runtime-liveness.ts — new shared resumability signal
  (SessionResumabilitySnapshot, isSessionResumable, needsSessionResumabilityProbe,
  loadSessionResumabilitySnapshot) and the new inconclusive branch in classifyTaskRuntimeLiveness
• apps/api/src/scheduled/stuck-tasks.ts — adapter 1 (getTaskRuntimeLiveness) wired
• apps/api/src/durable-objects/project-data/task-runtime-liveness.ts — adapter 2
  (getLocalTaskRuntimeLiveness, feeding processExpiredCleanups + checkWorkspaceIdleTimeouts) wired
• apps/api/tests/unit/services/task-runtime-liveness.test.ts — classifier unit coverage
• apps/api/tests/unit/stuck-task-slept-session-liveness.test.ts — new vertical slice against a real
  SQL engine for both adapters
• .claude/rules/58-terminal-verdicts-must-match-the-resumer.md, .claude/rules/02-quality-gates.md — process fix
• tasks/archive/2026-08-17-fix-slept-session-classified-as-dead.md — task record + post-mortem

Read-but-unmodified (boundary of the change): apps/api/src/services/session-recovery.ts,
apps/api/src/services/session-sleep.ts, apps/api/src/scheduled/session-sleep.ts,
apps/api/src/durable-objects/node-lifecycle.ts. No changes to packages/, apps/web, scripts/,
infra/, or specs/.

Documentation & Specs

.claude/rules/58-terminal-verdicts-must-match-the-resumer.md (new) and .claude/rules/02-quality-gates.md
(amended). No apps/www/src/content/docs/ change is required: this fixes an internal control-loop verdict
and changes no user-facing or self-hoster-facing behaviour, configuration, env var, or API contract.

Constitution & Risk Check

• Principle XI (No Hardcoded Values) — no new magic values. The preservation bound reuses the existing
  per-row session_snapshots.expires_at, which is written from the already-env-configurable snapshot TTL;
  this PR introduces no new timeout, limit, URL, or identifier.
• Principle XIII (Fail Fast / fail closed) — inverted deliberately and correctly here: the irreversible
  action is terminalization, so an unknown answer (resumabilityProbeOutcome === 'error') withholds the
  death verdict rather than proceeding. Rule 58 requirement 4.
• Risk: over-preservation / immortal tasks. Mitigated by the expires_at bounded escape, with
  absent/unparseable expiry treated as terminal. Covered by tests and by the discriminating control set.
• Risk: masking genuine failures. Mitigated by keeping workspace_missing conclusive and by the
  user-delete control test — a user-deleted workspace (snapshot row destroyed) terminalizes exactly as before.
• Risk: extra D1 load in a control loop. Mitigated by needsSessionResumabilityProbe() gating; zero
  extra reads for healthy candidates.

[codspeed-hq]

Merging this PR will not alter performance

✅ 6 untouched benchmarks

---

_{Comparing sam/fix-production-workspace-reaping-0t2e0e (61cb2ed) with main (1b89bf5)}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud