fix(api): a slept, restorable session is not conclusive runtime death

.claude/rules/02-quality-gates.md

@@ -95,7 +95,7 @@ Ask: "What test, if it existed before the breaking change was introduced, would
 - **If the bug involves cancellation through a timeout, retry, poll, or multi-step resource boundary**, the regression must prove the caller signal is composed with (not replaced by) internal timeouts, the exact caller reason survives provider-specific error handling, listeners/timers are cleaned up, and no later retry, poll, cleanup, cache write, or resource mutation starts after cancellation. Include a cancellation reason shaped like an otherwise retryable or idempotent domain error so classification catches cannot swallow it.
 - **If the bug involves streamed UI data that is later reconstructed from durable storage**, write a parity regression test for the persisted representation, not only the live stream. The test MUST include a partial/status-only update event and assert omitted fields do not clear previously visible metadata. See the retained incident lesson in this rule.
 - **If the bug involves lifecycle control across a runtime boundary** (agent/session/workspace/node stop, cancel, retry, replacement, suspend, or resume), the regression test MUST assert the runtime command is invoked before accepting the terminal state or dispatching replacement work. Database state changes and successful JSON responses are insufficient; the test must prove the external agent/node/workspace control side effect.
-- **If runtime liveness is represented in more than one control plane** (for example D1, a session Durable Object, and a runtime Durable Object), timeout and cleanup tests MUST cross those boundaries with deliberately stale secondary state. A heartbeat timeout or sweep may terminalize work only after the runtime owner reports a conclusively terminal lifecycle; sleep, wake, restore, replacement, probe failure, and unknown state are inconclusive. Use one shared lifecycle classifier for every cleanup path so a stale replica cannot strand recoverable work.
+- **If runtime liveness is represented in more than one control plane** (for example D1, a session Durable Object, and a runtime Durable Object), timeout and cleanup tests MUST cross those boundaries with deliberately stale secondary state. A heartbeat timeout or sweep may terminalize work only after the runtime owner reports a conclusively terminal lifecycle; sleep, wake, restore, replacement, probe failure, and unknown state are inconclusive. Use one shared lifecycle classifier for every cleanup path so a stale replica cannot strand recoverable work. A shared classifier is necessary but NOT sufficient: it must also derive its verdict from the record the recovery path actually reads, or it will confidently declare restorable work dead while every cleanup path agrees with it. See `.claude/rules/58-terminal-verdicts-must-match-the-resumer.md`.
 - **Inactivity is never successful completion evidence.** An idle, timeout, or cleanup sweep MUST NOT write `completed`; success requires an explicit successful task/runtime transition. If a shared liveness classifier proves the runtime conclusively dead, the sweep may write `failed` only with diagnostic context, a system `task_status_events` row, and failed trigger-execution synchronization. Gate workspace deletion on that same conclusive-death result; live and inconclusive tasks and workspaces remain intact.
 - **If the bug involves shell or process execution lifecycle** (process groups, child processes, cancellation, timeout, or cleanup after command completion), the regression test MUST cover the success path as well as failure/cancellation paths and prove spawned children are not left alive after the tool or command returns.
 - **If the bug involves a utility LLM call through a provider-compatible API**, the regression test MUST assert the exact provider payload controls that make the response contract reliable, not just the returned parsed text. For reasoning-capable models, this includes any explicit thinking/reasoning-disable parameters or response-format controls required for the utility to receive text in the field it reads.

.claude/rules/58-terminal-verdicts-must-match-the-resumer.md

@@ -0,0 +1,143 @@
+# A "Work Is Unrecoverable" Verdict Must Read The Same Record The Resumer Reads
+
+## When This Applies
+
+Any code that writes a **terminal verdict about recoverability** — "this runtime is
+conclusively gone", "this session is dead", "this job cannot be retried", "this workspace
+is unreachable" — while a **separate code path elsewhere can still restore that work**
+from a snapshot, checkpoint, replica, backup, or queued replay.
+
+The canonical pair in this repo:
+
+| Role      | Function                                                                         | Signal it reads                 |
+| --------- | -------------------------------------------------------------------------------- | ------------------------------- |
+| Destroyer | `classifyTaskRuntimeLiveness` (`apps/api/src/services/task-runtime-liveness.ts`) | `workspaces.status`             |
+| Resumer   | `loadRecoveryContext` (`apps/api/src/services/session-recovery.ts`)              | `session_snapshots.sleeping_at` |
+
+**Find the whole resumer before you mirror it.** The restore path is usually more than one
+function, and the one that reads most naturally as "the resumer" is often not the one that
+actually authorizes the restore. Here `loadRecoveryContext` merely assembles context; the real
+gate is `claimSessionSnapshotRecovery`
+(`apps/api/src/services/session-snapshot-recovery-lifecycle.ts`), whose `WHERE` clause adds a
+restorable `status`/`degradation` pair and `recovery_attempts < max`. Mirroring only the
+first function leaves the destroyer *looser* than the resumer — the opposite failure to the
+original bug, and just as real: work the resumer will never wake is preserved anyway, so the
+task hangs until the artifact's TTV expires instead of failing promptly. Enumerate every
+predicate on the path from "candidate" to "restored" and mirror the union.
+
+## Why This Rule Exists
+
+On 2026-08-16, 31+ production tasks (`2026-08-06` onward, still firing on `2026-08-17`)
+were terminalized as
+`"Task runtime is conclusively gone after reconciliation grace (workspace_deleted)."`
+while their sessions were asleep, unexpired, and fully restorable for another seven days.
+
+The chain was entirely composed of individually-correct steps:
+
+1. An agent's ACP turn ended normally (`end_turn`). The task stayed `in_progress` /
+   `awaiting_followup`.
+2. The session-sleep cron slept it after the idle interval — **correct**, and exactly what
+   the "aggressively sleep idle sessions" policy asks for. A snapshot was captured;
+   `workspaces.status` became `sleeping`; `session_snapshots.sleep_status='sleeping'` with
+   an `expires_at` seven days out.
+3. Five minutes later `NodeLifecycle` ran
+   `UPDATE workspaces SET status='deleted' ... WHERE status IN ('stopped','sleeping')`
+   (`node-lifecycle.ts:570-573`) — **rewriting the inconclusive `sleeping` marker into the
+   conclusive `deleted` marker.**
+4. The classifier read `workspaces.status`, saw `deleted`, and returned
+   `conclusive: true`. The stuck-task sweep wrote `failed`.
+
+Every step was locally defensible. The system was still wrong, because **the destroying
+side and the restoring side were reading different records.** `loadRecoveryContext` never
+reads `workspaces.status` at all — a `deleted` workspace row was, and remains, perfectly
+wakeable. Nothing forced the two to agree, and nothing tested them as a pair.
+
+Note that a shared classifier already existed, as `.claude/rules/02` requires. A single
+shared classifier is necessary but **not sufficient**: it prevented the _cleanup paths_
+from disagreeing with each other, while leaving the classifier free to disagree with the
+resumer.
+
+## Class of Bug
+
+**Destroyer/resumer signal divergence.** Two subsystems answer the same question — "is this
+work still recoverable?" — from different columns, and a third path (here, a TTL sweep)
+mutates the column only one of them reads. The failure is invisible in isolation: the
+classifier's logic is correct given its inputs, the resumer's logic is correct given its
+inputs, and the TTL sweep is doing its documented job.
+
+Tells:
+
+- A terminal verdict derived from a **status/lifecycle enum** rather than from the artifact
+  that actually enables recovery.
+- A TTL, GC, or retention sweep whose predicate spans an inconclusive state
+  (`WHERE status IN ('stopped','sleeping')`) and collapses it into a terminal one.
+- A recovery path whose precondition set is _narrower_ than the destroy path's — i.e. it can
+  restore things the destroyer already declared dead.
+
+## Hard Requirements
+
+1. **Derive the terminal verdict from the recovery precondition, not from a status enum.**
+   Before writing "unrecoverable", read the same record the resumer requires. If the resumer
+   would accept it, the verdict must be **inconclusive**.
+
+2. **Mirror the resumer's predicate explicitly, and say so in a comment naming the
+   function.** When the destroyer's predicate is deliberately _stricter_ than the resumer's,
+   the extra condition must be justified in that comment (in the canonical fix the only
+   addition is an expiry bound; see requirement 3).
+
+3. **Every "preserve" verdict needs a bounded escape** (`.claude/rules/47`). Preserving
+   recoverable work must not create an immortal task. Bound it on the artifact's own
+   retention (`expires_at`), and treat an **absent or unparseable** bound as _not_
+   recoverable so a malformed row cannot pin work open forever. The bound must be
+   env-configurable with a `DEFAULT_*` constant.
+
+4. **A failed recoverability lookup withholds the terminal verdict.** The destructive action
+   is the irreversible one, so an unknown answer must not resolve to "destroy".
+
+5. **Do not add a `*_reason` / `*_cause` column just to tell the causes apart** when an
+   existing artifact already discriminates. In the canonical fix, snapshot _presence_ is the
+   discriminator: a user-initiated delete destroys the snapshot row
+   (`session-snapshot-persistence.ts:deleteSessionSnapshotState`), an idle sleep keeps it.
+   Prefer the record that already exists over new schema.
+
+6. **Keep the lookup off the hot path** (`.claude/rules/47`). Probe only for candidates that
+   would otherwise be terminalized, so a control loop pays the extra read only when it is
+   about to take the destructive action.
+
+## Required Tests
+
+- **The incident, reproduced**: the artifact is recoverable, the status enum says dead →
+  assert **inconclusive**. Must FAIL against the pre-fix code; verify that once.
+- **The discriminating control**: same status enum, artifact genuinely absent → assert the
+  terminal verdict still fires. Without this, a test suite passes equally well if
+  terminalization were disabled outright.
+- **The bound**: expired artifact → terminal. Absent/unparseable bound → terminal.
+- **Scoping predicates against a real SQL engine** (`.claude/rules/28`): cross-tenant and
+  cross-resource fixtures, each proven discriminating by deleting the predicate.
+- **Every adapter** that feeds the classifier supplies the new signal
+  (`.claude/rules/44` — enumerate them; a signal wired into one adapter and not another
+  reintroduces the bug on the unwired path).
+
+## Quick Compliance Check
+
+- [ ] The terminal verdict reads the resumer's own record, not just a status enum
+- [ ] A comment names the resumer function the predicate mirrors
+- [ ] Any extra strictness vs. the resumer is justified in that comment
+- [ ] Preserve verdicts are bounded by an env-configurable retention; absent bound → terminal
+- [ ] A failed recoverability lookup withholds the terminal verdict
+- [ ] The probe fires only for otherwise-doomed candidates
+- [ ] Incident reproduction + discriminating control both exist, and the reproduction was
+      verified to fail pre-fix
+
+## References
+
+- Task: `tasks/archive/2026-08-17-fix-slept-session-classified-as-dead.md`
+- `.claude/rules/02-quality-gates.md` — "sleep, wake, restore, replacement, probe failure,
+  and unknown state are inconclusive"; one shared lifecycle classifier
+- `.claude/rules/47-control-loop-io-budget.md` — bounded escape paths, I/O budget
+- `.claude/rules/53-scheduled-handler-isolation-and-liveness-signals.md` — a signal that
+  cannot answer the question being asked of it
+- `.claude/rules/57-write-only-cross-boundary-state.md` — reconcile, don't just report
+- `.claude/rules/44-dual-write-migration-enumerate-writers.md` — enumerate every adapter
+- `.claude/rules/28-credential-resolution-fallback-tests.md` — SQL predicates need a real
+  SQL engine

apps/api/src/durable-objects/project-data/task-runtime-liveness.ts

@@ -6,9 +6,12 @@ import {
 
 import type { Env as WorkerEnv } from '../../env';
 import { createModuleLogger } from '../../lib/logger';
+import { DEFAULT_SESSION_SNAPSHOT_RECOVERY_MAX_ATTEMPTS } from '../../services/session-snapshot-artifacts';
 import {
   classifyTaskRuntimeLiveness,
   loadRuntimeWorkspaceSnapshot,
+  loadSessionResumabilitySnapshot,
+  needsSessionResumabilityProbe,
   type RuntimeAcpSessionSnapshot,
   type TaskRuntimeLiveness,
   type TaskRuntimeLivenessSignals,
@@ -55,7 +58,32 @@ export async function getLocalTaskRuntimeLiveness(
     }
   }
 
+  // Only probed for a workspace that would otherwise be declared conclusively
+  // dead, keeping this off the alarm's hot path (`.claude/rules/47`).
+  let resumabilityProbeOutcome: TaskRuntimeLivenessSignals['resumabilityProbeOutcome'] = 'not_run';
+  let sessionResumability: TaskRuntimeLivenessSignals['sessionResumability'] = null;
+  if (needsSessionResumabilityProbe(workspace, workspaceProbeOutcome)) {
+    try {
+      sessionResumability = await loadSessionResumabilitySnapshot(
+        env.DATABASE,
+        task.projectId,
+        workspace.id,
+        workspace.chatSessionId
+      );
+      resumabilityProbeOutcome = 'ok';
+    } catch (err) {
+      resumabilityProbeOutcome = 'error';
+      log.warn('session_resumability_query_failed', {
+        projectId: task.projectId,
+        workspaceId: task.workspaceId,
+        action: 'preserved',
+        error: err instanceof Error ? err.message : String(err),
+      });
+    }
+  }
+
   const baseSignals: TaskRuntimeLivenessSignals = {
+    projectId: task.projectId,
     taskWorkspaceId: task.workspaceId,
     workspace,
     workspaceProbeOutcome,
@@ -65,6 +93,12 @@ export async function getLocalTaskRuntimeLiveness(
     acpSessions: [],
     containerProbeOutcome: 'not_run',
     containerLifecycle: null,
+    resumabilityProbeOutcome,
+    sessionResumability,
+    resumabilityMaxRecoveryAttempts: positiveInt(
+      env.SESSION_SNAPSHOT_RECOVERY_MAX_ATTEMPTS,
+      DEFAULT_SESSION_SNAPSHOT_RECOVERY_MAX_ATTEMPTS
+    ),
   };
   const initialClassification = classifyTaskRuntimeLiveness(baseSignals);
   if (

apps/api/src/durable-objects/project-data/types.ts

@@ -47,6 +47,7 @@ export type Env = {
   SESSION_ACTIVITY_PROBE_TIMEOUT_MS?: string;
   SESSION_ACTIVITY_PROBE_MAX_ATTEMPTS?: string;
   SESSION_ACTIVITY_PROBE_MAX_CANDIDATES?: string;
+  SESSION_SNAPSHOT_RECOVERY_MAX_ATTEMPTS?: string;
   DO_ALARMS_ENABLED_KV_KEY?: string;
   CONTROL_LOOP_KILL_SWITCH_CACHE_MS?: string;
   CONTROL_LOOP_DISABLED_ALARM_RETRY_MS?: string;

apps/api/src/scheduled/stuck-tasks.ts

@@ -43,14 +43,18 @@ import * as schema from '../db/schema';
 import type { TaskRunner } from '../durable-objects/task-runner';
 import type { Env } from '../env';
 import { log } from '../lib/logger';
+import { parsePositiveInt } from '../lib/route-helpers';
 import { maybeJsonRecord } from '../lib/runtime-validation';
 import { ulid } from '../lib/ulid';
 import { persistError } from '../services/observability';
 import * as projectDataService from '../services/project-data';
+import { DEFAULT_SESSION_SNAPSHOT_RECOVERY_MAX_ATTEMPTS } from '../services/session-snapshot-artifacts';
 import { cleanupTaskRun } from '../services/task-runner';
 import {
   classifyTaskRuntimeLiveness,
   loadRuntimeWorkspaceSnapshot,
+  loadSessionResumabilitySnapshot,
+  needsSessionResumabilityProbe,
   type RuntimeAcpSessionSnapshot,
   type TaskRuntimeLiveness,
   type TaskRuntimeLivenessSignals,
@@ -437,7 +441,33 @@ export async function getTaskRuntimeLiveness(
     }
   }
 
+  // Only probed for a workspace that would otherwise be declared conclusively
+  // dead, so the sweep pays one extra point lookup only when it is about to
+  // terminalize a task (`.claude/rules/47`).
+  let resumabilityProbeOutcome: TaskRuntimeLivenessSignals['resumabilityProbeOutcome'] = 'not_run';
+  let sessionResumability: TaskRuntimeLivenessSignals['sessionResumability'] = null;
+  if (needsSessionResumabilityProbe(workspace, workspaceProbeOutcome)) {
+    try {
+      sessionResumability = await loadSessionResumabilitySnapshot(
+        env.DATABASE,
+        task.project_id,
+        workspace.id,
+        workspace.chatSessionId
+      );
+      resumabilityProbeOutcome = 'ok';
+    } catch (err) {
+      resumabilityProbeOutcome = 'error';
+      log.warn('stuck_task.session_resumability_query_failed', {
+        workspaceId: task.workspace_id,
+        projectId: task.project_id,
+        action: 'preserved',
+        error: err instanceof Error ? err.message : String(err),
+      });
+    }
+  }
+
   const baseSignals: TaskRuntimeLivenessSignals = {
+    projectId: task.project_id,
     taskWorkspaceId: task.workspace_id,
     workspace,
     workspaceProbeOutcome,
@@ -447,6 +477,12 @@ export async function getTaskRuntimeLiveness(
     acpSessions: [],
     containerProbeOutcome: 'not_run',
     containerLifecycle: null,
+    resumabilityProbeOutcome,
+    sessionResumability,
+    resumabilityMaxRecoveryAttempts: parsePositiveInt(
+      env.SESSION_SNAPSHOT_RECOVERY_MAX_ATTEMPTS,
+      DEFAULT_SESSION_SNAPSHOT_RECOVERY_MAX_ATTEMPTS
+    ),
   };
   const initialClassification = classifyTaskRuntimeLiveness(baseSignals);
   if (

apps/api/src/services/session-snapshot-artifacts.ts

@@ -22,6 +22,23 @@ export const DEFAULT_SESSION_SLEEP_RETRY_DELAY_MS = 5 * 60 * 1000;
 export const DEFAULT_SESSION_SLEEP_MAX_ATTEMPTS = 9;
 export const DEFAULT_SESSION_SNAPSHOT_RECOVERY_CLAIM_LEASE_MS = 10 * 60 * 1000;
 
+/**
+ * Whether a snapshot's `status`/`degradation` pair still permits a restore.
+ * The in-memory twin of `restorableSnapshotCondition()` in
+ * `session-snapshot-recovery-lifecycle.ts`, which is the SQL predicate
+ * `claimSessionSnapshotRecovery` uses to authorize a wake.
+ *
+ * Lives here (rather than beside the SQL) so the task-runtime liveness
+ * classifier can mirror the real resume gate without duplicating the rule
+ * (`.claude/rules/58-terminal-verdicts-must-match-the-resumer.md`).
+ */
+export function isRestorableSnapshot(status: string | null, degradation: string | null): boolean {
+  return (
+    (status === 'available' && degradation === 'none') ||
+    (status === 'degraded' && Boolean(degradation) && degradation !== 'none')
+  );
+}
+
 type SnapshotLeaseEnv = Env & {
   SESSION_SLEEP_CLAIM_LEASE_MS?: string;
   SESSION_SNAPSHOT_RECOVERY_CLAIM_LEASE_MS?: string;

apps/api/src/services/session-snapshot-recovery-lifecycle.ts

@@ -7,6 +7,7 @@ import { parsePositiveInt } from '../lib/route-helpers';
 import {
   DEFAULT_SESSION_SNAPSHOT_RECOVERY_CLAIM_LEASE_MS,
   DEFAULT_SESSION_SNAPSHOT_RECOVERY_MAX_ATTEMPTS,
+  isRestorableSnapshot,
   sessionLifecycleError,
   type SessionSnapshotRecoveryClaim,
 } from './session-snapshot-artifacts';
@@ -27,13 +28,6 @@ function restorableSnapshotCondition() {
   );
 }
 
-function isRestorableSnapshot(status: string | null, degradation: string | null): boolean {
-  return (
-    (status === 'available' && degradation === 'none') ||
-    (status === 'degraded' && Boolean(degradation) && degradation !== 'none')
-  );
-}
-
 function sessionRecoveryClaimLeaseMs(env: Env): number {
   return parsePositiveInt(
     env.SESSION_SNAPSHOT_RECOVERY_CLAIM_LEASE_MS,
@@ -244,10 +238,7 @@ export async function markSessionSnapshotAwakeInPlace(
       updatedAt: now,
     })
     .where(
-      and(
-        eq(schema.sessionSnapshots.chatSessionId, chatSessionId),
-        restorableSnapshotCondition()
-      )
+      and(eq(schema.sessionSnapshots.chatSessionId, chatSessionId), restorableSnapshotCondition())
     );
 }