Skip to content

feat: see background tasks and wake the session when they settle — on every backend - #284

Merged
saucam merged 2 commits into
mainfrom
feat/background-task-visibility
Aug 5, 2026
Merged

feat: see background tasks and wake the session when they settle — on every backend#284
saucam merged 2 commits into
mainfrom
feat/background-task-visibility

Conversation

@saucam

@saucam saucam commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Fixes the "background agents never return" hang, reported from a live session and reproduced from its audit trail.

The incident (session 882d0a15): a model spawned background agents through its harness, promised "I'll report as soon as the three agents land", and ended its turn. The landings arrived when no turn was in flight. codeoid tracked nothing, showed nothing, delivered nothing — the session sat idle until the owner interrupted it. (The ⏹ Interrupted by spiffe://… line was the owner's own Esc, attributed to the agent identity behind their API key — a display confusion, not a daemon action.)

Two structural gaps, not one bug

  1. The daemon had no model of harness-side background work. Zero sub-agent identities across 5 Agent-tool calls in the incident session (31 calls / 0 identities across four sessions) — the SDK's background tasks bypass the SubagentStart/Stop hooks entirely.
  2. Provider events are turn-scoped by construction. TurnRun's own contract documents that an event arriving between turns is "accepted and then discarded unread" — and settling between turns is precisely what a deferred task does.

The generic seam (works for any backend, present or future)

SessionScopedEvent + SessionProvider.onSessionEvent: a session-lifetime channel, deliberately separate from the turn queue because these events fire when the queue has no reader. Two events, mirroring the level+edge design the Claude SDK itself settled on:

event role
background_tasks level — the full live set, REPLACE semantics, so a missed event can never wedge a stale indicator
background_task_settled edge — the outcome digest, the thing a session must be woken with

Nothing provider-specific leaks into core: kind is the harness's own vocabulary and display-only (the daemon never branches on it), and the Claude SDK's subtype names appear in exactly one translation function, threaded as a pure emitSession parameter so the mapping is unit-testable without a process. pi / gemini / codex emit nothing until their harnesses grow background work; a future harness only implements this shape.

What the Session does with it

  • Visibility — live tasks on SessionInfo.backgroundTasks (additive field), broadcast on change, cleared on provider teardown (the level is per-harness-process). The web sidebar shows a pulsing N bg chip: an idle-looking session with live background work is exactly the state that used to read as a hang.
  • The wake — settles queue and deliver as one batched injection at idle (burst-collapse, the <fleet_events> rule), from two triggers so both arrival orders work: a settle landing while idle, and an idle transition with settles queued mid-turn. Exactly once per task; a failed wake requeues its digests. The injection is a normal send under system:background — it rides the ordinary turn machinery, so tools still ask for approval in guarded mode and the autonomous budget still applies. Nothing here grants authority; it only supplies the information the model was waiting on.

Verification

suite before after
daemon 2200 2233
web 387 392

Five session-level mutations each caught by exactly the right tests — no idle-transition wake, no immediate wake, dedup removed, level appending instead of replacing, teardown keeping dead tasks. The provider-mapping tests assert routing (session events reach emitSession and never the turn queue), forward-compat (an unknown settle status is still a settle), and malformed-payload tolerance.

Verified live with a real model: a claude session told to background sleep 10 && echo BG_RESULT_42 and end its turn.

[bg] VISIBLE: [{"id":"bbxox6mvs","kind":"local_bash","description":"Sleep 10s then echo result","status":"running"}]
[bg] wake rows: [{"subject":"system:background","detail":"reports=1 tasks=bbxox6mv"}]
[bg] REPORT DELIVERED: The background command finished (exit code 0). Output: BG_RESULT_42

The sequence that hung the incident session now closes.

Honest notes

  • One full-suite run showed 4 unreproducible failures mid-development; five hammer rounds of the timing-sensitive session suites plus two clean full runs followed. If something similar appears in CI, read it before retrying — but it predates nothing in this change that seven subsequent runs could detect.
  • Two adjacent findings are not in this PR Both follow-up findings are now folded in (second commit, 538027f):
    • The session_id: null interrupt (row 13824) was never written null. audit_log carried FOREIGN KEY … ON DELETE SET NULL, so destroying a session retroactively anonymized its entire audit trail — 40 rows across 13 actions in the reporting DB, each destroy row nulled by its own delete. The FK is gone for fresh DBs, and existing ones are rebuilt once, rows and ids preserved verbatim ("row 13824" must keep meaning row 13824). Already-nulled rows are unrecoverable. Mutation notes: gutting the rebuild fails the legacy-DB test; reinstating the FK in fresh DDL is an equivalent mutant — the rebuild strips it back out on open, defense-in-depth working as designed.
    • The 11/10 identity count resolves to exactly one leak: codeoid-subagent-a42e6a2a72e9, registered 2026-05-19 04:44:27 — the last registration before a daemon stop, i.e. the restart-orphan class (in-memory registrations die with the process), predating fix: stop losing turn-lifecycle events (dangling sub-agents, stuck 'thinking') #273's sweep. The orphan cascade itself stays a follow-up (a design piece: persist registrations or deactivate-by-parent server-side), now with precise evidence. What this PR fixes is why the forensics stayed ambiguous: failed deactivations now audit (subagent.identity.deactivation_failed) instead of only console-logging, and the destroy-time cascade audits its successes — so register/deactivate pairing is conclusive next time.

🤖 Generated with Claude Code

… every backend

The incident (real audit data, session 882d0a15): a model spawned background
agents through its harness, promised "I'll report as soon as the three agents
land", and ended its turn. The landings arrived when no turn was in flight.
codeoid tracked nothing, showed nothing, and delivered nothing — the session
sat idle until the owner interrupted it, 5 seconds after the last spawn.

Two structural gaps, not one bug:

1. The daemon had NO model of harness-side background work. Zero sub-agent
   identities across 5 Agent-tool calls in the incident session (31 calls, 0
   identities across four sessions) — the SDK's background tasks bypass the
   SubagentStart/Stop hooks entirely.
2. Provider events are TURN-scoped by construction. TurnRun's own contract
   documents that an event arriving between turns is "accepted and then
   discarded unread" — and settling between turns is precisely what a
   deferred task does.

## The generic seam

`SessionScopedEvent` + `SessionProvider.onSessionEvent`: a session-lifetime
channel, deliberately separate from the turn queue because these events fire
when the queue has no reader. Two events, mirroring the level+edge design the
Claude SDK itself settled on:

- `background_tasks` — a LEVEL: the full live set, REPLACE semantics, so a
  missed event can never wedge a stale indicator.
- `background_task_settled` — the EDGE carrying the outcome digest.

Nothing provider-specific leaks into core: `kind` is the harness's own
vocabulary and display-only, the daemon never branches on it, and the claude
provider's SDK subtype names appear in exactly one translation function
(threaded as a pure `emitSession` parameter, so the mapping is unit-testable
without a process). pi/gemini/codex emit nothing until their harnesses grow
background work; a future harness only implements this shape.

## What the Session does with it

- **Visibility**: live tasks on `SessionInfo.backgroundTasks` (additive
  field), broadcast on change, cleared on provider teardown — the level is
  per-harness-process, so a rebuilt provider starts empty. The web sidebar
  shows a pulsing "N bg" chip; an idle-looking session with live background
  work is exactly the state that used to read as a hang.
- **The wake**: settles queue and deliver as ONE batched injection at idle
  (burst-collapse, the <fleet_events> rule), from two triggers so both
  arrival orders work — a settle landing while idle, and an idle transition
  with settles queued mid-turn. Exactly once per task; a failed wake requeues
  its digests. The injection is a normal send under `system:background`: it
  rides the ordinary turn machinery, so tools still ask for approval in
  guarded mode and the autonomous budget still applies. Nothing here grants
  authority; it only supplies the information the model was waiting on.

## Verification

Daemon suite 2200 → 2233, web 387 → 392. Five session-level mutations each
caught by exactly the right tests (no idle-transition wake, no immediate
wake, dedup removed, level appending instead of replacing, teardown keeping
dead tasks), and the provider mapping asserts ROUTING — session events reach
emitSession and never the turn queue — plus forward-compat (an unknown settle
status is still a settle) and malformed-payload tolerance.

Verified live with a real model: a claude session told to background
`sleep 10 && echo BG_RESULT_42` and end its turn. The task appeared on the
wire while the session idled, the daemon woke it (`session.background_wake`,
subject `system:background`), and the model reported the exact output line.
The sequence that hung the incident session now closes.

One honest gap: one full-suite run showed 4 unreproducible failures before
this change was complete; five hammer rounds of the timing-sensitive session
suites plus two clean full runs followed. Noted rather than hidden — if it
recurs in CI, treat it as pre-existing flake, not this change's regression,
but do not auto-retry past it without reading the failures.
…T NULL FK

The follow-up findings from the background-task investigation, folded in.

## Finding 1 was not an interrupt bug

The "interrupt audited with session_id: null" (row 13824) was never written
null. audit_log carried

    FOREIGN KEY (session_id) REFERENCES sessions(id) ON DELETE SET NULL

so destroying a session retroactively anonymized its ENTIRE audit trail —
observed in the reporting database as 40 rows across 13 actions, each
session.destroy row nulled by its own delete. An audit log exists precisely
to survive its subjects; this FK made it forget them on schedule. It also
rejected legitimately history-first inserts, which is what the fold-into-
detail fallback in audit() was compensating for.

Fresh databases get the FK-free DDL. Existing ones are rebuilt once (SQLite
cannot drop a constraint via ALTER): rows and their ids are preserved
verbatim, because an audit citation like "row 13824" must keep meaning row
13824 across the rebuild. Rows already nulled are unrecoverable — the id was
destroyed at delete time, not hidden.

Mutation notes: gutting the rebuild fails the legacy-DB test; reinstating the
FK in the fresh DDL is an EQUIVALENT mutant — the rebuild migration strips it
back out on open, which is the defense-in-depth working, not a gap.

## Finding 2 resolved: one leaked identity, and why forensics was ambiguous

Pairing registered/deactivated audit rows by SUBJECT (the wimse URI — they
were attributable all along) identifies exactly one leak:
codeoid-subagent-a42e6a2a72e9, registered 2026-05-19 04:44:27, the last
registration before a daemon stop. That is the restart-orphan class —
in-memory registrations die with the process — and it predates #273's sweep,
which only covers in-process orphans. The restart-orphan cascade itself
(persisting registrations, or a server-side deactivate-by-parent) is a
design piece and stays a follow-up, now with precise evidence.

What WAS fixable here is why the investigation stayed ambiguous:

- a FAILED deactivation was console-logged but never audited, so "daemon
  died before deactivating" and "deactivation was attempted and failed" were
  indistinguishable from the audit trail. Failures now write
  subagent.identity.deactivation_failed with the error.
- the destroy-time cascade never audited its SUCCESSES either, so identities
  revoked at session destroy looked identical to leaks when pairing rows.
  It now writes the same subagent.identity.deactivated row the direct path
  does.

Daemon suite 2233 to 2237.
@saucam
saucam merged commit ac8c906 into main Aug 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants