feat: see background tasks and wake the session when they settle — on every backend - #284
Merged
Conversation
… every backend The incident (real audit data, session 882d0a15): a model spawned background agents through its harness, promised "I'll report as soon as the three agents land", and ended its turn. The landings arrived when no turn was in flight. codeoid tracked nothing, showed nothing, and delivered nothing — the session sat idle until the owner interrupted it, 5 seconds after the last spawn. Two structural gaps, not one bug: 1. The daemon had NO model of harness-side background work. Zero sub-agent identities across 5 Agent-tool calls in the incident session (31 calls, 0 identities across four sessions) — the SDK's background tasks bypass the SubagentStart/Stop hooks entirely. 2. Provider events are TURN-scoped by construction. TurnRun's own contract documents that an event arriving between turns is "accepted and then discarded unread" — and settling between turns is precisely what a deferred task does. ## The generic seam `SessionScopedEvent` + `SessionProvider.onSessionEvent`: a session-lifetime channel, deliberately separate from the turn queue because these events fire when the queue has no reader. Two events, mirroring the level+edge design the Claude SDK itself settled on: - `background_tasks` — a LEVEL: the full live set, REPLACE semantics, so a missed event can never wedge a stale indicator. - `background_task_settled` — the EDGE carrying the outcome digest. Nothing provider-specific leaks into core: `kind` is the harness's own vocabulary and display-only, the daemon never branches on it, and the claude provider's SDK subtype names appear in exactly one translation function (threaded as a pure `emitSession` parameter, so the mapping is unit-testable without a process). pi/gemini/codex emit nothing until their harnesses grow background work; a future harness only implements this shape. ## What the Session does with it - **Visibility**: live tasks on `SessionInfo.backgroundTasks` (additive field), broadcast on change, cleared on provider teardown — the level is per-harness-process, so a rebuilt provider starts empty. The web sidebar shows a pulsing "N bg" chip; an idle-looking session with live background work is exactly the state that used to read as a hang. - **The wake**: settles queue and deliver as ONE batched injection at idle (burst-collapse, the <fleet_events> rule), from two triggers so both arrival orders work — a settle landing while idle, and an idle transition with settles queued mid-turn. Exactly once per task; a failed wake requeues its digests. The injection is a normal send under `system:background`: it rides the ordinary turn machinery, so tools still ask for approval in guarded mode and the autonomous budget still applies. Nothing here grants authority; it only supplies the information the model was waiting on. ## Verification Daemon suite 2200 → 2233, web 387 → 392. Five session-level mutations each caught by exactly the right tests (no idle-transition wake, no immediate wake, dedup removed, level appending instead of replacing, teardown keeping dead tasks), and the provider mapping asserts ROUTING — session events reach emitSession and never the turn queue — plus forward-compat (an unknown settle status is still a settle) and malformed-payload tolerance. Verified live with a real model: a claude session told to background `sleep 10 && echo BG_RESULT_42` and end its turn. The task appeared on the wire while the session idled, the daemon woke it (`session.background_wake`, subject `system:background`), and the model reported the exact output line. The sequence that hung the incident session now closes. One honest gap: one full-suite run showed 4 unreproducible failures before this change was complete; five hammer rounds of the timing-sensitive session suites plus two clean full runs followed. Noted rather than hidden — if it recurs in CI, treat it as pre-existing flake, not this change's regression, but do not auto-retry past it without reading the failures.
jalbrethsen-highflame
approved these changes
Aug 5, 2026
rsharath
approved these changes
Aug 5, 2026
…T NULL FK
The follow-up findings from the background-task investigation, folded in.
## Finding 1 was not an interrupt bug
The "interrupt audited with session_id: null" (row 13824) was never written
null. audit_log carried
FOREIGN KEY (session_id) REFERENCES sessions(id) ON DELETE SET NULL
so destroying a session retroactively anonymized its ENTIRE audit trail —
observed in the reporting database as 40 rows across 13 actions, each
session.destroy row nulled by its own delete. An audit log exists precisely
to survive its subjects; this FK made it forget them on schedule. It also
rejected legitimately history-first inserts, which is what the fold-into-
detail fallback in audit() was compensating for.
Fresh databases get the FK-free DDL. Existing ones are rebuilt once (SQLite
cannot drop a constraint via ALTER): rows and their ids are preserved
verbatim, because an audit citation like "row 13824" must keep meaning row
13824 across the rebuild. Rows already nulled are unrecoverable — the id was
destroyed at delete time, not hidden.
Mutation notes: gutting the rebuild fails the legacy-DB test; reinstating the
FK in the fresh DDL is an EQUIVALENT mutant — the rebuild migration strips it
back out on open, which is the defense-in-depth working, not a gap.
## Finding 2 resolved: one leaked identity, and why forensics was ambiguous
Pairing registered/deactivated audit rows by SUBJECT (the wimse URI — they
were attributable all along) identifies exactly one leak:
codeoid-subagent-a42e6a2a72e9, registered 2026-05-19 04:44:27, the last
registration before a daemon stop. That is the restart-orphan class —
in-memory registrations die with the process — and it predates #273's sweep,
which only covers in-process orphans. The restart-orphan cascade itself
(persisting registrations, or a server-side deactivate-by-parent) is a
design piece and stays a follow-up, now with precise evidence.
What WAS fixable here is why the investigation stayed ambiguous:
- a FAILED deactivation was console-logged but never audited, so "daemon
died before deactivating" and "deactivation was attempted and failed" were
indistinguishable from the audit trail. Failures now write
subagent.identity.deactivation_failed with the error.
- the destroy-time cascade never audited its SUCCESSES either, so identities
revoked at session destroy looked identical to leaks when pairing rows.
It now writes the same subagent.identity.deactivated row the direct path
does.
Daemon suite 2233 to 2237.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the "background agents never return" hang, reported from a live session and reproduced from its audit trail.
The incident (session
882d0a15): a model spawned background agents through its harness, promised "I'll report as soon as the three agents land", and ended its turn. The landings arrived when no turn was in flight. codeoid tracked nothing, showed nothing, delivered nothing — the session sat idle until the owner interrupted it. (The⏹ Interrupted by spiffe://…line was the owner's own Esc, attributed to the agent identity behind their API key — a display confusion, not a daemon action.)Two structural gaps, not one bug
Agent-tool calls in the incident session (31 calls / 0 identities across four sessions) — the SDK's background tasks bypass theSubagentStart/Stophooks entirely.TurnRun's own contract documents that an event arriving between turns is "accepted and then discarded unread" — and settling between turns is precisely what a deferred task does.The generic seam (works for any backend, present or future)
SessionScopedEvent+SessionProvider.onSessionEvent: a session-lifetime channel, deliberately separate from the turn queue because these events fire when the queue has no reader. Two events, mirroring the level+edge design the Claude SDK itself settled on:background_tasksbackground_task_settledNothing provider-specific leaks into core:
kindis the harness's own vocabulary and display-only (the daemon never branches on it), and the Claude SDK's subtype names appear in exactly one translation function, threaded as a pureemitSessionparameter so the mapping is unit-testable without a process. pi / gemini / codex emit nothing until their harnesses grow background work; a future harness only implements this shape.What the Session does with it
SessionInfo.backgroundTasks(additive field), broadcast on change, cleared on provider teardown (the level is per-harness-process). The web sidebar shows a pulsingN bgchip: an idle-looking session with live background work is exactly the state that used to read as a hang.<fleet_events>rule), from two triggers so both arrival orders work: a settle landing while idle, and an idle transition with settles queued mid-turn. Exactly once per task; a failed wake requeues its digests. The injection is a normal send undersystem:background— it rides the ordinary turn machinery, so tools still ask for approval in guarded mode and the autonomous budget still applies. Nothing here grants authority; it only supplies the information the model was waiting on.Verification
Five session-level mutations each caught by exactly the right tests — no idle-transition wake, no immediate wake, dedup removed, level appending instead of replacing, teardown keeping dead tasks. The provider-mapping tests assert routing (session events reach
emitSessionand never the turn queue), forward-compat (an unknown settle status is still a settle), and malformed-payload tolerance.Verified live with a real model: a claude session told to background
sleep 10 && echo BG_RESULT_42and end its turn.The sequence that hung the incident session now closes.
Honest notes
Two adjacent findings are not in this PRBoth follow-up findings are now folded in (second commit,538027f):session_id: nullinterrupt (row 13824) was never written null.audit_logcarriedFOREIGN KEY … ON DELETE SET NULL, so destroying a session retroactively anonymized its entire audit trail — 40 rows across 13 actions in the reporting DB, each destroy row nulled by its own delete. The FK is gone for fresh DBs, and existing ones are rebuilt once, rows and ids preserved verbatim ("row 13824" must keep meaning row 13824). Already-nulled rows are unrecoverable. Mutation notes: gutting the rebuild fails the legacy-DB test; reinstating the FK in fresh DDL is an equivalent mutant — the rebuild strips it back out on open, defense-in-depth working as designed.codeoid-subagent-a42e6a2a72e9, registered 2026-05-19 04:44:27 — the last registration before a daemon stop, i.e. the restart-orphan class (in-memory registrations die with the process), predating fix: stop losing turn-lifecycle events (dangling sub-agents, stuck 'thinking') #273's sweep. The orphan cascade itself stays a follow-up (a design piece: persist registrations or deactivate-by-parent server-side), now with precise evidence. What this PR fixes is why the forensics stayed ambiguous: failed deactivations now audit (subagent.identity.deactivation_failed) instead of only console-logging, and the destroy-time cascade audits its successes — so register/deactivate pairing is conclusive next time.🤖 Generated with Claude Code