Skip to content

fix: background agents stall the session — adopt the CLI's own turns and let background work outlive the turn - #345

Merged
saucam merged 8 commits into
mainfrom
fix/background-agents-outlive-turn
Sep 28, 2026
Merged

saucam merged 8 commits into
mainfrom
fix/background-agents-outlive-turn

Conversation

@saucam

@saucam saucam commented Sep 27, 2026

Copy link
Copy Markdown
Collaborator

What was wrong

When the model used background agents, sessions stalled.
The model would end its turn with "I'll report when they land", and then nothing happened, sometimes for minutes and once for six hours, until the owner asked "is this done?".
That message was refused with "A tool approval is pending", naming no tool.
Then a wall of text appeared under "you".

The root cause, measured live against the real Claude CLI: when a background agent finishes, the CLI runs the main agent by itself.
codeoid had no turn open to receive that, so it dropped the whole turn: the reply, the tool calls, and any approval they needed.
That dropped approval is the invisible "A tool approval is pending".
The session's status then stuck, so codeoid's own wake-up never fired.
When the wake-up did eventually fire, it ran a second, duplicate turn about the same result.

Underneath all of it: the session treated every tool call, approval and sub-agent as belonging to the current turn, and background agents outlive the turn that started them.
So besides the stall, the end of each turn also auto-denied running background agents' approvals, marked their tools "cancelled", and revoked their identities.

What changes

  • Turns the CLI starts on its own are adopted. They're consumed like any other turn, so their replies, tool cards and approval prompts render.
    A short note, "↻ Continuing after background work finished", explains a turn that has no prompt.
    They act and are audited as system:background, until the owner joins one.
  • Background agents live past the turn. Their tool calls between turns show up, and their approvals are visible and answerable.
    They no longer leave the session stuck.
    A turn ending doesn't deny, cancel or revoke them.
    The rule applies at every boundary: turn exit, a mid-turn message, stall recovery, provider teardown, the CLI dying, and Stop.
    Cleanup happens when the CLI reports the background work has drained, or when the CLI dies.
  • No duplicate wake-up. For Claude, codeoid's own wake-up is only a fallback, sent if the CLI hasn't continued within 20 seconds.
  • Stop means stop. Mid-turn it ends the turn and leaves background agents running.
    With no turn running, it stops the background work through the SDK's stopTask.
  • Your message is never refused because of a background agent's pending approval.
  • The wake-up renders compactly as "N background tasks finished" in the web UI and the TUI, with the full text a click away.
    Digest text is escaped, so it can't forge task rows or end the block early.

Verification

  • Live, real Claude CLI, sandbox daemon:
    • Background agent finishes: the CLI's own turn is adopted and its report renders.
      There's exactly one turn and the session ends idle.
    • With codeoid's wake-up disabled, the CLI's own turn still arrives. This is what confirmed the root cause: before the fix, every event of that turn was dropped.
    • The CLI's own turn asks to use Write: the approval card appears, approval writes the file, and the session ends idle.
      This is the exact failure from real sessions.
    • Stop while idle, with a background agent running: the agent is stopped before it finishes, the session ends idle, and nothing lingers.
  • Tests: 52 new or changed, across the session, provider, TUI renderer, web and core.
    Every fix was mutation-checked: each test fails with its fix reverted.
    The suites pass: daemon bun test 2670 pass, 0 fail; web 573 pass; typecheck and lint clean.
  • Audit: four rounds, each with a correctness reviewer and a security reviewer.
    Every round found something, and all of it was fixed in this branch, one commit per round.
    Round 4, the cap, came back ship-with-fixes and conditional GO. Its fixes were verified with tests, mutation checks and the live runs above, not a fifth round.
  • Not run: the Highflame regression suite, because codeoid isn't in it.
    The live runs above stand in for it.

Known gaps / follow-ups

  • No cap on chains of self-started turns in autonomous mode.
    codeoid's old wake-up had the same shape, and the turn budget already limits autonomous mode.
  • A foreground sub-agent inside a stopped turn, while unrelated background work is live, isn't swept until the background work drains.
    Its approval is withdrawn once the CLI cancels the request.
    Telling foreground and background sub-agents apart precisely needs run_in_background tracking.
  • setModel and rotate don't check whether the session is busy. A session:send holder can tear down mid-turn and deny pending approvals.
    This predates the branch.
  • Duplicate approval audit rows (the approver's row and the gate's row).
    This predates the branch.
  • A cosmetic mislabel when the owner's message races the CLI's own continuation.
For agents: where the code is
  • src/daemon/providers/interface.ts:
    • SessionScopedEvent gains background_event (tool/sub-agent lifecycle arriving with no turn) and turn_started {run}.
    • TurnRun.bindGate.
    • SessionProvider.continuesAfterBackgroundWork and stopBackgroundTasks.
    • ToolApprovalFn takes an optional AbortSignal.
  • src/daemon/providers/claude/index.ts:
    • #emit: when no queue is open, #adoptTurn (see opensAdoptedTurn); otherwise #handleUndeliverable routes lifecycle events to background_event.
    • #makeTurnRun is shared.
    • The loop's finally, rebuild and hard abort each announce an empty background_tasks.
    • The PreToolUse stand-in carries agent_id.
    • stopBackgroundTasks calls Query.stopTask.
  • src/daemon/session.ts:
    • Turn-only statuses: #turnActive / #setTurnStatus / #settleBetweenTurns.
    • #adoptTurn binds a system:background gate and records a user turn for the harness input.
    • #reconcileWork / #denyPendingApprovals apply keepBackground at mid-turn and turn exit; teardown, stall and drain reconcile fully.
    • #backgroundToolMsgIds tags tool calls that carry an sdkAgentId or arrive between turns.
    • #revokeSubagent waits for the registration fence.
    • #watchdogPaused ignores background approvals and counts main-agent tools in flight.
    • Self-continuing providers: a fallback timer (#armBackgroundWakeFallback), cancelled by any turn start.
    • interrupt() stops background work when idle, and leaves tool cards to turn exit when a turn is in flight.
    • #withdrawOnAbort.
    • formatDigest.
  • packages/core/src/background-wake.ts: the wake parser, shared by web and TUI.
  • Rendering: web/src/components/transcript/MessageRow.tsx, src/tui/ansi/render-message.ts, src/tui/components/MessageRow.tsx.
  • Tests: src/tests/session-background-agents.test.ts (new) and src/tests/tui-background-wake.test.tsx (new), plus additions to provider-claude, ansi-render-message, web MessageRow and core background-wake.
  • Mock: src/daemon/providers/mock/session-provider.ts gains startOwnTurn, continuesAfterBackgroundWork, boundGates and stopBackgroundTasks.

🤖 Generated with Claude Code

saucam and others added 8 commits September 27, 2026 22:33
A model starts background agents, ends its turn promising to report when they
land, and then nothing happens until the owner asks "is this done?" — which is
refused with "A tool approval is pending", naming no tool. Seen across many
sessions (a six-hour stall in one), always on the claude backend.

The session treated every tool call, approval and sub-agent as belonging to
the current turn. Background agents keep working after turn_done, so:

- Their tool calls between turns moved an idle session to tool_running
  (auto-approved) or waiting_approval (manual), and nothing moved it back.
  The background wake is gated on idle, so the reports never delivered.
- Their tool_start went to the closed turn queue and was dropped, so the
  approval card never rendered: a prompt nobody could see or answer.
- Every owner message was refused while "waiting_approval", though no turn
  was in flight to protect.
- Turn exit, run while background work was live, auto-denied the background
  agent's approvals, marked its in-flight tools cancelled, and swept (and
  revoked) the agent itself.

Now:

- The provider routes a background agent's tool/sub-agent lifecycle events
  that have no turn to go to onto the session channel (background_event),
  handled by the same code a turn uses — the tool card and approval bar
  render. Carryover remains the fallback with no session listener.
- Turn-only statuses (thinking, tool_running) change only while a turn is
  in flight; waiting_approval still shows the bar, and a background approval
  decided between turns returns the session to idle, delivering reports.
- A message sent with no turn in flight starts one even while a background
  approval is pending; the approval survives it.
- Turn exit keeps a sub-agent's (or between-turns) tool calls, approvals and
  registration while the provider reports live background work; the drain
  of the background set reconciles whatever is left.
- The wake is delivered explicitly at turn exit (turn_done flips idle while
  the run is still active, so the status flip alone never reached the gate).
- The wake renders as a compact "N background tasks finished" notice in the
  web UI and TUI instead of a wall of text under "you" (parser in core).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d work

Measured live against the real CLI: when a background agent finishes, the CLI
delivers the result to the model itself and runs the main agent — no prompt
from codeoid. With no turn queue open, codeoid dropped that whole turn event
by event ("undeliverable (no-queue) — dropped": thinking, the reply, turn_done).
The owner saw nothing, and an approval requested inside that turn became the
"A tool approval is pending" nobody could see. codeoid's own background wake
was then a second, duplicate turn about the same result — the model said so
("This looks like the same task notification I already flagged").

- The provider opens a turn when main-agent activity arrives with none open
  (per-query init, untagged text/thinking, a primary model call, a main-agent
  tool call) and hands it to the session as `turn_started`. A sub-agent's
  output never opens a turn. The TurnRun handle is shared with runTurn.
- The session consumes an adopted turn like a prompted one — reply, tool
  cards, approvals, turn_done → idle — with a note saying why a turn began
  with no prompt.
- A backend that continues on its own (`continuesAfterBackgroundWork`) gets
  no duplicate wake: a result settling mid-turn is delivered in that turn,
  one settling while idle arms a 20s fallback that any turn start cancels,
  and only if none begins does the session wake itself.
- A turn the backend starts while the owner's message is on its way is
  joined, not raced by a second runTurn that would close its queue.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Audit round 1 on the background-agent fix.

- One reconcile rule for every boundary (mid-turn, turn exit, stall
  recovery, provider teardown, background drain). The mid-turn boundary
  still auto-denied and revoked live background agents when the owner's
  message joined a turn; now it keeps them like turn exit does.
- Every path that kills the CLI clears the background set and reconciles:
  stall recovery and teardown did not, and a query-loop rebuild never got
  the empty level the dead CLI can't send (the provider now announces it).
  Before, a stale set kept a dead agent registered — its token live — for
  the rest of the session.
- A settle that empties the background set reconciles, not just a level.
- The PreToolUse stand-in carries the sub-agent's id, so a background
  agent's auto-approved call no longer opens a phantom adopted turn.
- A self-continuing backend is woken only by the fallback timer; an idle
  flip (a background approval decided, an interrupt) no longer injects a
  duplicate turn beside the backend's own.
- Adopted turns bind their own gate and principal (TurnRun.bindGate):
  approvals and audit belong to system:background, not the last human.
- A between-turns gate waits for the background-event chain, so it can't
  decide before its tool card exists (leaked mapping, approvable card for
  a resolved call).
- A background approval no longer pauses the main turn's stall watchdog.
- destroy() can't re-arm the fallback or adopt a turn afterwards.
- Wake digests can't forge task rows or close the block early.
- The live TUI renderer (render-message.ts) shows the compact wake too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- The provider announces an empty background set whenever its query loop
  ends — a crash, a hard abort, backing-session recovery — not only on a
  rebuild. A dead CLI never sends one, and a stale set turned off sub-agent
  cleanup at every later boundary.
- A sub-agent is revoked only after its registration settles. Revoking
  mid-registration was a no-op and the registration then completed into a
  live token nothing tracked — likelier now that a short background agent
  can start and drain in one breath.
- The stall watchdog pauses while a main-agent tool is in flight, whatever
  the status: a background approval flipping the status to waiting_approval
  would otherwise get a healthy long tool killed at the stall window.
- A turn the backend started is rebound to the owner who joins it, so their
  steering is audited to them, not to system:background.
- An adopted turn records the harness's delivery as its user turn, keeping
  canonical history alternating for a later cross-backend seed.
- Wake digests normalize every line break (CR, NEL, U+2028/9), drop format
  characters, and neutralize the block tag in any spelling.
- Tests destroy their sessions, so no fallback timer fires into the next.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…abort

Audit round 3 (security).

- Wake digests no longer try to match the block's tag — fullwidth solidus,
  fullwidth letters, combining marks, a missing bracket and HTML entities
  all got past that. NFKC-fold, drop format and combining characters,
  escape every < and > (and their entities), and indent every continuation
  line, splitting on VT/FF too. No angle bracket survives, so no spelling of
  the tag can end the block.
- The interrupt's hard-abort fallback announces the empty background set
  itself: a new turn rebuilding the loop first would otherwise bump the
  generation and silence the old loop's own announcement.
- Clarify that only adopted runs expose bindGate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Audit round 3 (correctness). interrupt() was the one boundary outside the
background rule: pressing Stop — even with no turn in flight, while a
background approval bar sat between turns — revoked the live agent's
identity, dropped it from /who, and denied its approval while the
background set still reported it running.

Stop now ends the turn and what belongs to it: the turn's own approvals
are denied (and their cards dismissed), its sub-agents swept. A live
background agent and its approval are kept; the set is read after
interrupting, so a hard abort — which kills the CLI and announces the set
empty — still reconciles everything. The session lands on waiting_approval
when a background approval is still pending, else idle.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… tool output

Audit round 4.

- Stop with no turn in flight now stops the background work itself, through
  the SDK's stopTask (SessionProvider.stopBackgroundTasks), then reconciles
  fully. The previous commit kept live background agents on every Stop,
  which left a holder of only session:interrupt no way to halt a misbehaving
  one. Stop mid-turn still ends just the turn.
- Stop mid-turn no longer closes tool cards before the interrupted turn has
  drained: a result landing after interrupt() resolved was dropped from its
  card, the transcript and memory — on every session. Turn exit reconciles
  them once the stream ends, as before. When the interrupt turned into a
  hard abort (the set now empty), the approvals kept for the dead CLI are
  denied.
- An approval whose request the backend abandons (its AbortSignal fires, e.g.
  the CLI cancelled the agent) is withdrawn instead of left waiting.
- Wake digests decompose, escape, then recompose instead of stripping
  combining marks — which mangled Devanagari, Thai, Hebrew and Arabic for no
  security gain. Entities without a semicolon are escaped too; control
  characters and only the invisible bidi/zero-width/tag characters are
  dropped (ZWJ/ZWNJ survive).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@saucam
saucam merged commit 7a5408c into main Sep 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants