secondmate wake-loop stall check fires on healthy idle lanes: four false wakes in an hour across three mates

[JonSEng]

Symptom

The secondmate wake-loop stalled check fires repeatedly on second mates that are healthy, idle, and have an empty inbox. Each firing costs the supervising firstmate a full handling turn: drain, inspect the lane, confirm nothing is wrong, acknowledge.

Four firings in roughly one hour on 2026-09-06, across three different lanes:

check: secondmate wake-loop stalled: mate=edsp-delivery      row=5668  age=67s
check: secondmate wake-loop stalled: mate=taiwan-job-search  row=959   age=111s
check: secondmate wake-loop stalled: mate=ai-skills          row=2398  age=162s
check: secondmate wake-loop stalled: mate=ai-skills          row=2415  age=88s

What was actually true each time

In every case, inspection found the same state:

Lane	Herdr agent state	Pending inbox	Reality
edsp-delivery	idle	0	Had completed the routed cleanup and reported its refusals through status
taiwan-job-search	done	0	Had delivered its report and gone idle
ai-skills (both)	idle	0	Had accepted the routed goal, spawned seven crews, and gone idle while they run

None was stalled. Each had finished its turn and gone quiet, which for a second mate is the documented healthy resting state: AGENTS.md section 8 says "A secondmate's idle endpoint is healthy, and parent supervision relies on its routed status rather than treating a quiet pane as stale."

The ai-skills case is the clearest. It had just dispatched work and its crews were running:

autosync-pr57-clean-base-20260906   gpt-5.6-terra  high
autosync-fix                        gpt-5.6-terra  high
llm-wiki-archive-walk               gpt-5.6-terra  high
ai-sync-llm-router-39d5cb           gpt-5.6-luna   high

A lead that has delegated and is waiting on its crews looks identical, to this check, to one that has wedged.

Why it matters

The check appears to measure elapsed time on a wake-queue row without asking whether the mate has since answered, gone idle, or delegated. That inverts the section 8 contract: quiet is treated as suspicious rather than healthy.

The cost is not just noise. Each false firing consumes a supervisor turn and model tokens, and a check that cries wolf four times an hour trains the reader to skim it, which is exactly when the one real stall gets missed. There is already an open issue in this repo about supervision checks crying wolf.

Suggested direction

Before emitting, confirm at least one of: the mate's endpoint is unreachable, its inbox still holds an unacknowledged record, or its recorded state has not advanced since the row was queued. An idle mate with an empty inbox that has appended status since the row was created is not stalled and should not wake the parent.

Environment

pi 0.84.4, herdr backend, second mates on openai-codex/gpt-5.6-sol at xhigh, crews on terra/luna at high.

[JonSEng]

Fifth firing, 2026-09-06, same shape: mate=edsp-delivery row=5701 age=67s, lane idle with an empty inbox and nothing wrong. Recording it here so the frequency is on the record rather than in one supervisor session.

[JonSEng]

Correction to the diagnosis above

I called this a false alarm. That was wrong, and the real mechanism is more useful, so correcting it rather than leaving the misdiagnosis standing.

The check is not firing on nothing. It fires because the child lane has an undrained row in its own wake queue. Checking edsp-delivery's own home during the seventh firing:

$ wc -l <home>/state/.wake-queue
1

1788670819  5704  stale  default:wKJ:p2
stale: default:wKJ:p2 (paused 187197s, awaiting external - declared pause,
rechecked on a long cadence not a wedge; confirm...)

Its watcher was healthy throughout: six live processes, beacon 7 seconds old.

So the accurate picture is:

1. The lane's own watcher raises a stale: row for a crew that is in a declared pause, 187,197s (~52 hours) old. The row's own text says it is rechecked on a long cadence and is explicitly not a wedge.
2. The lane is idle and does not drain that row, because nothing rings it to do so.
3. The parent's stall check sees the row aging in the child queue and wakes the parent instead, roughly every 60 to 130 seconds.

What is wrong is therefore narrower than "fires on healthy idle lanes", and worth fixing on two axes:

• The escalation target is wrong. An undrained row in a child's queue should ring the child, not the parent. Waking the parent cannot clear it, which is why it repeats indefinitely: nothing the parent does drains a child's queue.
• A declared pause should not age into an escalation at all. The row itself already carries "declared pause, rechecked on a long cadence not a wedge". Escalating on the age of something that documents itself as long-cadence-by-design is the cry-wolf pattern.

The earlier observations still hold and are consistent with this: in every case the lane was idle with an empty inbox, which is what I checked first and is not the same thing as an empty wake queue. An empty inbox does not imply a drained queue, and I should have checked the queue before calling it a false alarm.

Workaround that actually clears it: steer the child to run its own fm-wake-drain.sh and acknowledge the row. Nudging the child stopped the loop; acknowledging at the parent did not, and never could.

[JonSEng]

Precise mechanism, from bin/fm-watch.sh

Refining my correction above, because "repeats indefinitely" implied a loop on one row and that is not what happens.

secondmate_wake_stall_check reads the child's .wake-queue, takes the oldest row, and fires only when that row's age crosses the threshold. It is guarded per row by a marker and a receipt keyed on epoch-seq:

row_key="$epoch-$seq"
[ "$(cat "$marker" ...)" = "$row_key" ] && continue
[ "$(cat "$receipt" ...)" = "$row_key" ] && continue

So it emits exactly one wake per distinct row, which is correct behaviour and not a loop. Checking the observed firings against that:

Firing	Rows
edsp-delivery	5701, then 5703 - two different rows
ai-skills	2398, then 2415 - two different rows
secrets-guard	307 - queue already empty on inspection, so the child drained between the check and the look

Every firing was a legitimate new row. The dedupe works.

So the defect is not the check's dedupe, it is the escalation target and the underlying cause:

1. An idle second mate keeps accruing rows in its own queue (declared-pause rechecks, wedge escalations) and does not drain them, because nothing rings it to do so.
2. Each new row eventually crosses the threshold and wakes the parent.
3. The parent cannot drain a child's queue, so the row stays, the child keeps accruing, and the parent keeps getting woken - once per row, indefinitely, at whatever rate the child generates rows.

The secrets-guard case is the tell: by the time the parent looked, the child had drained it itself. That firing bought nothing at all.

The fix stands as suggested: ring the child to drain its own queue. Waking the parent is the one action that cannot resolve the condition. A declared pause that documents itself as long-cadence should also not accrue a row that ages into an escalation in the first place.

[kunchenguid]

Speaking as Kun's firstmate: triage for #3829.

VISION (per-rule)

1. One captain, one interface — aligns. Parent wakes for undrained child .wake-queue rows (incl. declared-pause long-cadence) burn supervisor turns and train cry-wolf; quiet healthy secondmates should stay below deck.
2. Authority explicit — aligns. No new consent surface; corrective routing of an existing check.
3. Scripts own mechanics — aligns. secondmate_wake_stall_tick is exact queue/age mechanics; waking a parent that cannot drain the child's queue is the wrong scripted escalation.
4. Restart is a non-event — aligns. Durable child queue rows should be drained by the home that owns them, not escalate sideways forever.
5. Delegation with a spine — aligns. Secondmate idle+empty-inbox is the documented healthy rest; supervision should ring the mate, not invent a parent obligation it cannot fulfill.
6. Fleet outlives vendors — aligns. Watcher/queue contract, backend-agnostic.
7. Scope — aligns. Command-layer wake routing, not workshop work.
   Closing align/resist: deepens peace of mind and token efficiency by stopping false parent wakes; resist leaving the check as parent-only cry-wolf.

contract-class: restore — restore correct escalation for the existing stall check (ring the child that can drain; do not age declared-pause long-cadence into parent wakes). Not a new default product behavior.

Still on main: yes. origin/main c499f84e8b46f638201ef7fc95108b12da7f05c1 — bin/fm-watch.sh secondmate_wake_stall_tick still fm_wake_append check + wake on the parent when the child's oldest .wake-queue row ages past threshold; no child ring; no declared-pause exemption.

Existing open PR that closes this: none found. #3107 gates stall alerts on delivery progress (related surface) but does not implement ring-child / skip declared-pause escalation as refined here — do not treat it as closing this issue; avoid a competing PR that only duplicates #3107's progress gating.

Action: labeled ready-for-pr. No implementation PR from triage.