Sub-supervisor restarts the watcher with no backoff when it keeps returning the same wake reason

[vishalvx]

Observed on main at f66be0f.

Symptom

During an away-mode stretch, the sub-supervisor produced 258 and then 66 consecutive identical check: rearm-resurface escalations, back to back.
state/.supervise-daemon.watcher.err showed 61 consecutive watcher cycles with identical output and nothing else.
Each cycle is a captain-facing escalation, so the practical effect is an unusable away-mode session.

Cause

In bin/fm-supervise-daemon.sh, the watcher restart loop classifies a child exit three ways.

• rc != 0 or an empty reason calls record_crash and sleeps backoff_secs.
• A non-wake stdout line sleeps HOUSEKEEPING_TICK.
• A valid wake reason logs it, handles durable wakes, and calls start_watcher immediately, with no sleep at all.

The third path has no throttle of any kind.
That is correct for normal fleet traffic, where consecutive wakes differ.
But when the watcher keeps exiting with the same wake reason - for example an unconsumed recovery-marker episode kept alive by ongoing wake-queue traffic - the loop spins as fast as the watcher can start and exit.

The adjacent non-wake branch already carries a comment describing exactly this failure mode:

Non-wake stdout ... is NOT a wake: idling here prevents an escalation flood and a backoff-less child restart.

So the hazard is already recognized in the code, just not guarded for a repeating valid wake.

Suggested shape, not a prescription

Tracking consecutive occurrences of an identical wake reason and reusing the existing threshold, window, and backoff values would throttle it without calling it a crash, since it genuinely is not one.
We are running a local patch along those lines and it resolves the flood, but we are deliberately not opening a PR.
You will have a much better sense of whether this is the right layer for the fix.
Happy to share the patch if it is useful.

Related

The broken-pipe line reported separately was present alongside every one of these cycles and looked like the cause for a long time.
It is not.

[vishalvx]

For cross-reference, the broken-pipe line mentioned at the end is filed separately as #3275.

[vishalvx]

Correction to my earlier comment: the broken-pipe line is already tracked as #1011, filed a month before this report and with a more accurate mechanism analysis. My #3275 was a duplicate and is closed. #1011 is the right thread for that half.

[kunchenguid]

Speaking as Kun's firstmate: whole thread read (including #1011 / closed-dupe #3275 notes). Verified still unfixed on latest main (f66be0f).

In bin/fm-supervise-daemon.sh, a valid wake reason logs, handles durable wakes, then start_watcher with no sleep. Crash and non-wake branches already throttle; the repeating-identical-valid-wake path does not. That matches the reported rearm-resurface escalation flood. Distinct from #1011 (broken-pipe stderr noise is a false lead, not the restart-loop cause). No open PR implements identical-wake backoff on this loop (#2452 is escalation-buffer flush, not this). Do not open a competing PR from this pass.

VISION.md per-rule:

• One captain, one interface: aligns (identical-wake flood destroys away-mode peace of mind).
• Authority is explicit and never inferred: aligns (no new autonomy).
• Scripts own the mechanics, agents own the judgment: aligns (restart throttle is exact script mechanics).
• A restart is a non-event: aligns (restarts must not become an escalation storm).
• Delegation with a spine: aligns (supervision must stay usable under load).
• The fleet outlives any vendor: aligns (daemon loop is harness-orthogonal).
• Scope: aligns (command-layer supervision, not workshop).

contract-class: restore (non-wake branch already documents the flood hazard; valid repeating wake lacks the same promised throttle).

Labeling ready-for-pr.

[vishalvx]

Adjacent one-token defect found in the same restart loop while patching this locally, plus a correction to my own description above.

The defect

bin/fm-supervise-daemon.sh, in the non-wake-stdout branch of the same restart loop this issue is about:

sleep "${HOUSEKEEPING_TICK:-$HOUSEKEEPING_TICK_DEFAULT}"

HOUSEKEEPING_TICK is never assigned anywhere in the file. Only HOUSEKEEPING_TICK_DEFAULT=15 exists. So this line always sleeps the 15s default and silently ignores the documented FM_HOUSEKEEPING_TICK override.

The line a few below it, in the housekeeping gate, spells the same knob correctly:

if [ "$(_file_age "$STATE/.subsuper-last-housekeep")" -ge "${FM_HOUSEKEEPING_TICK:-$HOUSEKEEPING_TICK_DEFAULT}" ]; then

So the two consumers of one documented setting disagree, and the header at the top of the file documents FM_HOUSEKEEPING_TICK as controlling the housekeeping cadence with no mention that one of its two uses is inert. Observed at f66be0f. One-token fix.

Correction to my own report above

The bullet in my description that reads "A non-wake stdout line sleeps HOUSEKEEPING_TICK" is wrong in exactly this way. It sleeps the hardcoded 15s, never the configured value. I repeated the bug while describing the branch.

Why here rather than a separate issue

Same file, same function, same restart loop, three lines inside the branch this issue already quotes - and it corrects that quote. It seemed more useful in this thread than as a separate report someone would have to cross-reference. Happy to split it out if you would rather track it on its own.

It is independent of the backoff fix, so it can be taken separately either way.

[kunchenguid]

Speaking as Kun's firstmate: newer author activity after the prior ready-for-pr stamp (HOUSEKEEPING_TICK wiring in the same restart loop). Whole thread re-read, including #1011 / closed-dupe #3275 and the adjacent one-token defect.

Verified still unfixed on latest main (f66be0f). In bin/fm-supervise-daemon.sh a valid wake still logs, handles durable wakes, then start_watcher with no sleep. Crash and non-wake branches throttle; repeating identical valid wakes do not. Distinct from #1011. No open PR implements identical-wake backoff on this loop. Do not open a competing PR from this pass.

The adjacent defect is also real on main: the non-wake branch sleeps ${HOUSEKEEPING_TICK:-$HOUSEKEEPING_TICK_DEFAULT} but HOUSEKEEPING_TICK is never assigned; only HOUSEKEEPING_TICK_DEFAULT=15 exists, so FM_HOUSEKEEPING_TICK is ignored there while the housekeeping gate uses it correctly. Same file, same restart loop. In-scope here; can still land separately.

VISION.md per-rule:

• One captain, one interface: aligns (identical-wake flood destroys away-mode peace of mind).
• Authority is explicit and never inferred: aligns (no new autonomy).
• Scripts own the mechanics, agents own the judgment: aligns (restart throttle is exact script mechanics).
• A restart is a non-event: aligns (restarts must not become an escalation storm).
• Delegation with a spine: aligns (supervision must stay usable under load).
• The fleet outlives any vendor: aligns (daemon loop is harness-orthogonal).
• Scope: aligns (command-layer supervision, not workshop).

contract-class: restore (non-wake branch already documents the flood hazard; valid repeating wake lacks the same promised throttle. FM_HOUSEKEEPING_TICK already documented; one consumer is inert.)

Leaving ready-for-pr.

[devin-ai-integration]