fix(pi): restore watcher successors independently of hung wake delivery

[HelloWorldSungin]

Intent

Investigate the supervisor problems the captain has observed: workers finish or request validation but sit idle until the captain asks, and primary monitoring repeatedly stops after reporting a successful restart. The captain changed the supervisor model but does not know whether the model is responsible. Establish the actual cause or causes and recommend the smallest verified correction; do not assume model failure or treat another restart as a fix. The initial request authorized diagnosis only.

Captain's subsequent authorization: "yes lets fix the supervisor too". Implement the recommended smallest correction: monitoring must restart independently of slow or hung supervisor processing, so a second and subsequent worker notification cannot leave monitoring stopped. Preserve durable notifications and safe ownership across session replacement, and verify multiple consecutive cycles with deliberately hung processing. The diagnosis reproduced this mechanism using actual Pi extensions: monitoring restore was held while the supervisor was still processing a previous worker notification, so a later completion could leave monitoring stopped. The later model change was not its initiating cause. No live conversation discard or unrelated model-policy change is authorized.

What Changed

• Split restore from delivery in .pi/extensions/fm-primary-pi-watch.ts: the generation now tracks delivering (serialized pending-wake pump in flight) separately from restoring, which holds the single in-flight restore promise. A later actionable close arriving while an earlier wake is still being delivered now restores a successor arm on a side path instead of returning early, so a second or subsequent worker notification can no longer leave monitoring stopped. Concurrent callers await the shared restore's real result, side-path typed failures and thrown persistence errors are surfaced immediately rather than waiting for the hung delivery to settle, and arm close/error handling plus away-standby re-arming now check both flags.
• Added tests/fm-pi-hung-delivery-herdr-e2e.test.sh, a live isolated-Herdr-lab guard that hangs branch settlement and requires further worker completions to keep a live successor with a fresh watcher beacon, and registered it in bin/fm-test-run.sh. Extended tests/fm-pi-watch-extension.test.sh with held-failure, held-exception, slow-success, and slow-failure variants covering consecutive restores during hung delivery, session replacement, and wakes annotated with a shared restore failure.
• Documented the restore-versus-delivery invariants in the extension header and docs/watcher-continuity.md, and recorded the new live guard's command, scope limits, and 2026-09-11 Pi 0.85.1 run output in docs/verification/supervision.md.

Risk Assessment

✅ Low: The restore/delivery split and the two fix rounds are narrowly scoped and I traced the shared-promise dedup, duplicate-restore, deferred-close, away-mode, and shutdown paths without finding a reachable wrong result; the only substantiated issue is a non-portable stat call in a new opt-in test, and the intent's required behaviors are all present and covered by five portable modes plus the e2e guard.

Testing

I derived the scenarios from the captain's authorization (monitoring must restart independently of slow or hung supervisor processing, across multiple consecutive cycles, preserving durable notifications and replacement-session ownership). Three of them were driven against a real Pi process running in an isolated non-default Herdr lab: the live run showed three consecutive worker completions during a hung branch settlement each restoring a live successor watcher (successor=started for every cycle, zero successor=none) with a fresh watcher beacon after an unattended interval, and reverting only the extension to the base commit makes that same live run fail at the second completion, which is the reported symptom. The four remaining scenarios - restore-retry exhaustion, restore-time persistence exception, and the two shared in-flight restore outcomes, plus the model-policy/conversation-discard absence check - were only exercised in the in-process harness or by diff inspection, never against the live product, so they are reported as untested with the missing capability named. There is no UI surface in this change, so live evidence is CLI transcripts and persisted watcher state rather than screenshots. I made one test-legibility edit: the five parameterized modes printed identical pass lines, so the mode name is now in the label.

• Live validation: ✅ go - 3 of 7 scenarios driven live against the product

Scenario	Result	Live	Evidence
Three consecutive worker completions arrive while the supervisor is hung; a live watcher survives every cycle	✅ pass	live	FM_PI_HUNG_DELIVERY_HERDR_E2E=1 tests/fm-pi-hung-delivery-herdr-e2e.test.sh - real Pi in an isolated named Herdr lab; live-pi-watch-cycle-exits.log records successor=started for all three cycles and…
Monitoring beacon stays fresh across an unattended interval after the hung-settlement cycles	✅ pass	live	Same live lab run; .last-watcher-beat age asserted within the unattended window, captured in live-pi-beacon.txt
Adversarial: the reported bug reproduces on the pre-fix build - second completion leaves monitoring stopped	✅ pass	live	Live e2e re-run with only the extension reverted to base 668b61f: not ok - second actionable close did not restore a successor while settlement hung (live-pi-hung-delivery-e2e-BASE-prefix.txt)
Adversarial: the side-path restore fails after exhausting retries - main is told monitoring is down, and no later wake is delivered to clear it	⏸️ untested	no	The prior payload established this only in the in-process harness against a stubbed Pi host, not against the live product. Driving it live needs a way to inject arm-script failure into the real Herdr…
Adversarial: the side-path restore throws on persistence - a typed monitoring failure is surfaced instead of an unhandled rejection killing the Pi host, and pending wakes survive session replacement u…	⏸️ untested	no	The prior payload established this only in the in-process harness under node --unhandled-rejections=strict, not against the live product. A live result needs the e2e harness to corrupt the watcher s…
A concurrent in-flight restore returns its real result: successful restore still sends exactly one handling-delivery confirmation; a failing one annotates the later wake	⏸️ untested	no	The prior payload established this only through the in-process dispatch stub, not against the live product. A live result needs control over restore timing relative to settlement resolution in the rea…
No model-policy change and no live conversation discard is introduced	⏸️ untested	no	The prior payload established this only by diff inspection plus non-live regression cases; no end-user action was driven against the live product. A live result would require exercising a real Pi sess…

Evidence: Live Pi + Herdr lab run (fixed code) - three restored successors during hung settlement

ok - isolated Pi hung-settlement Herdr lab restored later-cycle successors and kept a fresh beacon

all fm-pi-hung-delivery-herdr-e2e tests passed

Evidence: Persisted watcher cycle-exit state from the live lab run

arm_pid=184908 watcher_pid=184938 reason=actionable-signal successor=started:186861 arm_pid=186844 watcher_pid=186861 reason=actionable-signal successor=started:190813 arm_pid=190788 watcher_pid=190813 reason=actionable-signal successor=started:193480

arm_pid=184908	watcher_pid=184938	origin=started	started_at=1789173547	ended_at=1789173547	exit_code=0	signal=none	reason=actionable-signal	beacon_age=0	lock_before=pid:184938|identity:linux-starttime=60784346 cmdline-hex=62617368002f686f6d652f73756e67696e2f2e6e6f2d6d697374616b65732f776f726b74726565732f6263383433326637633966382f30314d3239464530343037435843383838484442325248505a4b2f62696e2f666d2d77617463682e736800	lock_after=pid:none|identity:none	successor=started:186861
arm_pid=186844	watcher_pid=186861	origin=started	started_at=1789173547	ended_at=1789173549	exit_code=0	signal=none	reason=actionable-signal	beacon_age=0	lock_before=pid:186861|identity:linux-starttime=60784387 cmdline-hex=62617368002f686f6d652f73756e67696e2f2e6e6f2d6d697374616b65732f776f726b74726565732f6263383433326637633966382f30314d3239464530343037435843383838484442325248505a4b2f62696e2f666d2d77617463682e736800	lock_after=pid:none|identity:none	successor=started:190813
arm_pid=190788	watcher_pid=190813	origin=started	started_at=1789173549	ended_at=1789173550	exit_code=0	signal=none	reason=actionable-signal	beacon_age=0	lock_before=pid:190813|identity:linux-starttime=60784543 cmdline-hex=62617368002f686f6d652f73756e67696e2f2e6e6f2d6d697374616b65732f776f726b74726565732f6263383433326637633966382f30314d3239464530343037435843383838484442325248505a4b2f62696e2f666d2d77617463682e736800	lock_after=pid:none|identity:none	successor=started:193480

Evidence: Live Pi + Herdr lab run against BASE commit - fails at the second completion

not ok - second actionable close did not restore a successor while settlement hung

not ok - second actionable close did not restore a successor while settlement hung

Evidence: Watcher beacon freshness after the unattended interval in the live lab

-rw-r--r-- 1 sungin sungin 0 00:39:18 /tmp/fm-pi-hung-delivery-herdr-e2e.jubKrX/home/state/.last-watcher-beat

Evidence: Five hung-settlement modes against the real extension module (non-live harness)

ok - ... still hung (consecutive) ok - ... still hung (held-failure) ok - ... still hung (held-exception) ok - ... still hung (slow-success) ok - ... still hung (slow-failure)

ok - Pi restores later-cycle successors while an earlier branch settlement is still hung (consecutive)
ok - Pi restores later-cycle successors while an earlier branch settlement is still hung (held-failure)
ok - Pi restores later-cycle successors while an earlier branch settlement is still hung (held-exception)
ok - Pi restores later-cycle successors while an earlier branch settlement is still hung (slow-success)
ok - Pi restores later-cycle successors while an earlier branch settlement is still hung (slow-failure)

Evidence: Same portable scenario against BASE commit - monitoring left with zero live arms

Error: timeout waiting for second successor during hung settlement: live=0 rows=2

not ok - Pi hung settlement must still restore later-cycle successors (consecutive): file://~/.no-mistakes/worktrees/bc8432f7c9f8/01M29FE0407CXC888HDB2RHPZK/[eval1]:69
  throw new Error(`timeout waiting for ${label}: live=${liveArms().length} rows=${armRows().length}`);
        ^

Error: timeout waiting for second successor during hung settlement: live=0 rows=2
    at waitFor (file://~/.no-mistakes/worktrees/bc8432f7c9f8/01M29FE0407CXC888HDB2RHPZK/[eval1]:69:9)
    at async file://~/.no-mistakes/worktrees/bc8432f7c9f8/01M29FE0407CXC888HDB2RHPZK/[eval1]:116:1

Node.js v22.23.2: expected exit 0, got 1

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info

• ⚠️ .pi/extensions/fm-primary-pi-watch.ts:815 - The side-path restore is invoked as void restoreContinuity(owner, newest.predecessorArmPid) and its { failure } result is discarded. restoreAfterActionableClose never surfaces or schedules anything itself: on total failure it just returns the typed message (retries exhausted, or the early break when the lock is read-only / no live session). Concrete sequence: wake A is delivered to the branch and the settlement promise never resolves (the exact hung case this fix targets); worker 2's arm closes actionably; the side path restores; that restore fails after retryLimit=5 backoff attempts (or because the lock moved). owner.child is null, owner.retryTimer is null, deferredClose is null, and no surfaceFailure runs - monitoring is stopped and silent, and the pump's finally that would have appended the failure to the wake never runs because the delivery is hung. Before this change the same restoration failure always reached main as text appended to the wake. Smallest remedy: await the side-path result and surfaceFailure(owner, failure) (or schedule the bounded retry) when it is non-empty.
• ⚠️ .pi/extensions/fm-primary-pi-watch.ts:797 - restoreContinuity returns { failure: "" } - indistinguishable from a verified successful restore, and with no recovery - whenever another restore is merely in flight. The serialized pump calls it at line 864 and consumes that value as truth. Concrete sequence: wake A's branch settlement hangs; worker 2's close starts a side restore whose successor is slow to report readiness (armReadyTimeoutMs plus up to 5 backoff retries, several seconds); the settlement then resolves, the pump loops to pending B, calls restoreContinuity, sees restoring === true, and gets { failure: "" }. Two wrong results follow without any error: (a) restoration.recovery is undefined, so deliverActionableWake skips confirmHandlingDeliveryWithRetry, and fm-watch-arm.sh --handling-delivered never transitions the .watcher-down recovery marker from downtime to handling for that generation; (b) if the concurrent restore ultimately fails, wake B is delivered with repairFailed=false and no failure annotation, so main is told monitoring is fine while it is stopped. Smallest remedy: store the in-flight restore promise on the generation and have a concurrent caller await and return that promise's real result instead of a synthetic empty success.
• ℹ️ .pi/extensions/fm-primary-pi-watch.ts:36 - Acknowledging the deliberate, documented boundary rather than requesting a change: while a settlement stays hung, (a) the queued later wakes are never delivered to anyone - the portable test asserts session.prompts.length === 0 for cycles 2 and 3 - and (b) a non-actionable failure close of a restored successor only records owner.deferredClose, which is drained in the pump's finally and therefore never while the delivery is hung. Both match the header comment's stated contract and the authorized scope (monitoring restart only), so no action here.

🔧 Fix applied.
2 issues (1 warning, 1 info) still open:

• ⚠️ .pi/extensions/fm-primary-pi-watch.ts:815 - The new side path awaits restoreContinuity OUTSIDE the pump's try/catch, so a thrown error becomes an unhandled promise rejection instead of a typed monitoring failure. Every caller of processPendingActionables uses bare void (lines 742, 791, 931, 1173, 1228) with no .catch, so under Node's default --unhandled-rejections=throw the whole Pi host process dies. Concrete path: wake A's branch settlement hangs; worker 2's arm closes actionably; the side path calls restoreContinuity -> restoreAfterActionableClose -> startArm -> markLoaded() (:258-261), whose mkdirSync(state)/writeFileSync(marker) throw on EACCES/ENOSPC/read-only state dir - the same persistence-failure class the file already models elsewhere (see test_pi_replacement_persistence_failure_stops_arm_child). On the pump path that throw is caught at :904-907 and surfaced as watcher: FAILED - Pi extension could not deliver an actionable wake; on the side path nothing catches it. Note the shared promise makes this leak even when the pump also awaits it: the pump's catch handles its own awaiter, the side path's still rejects. Smallest remedy: wrap lines 813-818 in try/catch and surfaceFailure the detail, mirroring the pump's existing catch.
• ℹ️ .pi/extensions/fm-primary-pi-watch.ts:816 - Acknowledging a deliberate consequence of the two combined fixes rather than requesting a change: when the pump and the side path await the SAME failing restore, main receives the typed restoration failure twice - once standalone from surfaceFailure at :816 and once appended to the later wake at :868/:871. The new slow-failure test mode asserts both messages, so this is the fixer's intended behavior and it satisfies the user's instruction to surface the failure without waiting on the hung settlement. Impact is bounded: main's remediation is fm_watch_arm_pi, whose second call returns watcher: unchanged. No action.

🔧 Fix applied.
1 info still open:

• ℹ️ tests/fm-pi-hung-delivery-herdr-e2e.test.sh:102 - beacon_age_s calls GNU-only stat -c %Y "$beat" with no stat -f %m fallback. On macOS/BSD that command fails silently, mtime is empty, $((now - mtime)) is a bash arithmetic syntax error, and [ "" -le 13 ] errors, so the guard reports "watcher beacon went stale during the unattended interval (age=s)" while the beacon is actually fresh. Every other mtime reader in this repo uses the portable pair (tests/lib.sh:353, tests/fm-secondmate-reconcile.test.sh:950, tests/fm-omp-primary-live-e2e.test.sh:265, bin/fm-wake-lib.sh:95-97). Remedy: stat -c %Y "$beat" 2>/dev/null || stat -f %m "$beat" 2>/dev/null.

✅ **Test** - passed

✅ No issues found.

• Live validation: ✅ go - 3 of 7 scenarios driven live against the product

Scenario	Result	Live	Evidence
Three consecutive worker completions arrive while the supervisor is hung; a live watcher survives every cycle	✅ pass	live	FM_PI_HUNG_DELIVERY_HERDR_E2E=1 tests/fm-pi-hung-delivery-herdr-e2e.test.sh - real Pi in an isolated named Herdr lab; live-pi-watch-cycle-exits.log records successor=started for all three cycles and…
Monitoring beacon stays fresh across an unattended interval after the hung-settlement cycles	✅ pass	live	Same live lab run; .last-watcher-beat age asserted within the unattended window, captured in live-pi-beacon.txt
Adversarial: the reported bug reproduces on the pre-fix build - second completion leaves monitoring stopped	✅ pass	live	Live e2e re-run with only the extension reverted to base 668b61f: not ok - second actionable close did not restore a successor while settlement hung (live-pi-hung-delivery-e2e-BASE-prefix.txt)
Adversarial: the side-path restore fails after exhausting retries - main is told monitoring is down, and no later wake is delivered to clear it	⏸️ untested	no	The prior payload established this only in the in-process harness against a stubbed Pi host, not against the live product. Driving it live needs a way to inject arm-script failure into the real Herdr…
Adversarial: the side-path restore throws on persistence - a typed monitoring failure is surfaced instead of an unhandled rejection killing the Pi host, and pending wakes survive session replacement u…	⏸️ untested	no	The prior payload established this only in the in-process harness under node --unhandled-rejections=strict, not against the live product. A live result needs the e2e harness to corrupt the watcher s…
A concurrent in-flight restore returns its real result: successful restore still sends exactly one handling-delivery confirmation; a failing one annotates the later wake	⏸️ untested	no	The prior payload established this only through the in-process dispatch stub, not against the live product. A live result needs control over restore timing relative to settlement resolution in the rea…
No model-policy change and no live conversation discard is introduced	⏸️ untested	no	The prior payload established this only by diff inspection plus non-live regression cases; no end-user action was driven against the live product. A live result would require exercising a real Pi sess…

• FM_PI_HUNG_DELIVERY_HERDR_E2E=1 tests/fm-pi-hung-delivery-herdr-e2e.test.sh (real Pi 0.85.1 in an isolated named Herdr lab)
• Same live e2e with .pi/extensions/fm-primary-pi-watch.ts reverted to base 668b61f1 - reproduces the failure
• tests/fm-pi-watch-extension.test.sh (full file, 100+ cases including the 5 new hung-settlement modes)
• test_pi_hung_settlement_later_cycles_restore_successor in modes consecutive, held-failure, held-exception, slow-success, slow-failure
• Same portable mode consecutive against the base extension - reproduces live=0 rows=2
• Captured .watch-cycle-exits.log and .last-watcher-beat from the live lab run as product-state evidence
• git diff 668b61f1..690534f6 -- .pi/extensions/fm-primary-pi-watch.ts filtered for model/thinking/history-discard lines - none

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.