fix: prevent stale paused-task wakes from triggering false stalls

[wjkawecki-jt]

Intent

A paused task's declared wait was not treated as authoritative for bounded stale rechecks, so a task legitimately waiting could be reported as a false stall.

What Changed

• Treat paused: declarations as authoritative for bounded stale-wake rechecks regardless of endpoint-liveness reads.
• Keep stricter liveness gating for captain-held transfers, while allowing authoritative working state to restore normal wedge tracking when a pause recheck lapses.
• Expanded watch triage coverage, updated Herdr fixture cleanup, and aligned architecture/configuration documentation with the revised behavior.

Risk Assessment

✅ Low: The change is bounded to paused-wait stale triage, updates its documentation, and adds behavioral regression coverage; the reviewed paths preserve rechecks, release handling, and wedge escalation without a source-verifiable defect.

Testing

Ran the focused watcher triage test file; the new stale-wake regression scenarios passed, including end-to-end durable queue and watcher behavior. No reviewer-visible artifact was produced because this is CLI behavior rather than a rendered UI.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

• tests/fm-watch-triage.test.sh
• Live declared-wait cadence, pane-hash churn, re-surfacing, pause release, captain-held fallback, and authoritative working-state reconciliation

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

_{Reviews (4): Last reviewed commit: "test(watch): age pause recheck fixture" | Re-trigger Greptile}

[tiago-peixoto]

We fixed the same class of bug in a fork of firstmate recently, in the same function, so I read this diff closely rather than skimming it.
Two of the three holes our own review found afterwards look live here, one does not, and I would rather say which is which than hand over a list.

First, what this gets right, because it took us a while to get here too: endpoint liveness is not evidence that a declared wait ended.
Splitting paused: (absorb regardless of liveness) from captain-held: (keep the stricter gate) in pause_state_class is exactly the line we ended up drawing, and test_declared_waits_use_bounded_cadence_until_released genuinely covers the case it names - a declared wait surviving a pane that keeps minting new hashes.

The two notes below are both about the same thing: the bound holds across a churning pane hash, but the status log is a separate source of churn, and it is the one that bit us.

1. Any later line from any producer retracts the declaration.

last_status_line (bin/fm-classify-lib.sh:110-114) returns the last non-blank line, so the declaration is whatever was written most recently.
The main loop at bin/fm-watch.sh:1914-1916 clears pause tracking as soon as that line is no longer paused:/captain-held:.
Two producers in the tree append into a worker's own status log without the worker doing anything: bin/fm-send.sh:645 writes resolved [key=$k]: ... into $STATE/$RESOLVE_TASK_ID.status, and bin/fm-pending-reply-lib.sh:1149 writes the same shape fully automatically.
The worker's own routine working: progress line does it too.

The sequence: a worker declares paused: on an external wait; something unrelated resolves a key, or the worker reports progress; the declaration is gone and the wedge ladder restarts from zero on a worker that is still waiting on the same external thing it declared.

What we measured on our side before we fixed it, over four days in one lane: stale (possible wedge) wakes were 320 wakes / 1,598 model requests / 37.8% of supervisor cost.
Of the 230 whose target and reason we could parse, 169 - 73% - named a worker that had already declared a pause, and they repeat on the same workers.
One was escalated thirty times while declaring, each time, that it was waiting on something external.

2. The re-surface throttle is skipped, rather than applied, when the log grows.

resurface_absorbed (bin/fm-watch.sh:737-749) keeps both age gates inside the scope-match branch, so a scope mismatch falls through to an unconditional wake.
The scope passed at :874, :921 and :1080 is declared:$(fm_wake_signal_sig "$statusf"), which for a .status path resolves to status_observed_signature (bin/fm-classify-lib.sh:1080-1109) - and that encodes size, so it changes on every append.

I ran your resurface_absorbed, age_of and status_observed_signature verbatim, stubbing only wake and fm_wake_append:

poll 1  (quiet log, age=4000s):       wakes=1   <- correct, cadence elapsed
poll 2  (quiet log, throttle 0s old): wakes=1   <- correct, throttled
append 1 (age=0s, throttle 0s old):   wakes=2
append 2 (age=0s, throttle 0s old):   wakes=3
append 3 (age=0s, throttle 0s old):   wakes=4
append 4 (age=0s, throttle 0s old):   wakes=5
append 5 (age=0s, throttle 0s old):   wakes=6

Five appends, five wakes, each at age 0 against a throttle 0 seconds old.
This is reachable when the append leaves paused: last - a re-declaration or a keepalive - which is the case note 1 does not cover.
Between them, an append either bypasses the throttle or ends the declaration outright.

3. The one that does NOT apply here - but might apply to the fix.

Our worst bug was that the re-surface age was anchored on the status file's mtime, which every append resets, so a worker whose own reporter appended every few minutes suppressed supervision indefinitely with no recheck.
Your age is mtime-anchored (bin/fm-watch.sh:863-866), but that silence cannot happen in this code, because note 2 masks it: the same append that resets the mtime also changes the throttle scope, and the mismatch skips the age gate.
You get noise where we got silence.

The reason I am mentioning it at all is what we measured when we removed the masking.
Same probe, only the scope changed to a stable value:

poll 1  (quiet log, age=4000s):        wakes=1
poll 2  (quiet log, throttle 0s old):  wakes=1
append 1..5 (age=0s, throttle 0s old): wakes=1   [no further wakes]

Zero wakes across five appends.
In our fork that turned into an ~18-hour silence with no recheck, and only a second review pass caught it.

On the test: test_declared_waits_use_bounded_cadence_until_released ages the declaration by back-dating the status file (touch -mt) and churns the pane (> "$capture_file"), and every status write in the file is an overwrite rather than an append.
That is the same shape as the gap in our own suite - it proves the bound for a log that has stopped changing, which is why finding 3 above survived our review for a cycle.

For what it is worth, findings 1 and 2 were introduced by our own instructions to the worker who wrote our fix, and caught by review afterwards.
We are not arriving as experts here; we are arriving with scars.
Happy to share the probe scripts if they are useful.