Skip to content

fix(bin): keep crews with a current declared pause off wedge escalation - #2738

Open
Omar-Nawaf wants to merge 5 commits into
kunchenguid:mainfrom
Omar-Nawaf:fm/firstmate-declared-pause-wedge-upstream-pr
Open

fix(bin): keep crews with a current declared pause off wedge escalation#2738
Omar-Nawaf wants to merge 5 commits into
kunchenguid:mainfrom
Omar-Nawaf:fm/firstmate-declared-pause-wedge-upstream-pr

Conversation

@Omar-Nawaf

@Omar-Nawaf Omar-Nawaf commented Aug 21, 2026

Copy link
Copy Markdown

Intent

Publish the already-implemented declared-pause wedge correction as one upstream PR for kunchenguid/firstmate issue #2614 now that publication is authorized and the maintainer marked it ready for PR. On current origin/main, a current declared paused: external wait must use FM_PAUSE_RESURFACE_SECS and must never produce repeated idle/busy possible-wedge or demand-deep-inspection escalation in ordinary or away-mode supervision; a newer non-pause status must supersede the wait and restore normal bounded wedge detection so an old pause cannot hide a real wedge forever, while authoritative active-run precedence remains intact. Keep this policy in the existing shared classifier owner, preserve non-paused max-defer and wedge reliability, and do not broaden the change into speculative busy-pane damping. Preserve or improve executable regressions for paused suppression, pause supersession, ordinary escalation, away-mode watcher routing, and daemon rechecks; run focused classifier, watcher, daemon, and AFK suites, documentation audience checks, exact bin/fm-lint.sh, and the complete no-mistakes pipeline. Port only the intended correction from preserved commit d6f4659 onto current origin/main without altering the preserved source branch or old copy, and include no private data/state/config, credentials, local paths, project content, agent co-author, or unrelated changes. The external-healthy-watcher test was treated as branch-relevant, reproduced identically on this branch and clean origin/main, and proven to fail only under Node 22 TS import support while both pass unchanged under available Linux Node 24, so do not weaken or alter that unrelated assertion. Push only fm/firstmate-declared-pause-wedge-upstream-pr through the captain-authorized Omar-Nawaf/firstmate fork and open one PR against kunchenguid/firstmate:main referencing #2614; never merge.

What Changed

  • Added status_pause_damps_wedge to bin/fm-classify-lib.sh and applied it in bin/fm-watch.sh's pause_state_class, so a crew whose latest status declares a paused: external wait takes the bounded FM_PAUSE_RESURFACE_SECS recheck regardless of agent liveness or pane busy state, instead of surfacing immediately or climbing possible-wedge / demand-deep-inspection escalation; a newer non-pause status line restores ordinary wedge detection, and an authoritative active-run verdict keeps its existing precedence.
  • Kept the pause re-surface throttle (.paused-resurfaced-<key>) across a busy flicker: clear_pause_state no longer deletes it, and the main loop drops it only once the latest status stops declaring a wait, so a short busy turn that appends no status line cannot trigger an off-cadence recheck while the bounded recheck still fires once the throttle ages past the cadence.
  • Extended tests/fm-watch-triage.test.sh with a busy-flicker cadence regression and reworked the live-pause test to pin the bounded cadence over the old liveness override, and aligned docs/architecture.md, docs/configuration.md, and the classifier/watcher/daemon comments with the latest-status rule.

Risk Assessment

✅ Low: The change is confined to the declared-pause branch of pause_state_class plus the resurface-throttle lifecycle, keeps the policy in the shared classifier, leaves authoritative run-step precedence and all non-paused wedge paths untouched, and the traced state transitions (supersession, busy flicker, away-mode handoff, daemon clear) all keep the pause bounded rather than silenced, with executable tests pinning each behavior.

Testing

Exercised the intent's full acceptance surface: proved both new pause-wedge regressions fail on base d71f4b9 and pass on target; ran the focused watcher, classifier, daemon, AFK, daemon-lifecycle, and documentation-audience suites (all green except one pre-existing, base-reproducible procevent environment failure unrelated to the change); and captured a live fm-watch.sh CLI transcript demonstrating pause absorb, the bounded FM_PAUSE_RESURFACE_SECS recheck without possible-wedge or deep-inspection escalation, supersession by a newer non-pause status, and restored wedge escalation afterward. Lint and full-pipeline phases are owned by the outer executor. No UI surface is involved, so the CLI transcript is the reviewer-visible artifact.

Evidence: Pause-lifecycle end-user watcher transcript (absorb → bounded recheck → supersede → wedge restored)

Source: Pause-lifecycle end-user watcher transcript (absorb → bounded recheck → supersede → wedge restored)

== Phase 1: crew declares a pause; pane sits idle and stale == watcher wake> (none - absorbed silently) state: .paused marker recorded (long-cadence tracking) state: no wedge timer started, no possible-wedge escalation == Phase 2: pause ages past FM_PAUSE_RESURFACE_SECS -> ONE bounded recheck == watcher wake> stale: test:fm-demo-crew (paused 501s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds) => no 'possible wedge', no 'demand-deep-inspection' in the wake == Phase 3: newer non-pause status supersedes the wait == state: pause tracking cleared - wait superseded state: wedge timer restarted - bounded wedge detection restored == Phase 4: pane still frozen past the wedge threshold -> real wedge escalates == watcher wake> stale: test:fm-demo-crew (idle 501s, possible wedge, escalation 1)

############################################################
# Demo: declared 'paused:' external wait vs wedge detection
# fm-watch.sh at 0a42e3a
############################################################

== Phase 1: crew declares a pause; pane sits idle and stale ==
  status log: paused: waiting for the vendor rate-limit reset
  watcher wake> (none - absorbed silently)
  state: .paused marker recorded (long-cadence tracking)
  state: no wedge timer started, no possible-wedge escalation

== Phase 2: pause ages past FM_PAUSE_RESURFACE_SECS -> ONE bounded recheck ==
  watcher wake> stale: test:fm-demo-crew (paused 501s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds)
  => recheck labeled as awaiting-external (bounded, informational)
  => no 'possible wedge', no 'demand-deep-inspection' in the wake

== Phase 3: newer non-pause status supersedes the wait ==
  status log gains: working: vendor reset done, resuming the audit
  watcher wake> (none - absorbed silently)
  state: pause tracking cleared - wait superseded
  state: wedge timer restarted - bounded wedge detection restored

== Phase 4: pane still frozen past the wedge threshold -> real wedge escalates ==
  watcher wake> stale: test:fm-demo-crew (idle 501s, possible wedge, escalation 1)
  => an old pause cannot hide a real wedge: escalation is back

Demo complete: absorb -> bounded recheck -> supersede -> wedge restored.
Evidence: Before-fix reproduction: both regression tests fail on base commit d71f4b9

Source: Before-fix reproduction: both regression tests fail on base commit d71f4b9

=== BASE d71f4b9: test_declared_pause_and_exited_captain_hold_are_bounded === not ok - a live declared pause surfaced immediately: stale: test:fm-gate exit=1 === BASE d71f4b9: test_paused_busy_flicker_keeps_resurface_cadence === not ok - a busy pane erased the declared pause's re-surface throttle exit=1

=== BASE d71f4b9: test_declared_pause_and_exited_captain_hold_are_bounded ===
not ok - a live declared pause surfaced immediately: stale: test:fm-gate
exit=1

=== BASE d71f4b9: test_paused_busy_flicker_keeps_resurface_cadence ===
not ok - a busy pane erased the declared pause's re-surface throttle
exit=1
Evidence: Watcher triage suite on target (59 ok incl. all pause/wedge/AFK regressions; 1 pre-existing procevent env failure)

Source: Watcher triage suite on target (59 ok incl. all pause/wedge/AFK regressions; 1 pre-existing procevent env failure)

FM_TEST_BEGIN 2026-08-30T19:07:55Z tests/fm-watch-triage.test.sh family=watcher-wake-lock expected_gate_skip=none
ok - status_span_has_actionable: benign absorbed, captain events surfaced, classified events not re-fired
ok - an actionable event is not hidden by later routine appends, and is named as itself
ok - span classification retires closed decisions and surfaces rejected transitions for reconciliation
ok - a malformed seen signature causes the whole status log to be classified
ok - stale_is_terminal: terminal status surfaces, non-terminal and no-status are benign
ok - classifier primitives: keyed decisions and activity phases, captain relevance, window-to-task, and overrides
ok - crew_is_provably_working: only working+run-step/pane is provable; idle/finished/parked/failed/unknown surface
ok - status_is_paused: only the leading paused verb matches, paused is not captain-relevant, and the two declared-wait verbs stay separable
ok - crew_absorb_class: working/paused/none from one read; crew_is_paused and crew_is_provably_working agree
ok - crew_worktree_written_since: real writes are evidence; no worktree, no anchor, quiet trees, .git churn and a mate's own home are not
ok - an empty FM_WORKTREE_WRITE_PRUNE widens the probe to the whole depth-bounded tree instead of disabling it
ok - an empty FM_WORKTREE_WRITE_PRUNE exported into the environment prunes nothing, widening the probe
ok - the worktree write probe is wall-clock bounded, and hitting the bound reads as no write evidence
ok - signal_crew_provably_working: benign only when every referenced crew is provably working
ok - a secondmate's status signal is never absorbed as provably working; crewmates are unaffected
ok - a no-verb signal whose crew is provably working is absorbed (no exit, no queue, suppressor advanced, beacon present)
ok - a bare turn-end whose crew is provably working (busy pane) is absorbed
ok - a bare turn-end whose crew is not provably working is surfaced (the swallowed-finish fix)
ok - a no-verb working: note whose crew is idle with no running pipeline is surfaced
ok - a secondmate's status note surfaces even while its own agent is busy
ok - a self-announced close never wakes its own home, and the next real note still does
ok - captain-relevant signal is surfaced (queue + exit) and marked surfaced
ok - a captain event hidden behind a later routine append is still surfaced (queue + exit)
ok - a finished release reported before routine cleanup chatter is still surfaced
ok - a routine append after an already-classified event is absorbed (no re-wake)
ok - unreadable status reports are bounded without advancing classification
ok - permission recovery surfaces content from the unadvanced position
ok - a stale pane sitting on a terminal status is surfaced (queue + exit)
ok - a stale terminal-looking status is overridden and absorbed while a run is actively working, then wedge-escalated
ok - provably-working non-terminal stale is absorbed on first sight, then wedge-escalated past the threshold
ok - consecutive wedge escalations on the same pane accumulate and demand deep inspection at the threshold
ok - a pane becoming active again resets the consecutive wedge-escalation counter
ok - a busy worker below the turn-age bound remains working with no escalation
ok - a busy worker with a stable pane hash still escalates once its completed-turn age reaches the bound
ok - a busy worker whose pane hash changes every poll still escalates once its completed-turn age reaches the bound
ok - touching a busy worker's completed-turn marker resets the age and prevents an old-age escalation
ok - repeated busy turn-age escalations reuse the existing escalation counter and demand deep inspection at the threshold
ok - the production default busy-turn-age bound is 3600s (5min under does not wedge, 66min over does)
ok - a busy pane under a declared pause is rechecked on the long cadence, and lifting the pause restores the wedge escalation
ok - away mode hands a busy declared pause to the daemon as a plain stale, and lifting the declaration restores the wedge escalation
ok - away mode wakes the daemon once per declaration for a busy pane whose footer ticks on every capture
ok - a not-provably-working non-terminal stale is surfaced immediately (never left to wait out the timer)
ok - a declared pause is absorbed on first sight, then re-surfaced as a recheck past the threshold, never wedge-escalated
ok - a paused crew's short busy turn keeps the bounded re-surface cadence instead of an immediate recheck
ok - current declared pauses use the bounded pause cadence regardless of agent liveness, while exited captain-held panes remain bounded
ok - a declared paused secondmate re-surfaces on the bounded normal-mode cadence
ok - a captain-held secondmate re-surfaces on the bounded normal-mode cadence
ok - a non-paused secondmate retains normal stale suppression
ok - a resumed secondmate clears pause and stale tracking before stale exemption
ok - unchanged stale hashes reclassify when a crew enters or leaves pause
ok - a declared pause is periodically rechecked against authoritative active-run state
ok - a paused status overridden by authoritative working preserves its wedge timer and escalates
ok - matching non-terminal stale suppressors repair missing or corrupt stale-since timers
ok - a quiet pane writing its own worktree is deferred, while one writing nothing still wedge-escalates on the unchanged schedule
ok - a write deferral re-surfaces once on the bounded pause cadence, so a churning worktree cannot stay invisible
ok - a secondmate's own home supervision churn is not crew write evidence, so a pane recording that home keeps the unchanged escalation schedule
ok - an idle-window timer repair drops a finished write-deferral chain, so the next deferral gets a fresh re-surface window
ok - both first-sight paths through a captain-relevant status drop a finished write-deferral chain with the idle window
ok - triage log capping handles wc byte counts with leading spaces
not ok - the fixture captured no process-event result
FM_TEST_END 2026-08-30T19:10:54Z tests/fm-watch-triage.test.sh exit=1 duration_ms=179755 gate_skip=false
FM_TEST_SUMMARY total=1 failed=1 skipped_gate=0 duration_ms=179798
FM_TEST_SUMMARY_FAMILY family=watcher-wake-lock count=1 duration_ms=179755 failed=1
FM_TEST_SLOWEST rank=1 script=tests/fm-watch-triage.test.sh duration_ms=179755
exit=1
Evidence: Classifier, daemon, docs-audience, AFK, daemon-lifecycle suites on target (199 ok, 0 failed)

Source: Classifier, daemon, docs-audience, AFK, daemon-lifecycle suites on target (199 ok, 0 failed)

FM_TEST_BEGIN 2026-08-30T19:08:55Z tests/fm-classify-corr-token.test.sh family=unclassified expected_gate_skip=none
ok - a correlation token between the verb and the key breaks neither opening nor closing
ok - the token is read through before the key, after it, doubled, bracketed, and unkeyed
ok - the untokened opener/closer pair behaves exactly as before
ok - prose, malformed, wrong-length and unknown name=value tokens stay non-transitions
ok - a token-first line cannot impersonate any opening or closing verb
ok - untokened captain-relevance, pause, terminal-verb and captain-held verdicts are unchanged
ok - captain-relevance, terminal-verb, pause and captain-held all read through the token
ok - the daemon and crew-state verb case arms read through the token
ok - reading through the token does not weaken the reserved-key namespace rule
ok - the cursor-backed fold and the whole-file fold agree on every correlated transition
ok - a cursor persisted under the previous reading is discarded and refolded
ok - both real correlation-token writers produce lines this classifier reads through
FM_TEST_END 2026-08-30T19:09:11Z tests/fm-classify-corr-token.test.sh exit=0 duration_ms=16012 gate_skip=false
FM_TEST_BEGIN 2026-08-30T19:09:11Z tests/fm-daemon.test.sh family=watcher-wake-lock expected_gate_skip=none
ok - fm-afk-start.sh fails before daemon startup when the afk flag cannot be written
ok - fm-afk-start.sh ignores stale pidfile-only live pids
ok - fm-afk-start.sh reclaims stale daemon locks whose live pid identity no longer matches
ok - supervise daemon state root is scoped by FM_HOME
ok - routine signal self-handles
ok - captain-relevant status verbs escalate
ok - check + unknown escalate; heartbeat self-handles
ok - transient stale self-handles and records a persistence marker
ok - enriched stale wedges bypass status absorption except under a declared wait, without disturbing busy workers
ok - an enriched wedge under a declared wait uses the pause cadence and restores wedge detection on resume
ok - stale + terminal status escalates immediately
ok - stale escalation and current wait cadence remain independent
ok - paused reasons with captain phrases remain pause-classified
ok - a captain-held transfer classifies as pause, not as a wedge candidate
ok - handle_wake on a paused stale records a pause marker, drops the wedge marker, and does not escalate
ok - handle_wake records a declared pause from a routine signal for long-cadence rechecks
ok - a terminal signal clears pause and stale tracking across both supervisors
ok - housekeeping migrates a normal-watcher's declared pause into daemon tracking
ok - housekeeping clears an already-resumed watcher pause across both supervisors
ok - housekeeping seeds pause tracking from status without a watcher marker
ok - persistent stale escalates after threshold and clears its marker
ok - resumed (busy) stale clears its marker without escalating
ok - housekeeping re-surfaces a stale declared pause on the long cadence and resets its window
ok - housekeeping re-surfaces a forgotten captain hold on the long cadence and resets its window
ok - a busy pane cannot gate the pause clear once its crew's status no longer declares the wait
ok - housekeeping matures a busy pane's declared-wait window into exactly one recheck per window
ok - housekeeping clears a paused marker once the crew is no longer declaring the pause
ok - housekeeping clears the pause marker once a captain hold is answered
ok - housekeeping moves an existing stale marker to pause before wedge escalation
ok - housekeeping moves a captain hold's existing stale marker to pause before wedge escalation
ok - housekeeping clears tracking when a crew leaves pause
ok - persistent herdr stale resolves the target from metadata and escalates
ok - herdr idle busy-footer stale clears through capture corroboration
ok - resumed herdr stale clears through backend-aware busy state
ok - persistent Orca stale resolves the terminal from metadata
ok - multiple escalations flush as a single batched digest
ok - batch flush measures max-delay from the first append, not the last
ok - catch-all scan escalates a missed terminal once, not twice
ok - handle_wake routes routine->self and captain->escalate
ok - INJECT_SKIP forces self-handle, bypassing captain-relevant classification
ok - is_wake_reason distinguishes watcher wake reasons from singleton-status stdout
ok - terminal-stale escalate removes its marker so housekeeping does not re-escalate
ok - captain signal escalate marks seen so the catch-all scan does not re-fire
ok - _collapse_newlines replaces newlines with literal separator
ok - afk flag absent: daemon does not inject, buffer preserved
ok - busy-guard defers injection when supervisor pane is busy
ok - marker detection: marker -> stay afk, no marker -> exit afk
ok - /afk invocation is exempt from afk exit (no self-cancel)
ok - should_exit_afk returns false when afk is not active
ok - strip_injection_marker removes the sentinel marker cleanly
ok - pane_input_pending detects partial input on the cursor line
ok - pane_input_pending: a blank unidentified cursor row defers (strict container-proof rule)
ok - pane_input_pending: only proven empty agent prompts pass
ok - fm_tmux_composer_state: a bare shell prompt ($/%/#/>) reads unknown, never empty (dead-shell injection safety)
ok - fm_tmux_composer_state: a bordered composer box and bare agent glyphs (❯/›) still read empty
ok - fm_tmux_composer_state: only matching edge borders form a composer box
ok - pane_input_pending preserves bright placeholder-like drafts in styled captures
ok - classify_signal dedupes against the catch-all scan seen marker
ok - turn-end markers stay routine while mixed actionable status batches escalate
ok - an actionable event is escalated to an away captain despite later routine appends
ok - the daemon commits exactly the endpoint captured by classification
ok - a stale wake escalates a blocker hidden by later progress
ok - a stale span read failure surfaces without advancing its marker
ok - a recreated status rejects the old captured identity
ok - an unverifiable status identity surfaces without advancing markers
ok - a status read failure surfaces without advancing the daemon suppressor
ok - catch-all routine scans advance before later actionable appends
ok - failed escalation writes retain durable wakes and classification positions
ok - catch-all markers advance only after escalation buffering succeeds
ok - classification failure retains the complete durable wake batch
ok - missing-status stale wakes remain ordinary and acknowledgeable
ok - transient unreadable signals recover without advancing their position
ok - daemon catch-all reclassifies permission-recovered status content
ok - a permanent classification failure is reported, acknowledged, and never advances its position
ok - the away-mode catch-all scan surfaces a masked event once
ok - classify_stale dedupes against the signal path seen marker
ok - AFK nonterminal working:+merged keeps wedge aging and re-escalates at bound
ok - genuine done: and merge-check events still escalate
ok - pane_input_pending: an idle bordered composer is NOT pending (afk-invx-i5)
ok - pane_input_pending: text inside a bordered composer is still pending
ok - submit-ACK confirms a submit when the composer returns to a bordered-empty box
ok - submit-ACK reports pending on a persistently swallowed Enter (type-once)
ok - max-defer on an empty stuck pane types once, alarms, and preserves the buffer
ok - max-defer flushes and clears the buffer on an empty bordered pane
ok - max-defer on a pending composer alarms without typing
ok - normal flush clears a stale wedge marker
ok - below MAX_DEFER: no inject, no alarm, buffer preserved
ok - max-defer does not flush or alarm while afk is inactive
ok - library mode: sourcing the daemon defaults FM_WEDGE_ALARM_EXEC to discard (no test can fire a real notification)
ok - wake helpers replace inherited notifier overrides with the safe recorder
ok - the discard seam suppresses every notifier, including command: (fires nothing)
ok - direct notifier helpers honor the discard seam, including command:
ok - osascript channel

... [3148 bytes truncated] ...

tly disappear
ok - local links resolve while dates, versions, commands, and incident prose remain semantically reviewed
FM_TEST_END 2026-08-30T19:09:39Z tests/fm-documentation-audiences.test.sh exit=0 duration_ms=631 gate_skip=false
FM_TEST_BEGIN 2026-08-30T19:09:39Z tests/fm-afk-return.test.sh family=afk expected_gate_skip=none
ok - return catch-up precedes Bearings, owns live blocker remediation, preserves evidence once, and clears idempotently
ok - tmux and Herdr blockers require the same explicit durable reclassification before ordinary work
ok - needs-decision remains reportable without masquerading as a firstmate-actionable blocker
ok - AFK return re-drains published wakes until handling acknowledges
ok - away-mode re-entry fails closed while the prior return catch-up is pending
ok - check retries recorded terminal teardown and keeps catch-up gated until success
FM_TEST_END 2026-08-30T19:09:41Z tests/fm-afk-return.test.sh exit=0 duration_ms=1759 gate_skip=false
FM_TEST_BEGIN 2026-08-30T19:09:41Z tests/fm-afk-launch.test.sh family=real-herdr-gated expected_gate_skip=herdr
ok - clear-stale: removes escalations buffer, sidecar, and wedge marker
ok - clear-stale: leaves the durable wake-queue intact (no pending work dropped)
ok - launcher paths: relative home and state ignore CDPATH before daemon command construction
ok - launcher paths: absolute symlink spellings are preserved
ok - launcher paths: unresolved relative FM_HOME fails loudly
ok - launcher paths: unresolved relative FM_STATE_OVERRIDE fails loudly
ok - refresh: daemon already alive - stale artifacts preserved (current session's buffer kept)
ok - stop-ordering: daemon SIGTERM'd while .afk still present (flush is not a no-op)
ok - stop-ordering: .afk cleared last
ok - stop-ordering: daemon-terminal record removed
ok - stop identity: stale lock cannot signal an unrelated live process
ok - failed start: away flag and delivery artifacts roll back
ok - concurrent start: one serialized daemon terminal remains tracked
ok - launcher lock: incomplete publication receives initialization grace
ok - launcher signal: TERM exits and releases the lifecycle lock
fm-afk-launch: daemon launched in non-visible herdr workspace ws-partial (pane lab:pane-exact), supervising lab:captain
ok - herdr create: malformed response recovers durable exact ownership
fm-afk-launch: herdr create failed after returning exact ids; closing lab:pane-exact
fm-afk-launch: recorded terminal teardown is unconfirmed; preserving exact id
ok - herdr create error: unconfirmed exact id is persisted for reconciliation
fm-afk-launch: failed to run daemon in herdr pane lab:pane-exact; closing it
fm-afk-launch: recorded terminal teardown is unconfirmed; preserving exact id
ok - herdr run failure: unconfirmed exact id remains reconcilable
fm-afk-launch: failed to persist daemon terminal record; closing tmux:exact-session
ok - record failure: newly created terminal is closed by exact id
fm-afk-launch: daemon did not become ready; closing tmux:exact-session
ok - readiness failure: exact terminal and durable record roll back
fm-afk-launch: daemon did not become ready; closing tmux:exact-session
fm-afk-launch: recorded terminal teardown is unconfirmed; preserving exact id
ok - readiness failure: unconfirmed terminal retains its reconciliation id
ok - tmux absence: clean missing differs from transport probe failure
ok - native lifecycle: launcher owns state with no terminal
ok - native lifecycle: uniform stop clears state without closing a terminal
ok - native entry: launcher-prepared lifecycle state is not rewritten
fm-afk-launch: reconciling leaked daemon terminal tmux:exact-session
fm-afk-launch: recorded terminal teardown is unconfirmed; preserving exact id
ok - teardown failure: exact terminal record is preserved
ok - record publication: failed atomic rename preserves the complete prior record
fm-afk-launch: daemon terminal record is malformed; refusing to act on it
ok - record read: malformed record fails closed without acting on a partial id
fm-afk-launch: daemon terminal record is malformed; refusing to act on it
fm-afk-launch: malformed daemon terminal record; refusing to stop away mode
ok - stop: malformed terminal record preserves away state and fails closed
fm-afk-launch: failed to create detached tmux daemon session 'fm-afk-daemon-581419948-1481616-14583-1788117013'
ok - tmux launch: planned exact target is recorded before creation and removed on failure
fm-afk-launch: failed to create detached tmux daemon session 'fm-afk-daemon-4135890725-1481679-24330-1788117013'
ok - tmux launch: unique names eliminate collision teardown
ok - stop validation: malformed record causes no daemon or state side effects
ok - launcher lock: incomplete metadata fails acquisition and releases lock
fm-afk-launch: failed to clear away-mode flag
fm-afk-launch: away mode stopped; terminal teardown remains recorded for retry
ok - stop state: away-flag removal failure is surfaced
fm-afk-launch: away-mode daemon did not exit after SIGTERM; preserving lifecycle state
ok - stop liveness: captured live daemon preserves lifecycle state after lock release
fm-afk-launch: daemon terminal record is malformed; refusing to act on it
fm-afk-launch: daemon terminal record is malformed; refusing to act on it
ok - refresh record: malformed terminal identity fails closed
fm-afk-launch: failed to clear stale away-mode artifacts
ok - clear failure: native entry aborts and restores prior state
fm-afk-launch: reconciling leaked daemon terminal tmux:exact-session
fm-afk-launch: terminal close command failed, but exact absence was confirmed
ok - confirmed absence: cleanup succeeds and removes the stale record
fm-afk-launch: rollback restoration incomplete; backup retained at /tmp/fm-afk-restore-fail.2xL1sF/state/.afk-launch-backup.o5wZtx
ok - rollback restore: incomplete restoration retains its recovery backup
ok - flag failure: lifecycle aborts without active state
ok - herdr e2e: captain tab pane count unchanged after start (no split)
ok - herdr e2e: daemon launched in a separate non-visible workspace
ok - herdr e2e: daemon pane is NOT in the captain's tab
ok - herdr e2e: daemon terminal scoped to the lab session
ok - herdr e2e: captain tab pane count restored after stop
ok - herdr e2e: daemon workspace removed by exact id on stop
ok - herdr e2e: record + .afk cleared on stop
ok - tmux e2e: captain window pane count unchanged after start (no split-window)
ok - tmux e2e: daemon launched in a separate detached session
ok - tmux e2e: captain window pane count unchanged after stop
ok - tmux e2e: daemon session killed by exact id on stop
ok - tmux e2e: record + .afk cleared on stop
FM_TEST_END 2026-08-30T19:10:14Z tests/fm-afk-launch.test.sh exit=0 duration_ms=33366 gate_skip=false
FM_TEST_BEGIN 2026-08-30T19:10:14Z tests/fm-wake-daemon-lifecycle-e2e.test.sh family=watcher-wake-lock expected_gate_skip=none
ok - lifecycle: routine self-handles, terminal survives a watcher restart, buffers once, no dup, injects once
ok - lifecycle: stale pane transient self-handles, persistent escalates once and clears, resumed clears quietly
FM_TEST_END 2026-08-30T19:10:21Z tests/fm-wake-daemon-lifecycle-e2e.test.sh exit=0 duration_ms=7133 gate_skip=false
FM_TEST_SUMMARY total=6 failed=0 skipped_gate=0 duration_ms=86077
FM_TEST_SUMMARY_FAMILY family=afk count=1 duration_ms=1759 failed=0
FM_TEST_SUMMARY_FAMILY family=pure-contract-unit count=1 duration_ms=631 failed=0
FM_TEST_SUMMARY_FAMILY family=real-herdr-gated count=1 duration_ms=33366 failed=0
FM_TEST_SUMMARY_FAMILY family=unclassified count=1 duration_ms=16012 failed=0
FM_TEST_SUMMARY_FAMILY family=watcher-wake-lock count=2 duration_ms=34150 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-afk-launch.test.sh duration_ms=33366
FM_TEST_SLOWEST rank=2 script=tests/fm-daemon.test.sh duration_ms=27017
FM_TEST_SLOWEST rank=3 script=tests/fm-classify-corr-token.test.sh duration_ms=16012
FM_TEST_SLOWEST rank=4 script=tests/fm-wake-daemon-lifecycle-e2e.test.sh duration_ms=7133
FM_TEST_SLOWEST rank=5 script=tests/fm-afk-return.test.sh duration_ms=1759
FM_TEST_SLOWEST rank=6 script=tests/fm-documentation-audiences.test.sh duration_ms=631
exit=0
Evidence: Procevent failure diagnosis: pre-existing on clean base commit, environment-linked

Source: Procevent failure diagnosis: pre-existing on clean base commit, environment-linked

FM_TEST_BEGIN 2026-08-30T19:13:21Z tests/fm-procevent.test.sh family=unclassified expected_gate_skip=none
ok - no configured source means no generated state and no process
not ok - reconcile never claimed the registered source
FM_TEST_END 2026-08-30T19:13:31Z tests/fm-procevent.test.sh exit=1 duration_ms=10447 gate_skip=false
FM_TEST_SUMMARY total=1 failed=1 skipped_gate=0 duration_ms=10489
FM_TEST_SUMMARY_FAMILY family=unclassified count=1 duration_ms=10447 failed=1
FM_TEST_SLOWEST rank=1 script=tests/fm-procevent.test.sh duration_ms=10447
exit=1

=== corroboration: same failure on clean base commit d71f4b9 ===
--- dedicated suite tests/fm-procevent.test.sh on base tree:
ok - no configured source means no generated state and no process
not ok - reconcile never claimed the registered source
BASE dedicated procevent exit=1
--- fm-watch-triage's procevent test isolated on base tree: 'not ok - the fixture captured no process-event result' (see base-commit-repro method)
- Outcome: ⚠️ 1 warning across 1 run (11m2s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 2 infos
  • ℹ️ bin/fm-watch.sh:732 - pause_state_class now short-circuits to 'paused' via status_pause_damps_wedge at lines 734-737 and 755-756, but kind=$(window_kind &#34;$win&#34;) on line 732 (a meta-file scan through fm_backend_meta_for_window + grep) is still evaluated before that check and is only consumed by the captain-held liveness gates. Every stale poll of a paused: pane now pays a metadata scan whose result is never used, and the comment above it ("reused by both liveness gates below") no longer describes the paused path. Move the window_kind read below the first status_pause_damps_wedge return (or into the two captain-held branches) so the declared-pause path keeps the zero-cost profile the comment promises.
  • ℹ️ Three of the four commits carry internal pipeline subjects (no-mistakes(document): ..., no-mistakes(review): ...). No Co-Authored-By trailers, local paths, or private data are present, so the intent's forbidden-content constraints hold; this is only a note for the PR step that these pipeline-stage prefixes will be visible in the upstream kunchenguid/firstmate PR history unless the branch is squashed or the messages are reworded there.
⚠️ **Test** - 1 warning
  • ⚠️ tests/fm-watch-triage.test.sh:2793 - tests/fm-watch-triage.test.sh cannot go fully green in this environment: test_procevent_captured_result_surfaces_proactively fails because the detached process-event runner never publishes to the durable wake queue (register+reconcile report started=1 but no .wake-queue row ever appears). This is pre-existing and unrelated to the change: the identical test and the dedicated tests/fm-procevent.test.sh fail the same way on the clean base commit d71f4b9. All 59 other watcher-triage tests, including every pause/wedge regression, pass. Remote CI on its own runners owns this coverage; the user should decide whether the local procevent environment gap needs separate investigation.
  • bin/fm-test-run.sh tests/fm-watch-triage.test.sh (target 0a42e3a: 59 ok; 1 pre-existing procevent env failure)
  • bin/fm-test-run.sh tests/fm-classify-corr-token.test.sh tests/fm-daemon.test.sh tests/fm-documentation-audiences.test.sh tests/fm-afk-return.test.sh tests/fm-afk-launch.test.sh tests/fm-wake-daemon-lifecycle-e2e.test.sh (6 suites, 199 ok, 0 failed, 86s)
  • Before-fix reproduction: test_declared_pause_and_exited_captain_hold_are_bounded and test_paused_busy_flicker_keeps_resurface_cadence run against base-commit d71f4b9 binaries via git-archive tree in /tmp — both fail there (not ok - a live declared pause surfaced immediately, not ok - a busy pane erased the declared pause&#39;s re-surface throttle) and pass on target
  • Manual end-user demo driving real bin/fm-watch.sh: declared pause absorbed silently with no wedge timer; aged pause re-surfaces once labeled awaiting external with no possible wedge/demand-deep-inspection; newer working: status clears pause tracking and restarts the wedge timer; frozen pane past threshold escalates possible wedge again
  • Isolated re-runs of the failing procevent test on target and base trees plus bin/fm-procevent.sh register/reconcile manual reproduction proving the failure is pre-existing and environment-linked, not introduced by this change
  • Checked commits/diff for private data, credentials, local paths, and agent co-author trailers: none present; diff confined to the intended classifier/watcher/daemon/docs/test files
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2c35a9345f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bin/fm-watch.sh Outdated
Comment on lines 377 to 379
if status_pause_damps_wedge "$last" \
|| { ! afk_present && status_is_paused_or_captain_held "$last"; }; then
handle_paused_stale "$win" "$task" "$h"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reconcile the run before damping away-mode wedges

Captain, when AFK is active and a task starts an authoritative run without appending a new status event, an older paused: line still makes this new first clause call handle_paused_stale before any fm-crew-state.sh read. That clears the short wedge timer and leaves a hung busy run on repeated hour-long pause rechecks, so the old pause can mask exactly the busy-run wedge this bound detects; gate the pause on current-state reconciliation so run-step working retains precedence.

AGENTS.md reference: AGENTS.md:L133-L133

Useful? React with 👍 / 👎.

@Omar-Nawaf
Omar-Nawaf force-pushed the fm/firstmate-declared-pause-wedge-upstream-pr branch from 2c35a93 to 1427342 Compare August 21, 2026 14:44
@Omar-Nawaf
Omar-Nawaf force-pushed the fm/firstmate-declared-pause-wedge-upstream-pr branch from 1427342 to bdf8b07 Compare August 21, 2026 19:01
@Omar-Nawaf
Omar-Nawaf force-pushed the fm/firstmate-declared-pause-wedge-upstream-pr branch from bdf8b07 to 8910550 Compare August 26, 2026 09:00
@Omar-Nawaf Omar-Nawaf changed the title fix(bin): keep declared pauses off wedge cadence fix(bin): keep a current declared pause off wedge escalation in watcher and daemon Aug 26, 2026
@greptile-apps

greptile-apps Bot commented Aug 26, 2026

Copy link
Copy Markdown

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Reviews (6): Last reviewed commit: "no-mistakes: apply CI fixes" | Re-trigger Greptile

@Omar-Nawaf
Omar-Nawaf force-pushed the fm/firstmate-declared-pause-wedge-upstream-pr branch from 8910550 to c88c8ff Compare August 27, 2026 06:14
@Omar-Nawaf Omar-Nawaf changed the title fix(bin): keep a current declared pause off wedge escalation in watcher and daemon fix(bin): keep current declared pauses on the bounded recheck cadence instead of wedge escalation Aug 27, 2026
@Omar-Nawaf
Omar-Nawaf force-pushed the fm/firstmate-declared-pause-wedge-upstream-pr branch from c88c8ff to 2ee8409 Compare August 27, 2026 06:31
@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate:

Reviewed HEAD 2ee840966b656ceb5ef5219685e69777d58894b8 vs main 10b93b2cc6f4241e87fccaee2e357c33a7347a53. Whole thread read (Greptile success on this HEAD; Codex on an older SHA). Full diff reviewed: bin/fm-classify-lib.sh, bin/fm-watch.sh, bin/fm-supervise-daemon.sh, docs/architecture.md, docs/configuration.md, tests/fm-watch-triage.test.sh. No .github/workflows/*. Omar-Nawaf is not blocked. Not disguised security.

Class: corrective for the remaining #2614 live/idle declared-pause path. status_pause_damps_wedge in fm-classify-lib.sh is a latest-line paused: predicate. pause_state_class now admits a current paused: declaration to the FM_PAUSE_RESURFACE_SECS cadence without an agent-liveness read (captain-held keeps the stricter dead-agent gate). clear_pause_state keeps .paused-resurfaced-<key> across a short busy flicker so the throttle belongs to the declaration. Author body says Fixes #2614; inspected remaining code matches that idle/live declared-pause wedge, not the busy-pane #3149 class.

#3155 / #3147 leftover check: both are merged (#3147 2026-08-27 02:38 PT-converted 02:38Z / #3155 06:15Z). Against current main this HEAD's daemon hunk is comment-only ("idle pane" → "unattended pane"); busy_turn_bound_check is comment-only (the undecorated away-mode handoff is already on main from #3147). That is not leftover functional overlap with the merged busy-pane pair. The remaining delta is a distinct #2614 fix. Do not claim this closes #3149 (#3155 already closed it).

Competing open leftover on the same remaining pause_state_class liveness gate: #3160 (wjkawecki-jt). Not a standing hold pair, and not a reason to treat #2738 as leftover of #3155/#3147. Still not merging this HEAD.

VISION (per rule, inspected evidence):

  • One captain, one interface — aligns. A current paused: wait was burning handling turns / wedge-escalating; this keeps the absorb honest and the long-cadence recheck visible (pause_state_class + rewritten live-pause test).
  • Authority is explicit — aligns. No new autonomy; captain-held still fail-open surfaces a live decision gate.
  • Scripts own the mechanics — aligns. Classifier predicate + watcher bookkeeping; no agent judgment mixed in.
  • A restart is a non-event — aligns. Durable pause markers / resurface throttle remain disk records.
  • Delegation with a spine — aligns. Restores the existing away-mode / pause contract, not a new task shape.
  • The fleet outlives any vendor — aligns. Status-line declaration, not a vendor footer.
  • Scope — aligns. Command-layer supervision, not workshop.

Attestation: mismatch. Body stamps head_sha=c88c8ffb…; PR head is 2ee84096…. Require-no-mistakes failed on 33046154853. Waiting on the author: re-run git push no-mistakes so the structured stamp binds to current HEAD.

CI: current-HEAD CI run 33046154902 is still in progress. Not merge-ready.

First-time fork CI: current-HEAD CI 33046154902 is already in flight. I did not approve stale SHA action_required runs.

Not merging. Waiting on the author for a HEAD-matching attestation (and green CI). Distinct remaining #2614 fix, but auto-merge gates are not met. Not with the captain.

Merge-eligible: NO. Captain-flag NOW: NO.

@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate:

Reviewed HEAD 2ee840966b656ceb5ef5219685e69777d58894b8 vs main 10b93b2cc6f4241e87fccaee2e357c33a7347a53. Whole thread read (no prior firstmate stamp). Full diff reviewed: bin/fm-classify-lib.sh (status_pause_damps_wedge), bin/fm-watch.sh (pause_state_class / busy-flicker cadence), bin/fm-supervise-daemon.sh (comment-only wording), docs, and tests/fm-watch-triage.test.sh. No .github/workflows/*. Not disguised security. Omar-Nawaf is not blocked.

Class: corrective. Targets #2614: a current paused: declaration should own FM_PAUSE_RESURFACE_SECS instead of repeated idle/busy possible-wedge / demand-deep-inspection escalation. Diff admits paused: on the latest status line without requiring agent-dead, keeps busy-flicker from erasing the pause re-surface throttle, and restores ordinary wedge aging when the declaration lifts.

VISION (per rule, inspected evidence):

  • One captain, one interface — aligns. Stops burning handling turns on declared external waits that should self-handle on the long cadence (issue body + watch/classify paths).
  • Authority is explicit — aligns. No new autonomy; honors the crew's own declared wait.
  • Scripts own the mechanics — aligns. Shared classifier predicate + watcher bookkeeping.
  • A restart is a non-event — aligns. Durable pause markers / recheck cadence preserved across busy flicker.
  • Delegation with a spine — aligns. Does not suppress undeclared wedges; lifting pause restores genuine stale.
  • The fleet outlives any vendor — aligns. Status-line vocabulary, not a harness UI scrape.
  • Scope — aligns. Watcher/classifier supervision, not workshop.

Related merged work: #3155 squash-merged and closed #3149 (daemon declared-wait under busy pane); #3147 was the watcher handoff half. This PR does not close #3149. Daemon hunk here is comment-only. It still shares busy-declared-wait / pause-cadence territory with that landed pair.

Leftover overlap with open #3160 (wjkawecki-jt): both rewrite pause_state_class so a live paused: endpoint stays on the bounded cadence / off bare durable stale wakes. Do not land one leftover while the other is open.

Would close #2614 from body+code if this were the surviving leftover (author Fixes #2614; inspected classify/watch changes match). Fetch why was not treated as proof.

Attestation: head_sha c88c8ffb… ≠ current HEAD 2ee84096…MISMATCH. Waiting on the author to rebind via git push no-mistakes.

First-time fork CI: after that full-diff review I approved workflow runs 33046154902 (CI) and 33046154853 (Require no-mistakes) for this HEAD.

Not merging. Overlap leftover with #3160 (+ conceptual adjacency to merged #3147/#3155) and attestation mismatch. Not with the captain. Merge-eligible: NO.

@Omar-Nawaf
Omar-Nawaf force-pushed the fm/firstmate-declared-pause-wedge-upstream-pr branch from 14c46dd to 20e0811 Compare September 2, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants