fix(bin): keep crews with a current declared pause off wedge escalation - #2738
fix(bin): keep crews with a current declared pause off wedge escalation#2738Omar-Nawaf wants to merge 5 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c35a9345f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if status_pause_damps_wedge "$last" \ | ||
| || { ! afk_present && status_is_paused_or_captain_held "$last"; }; then | ||
| handle_paused_stale "$win" "$task" "$h" |
There was a problem hiding this comment.
Reconcile the run before damping away-mode wedges
Captain, when AFK is active and a task starts an authoritative run without appending a new status event, an older paused: line still makes this new first clause call handle_paused_stale before any fm-crew-state.sh read. That clears the short wedge timer and leaves a hung busy run on repeated hour-long pause rechecks, so the old pause can mask exactly the busy-run wedge this bound detects; gate the pause on current-state reconciliation so run-step working retains precedence.
AGENTS.md reference: AGENTS.md:L133-L133
Useful? React with 👍 / 👎.
2c35a93 to
1427342
Compare
1427342 to
bdf8b07
Compare
bdf8b07 to
8910550
Compare
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Reviews (6): Last reviewed commit: "no-mistakes: apply CI fixes" | Re-trigger Greptile |
8910550 to
c88c8ff
Compare
c88c8ff to
2ee8409
Compare
|
Speaking as Kun's firstmate: Reviewed HEAD Class: corrective for the remaining #2614 live/idle declared-pause path. #3155 / #3147 leftover check: both are merged ( Competing open leftover on the same remaining VISION (per rule, inspected evidence):
Attestation: mismatch. Body stamps CI: current-HEAD CI run First-time fork CI: current-HEAD CI Not merging. Waiting on the author for a HEAD-matching attestation (and green CI). Distinct remaining #2614 fix, but auto-merge gates are not met. Not with the captain. Merge-eligible: NO. Captain-flag NOW: NO. |
|
Speaking as Kun's firstmate: Reviewed HEAD Class: corrective. Targets #2614: a current VISION (per rule, inspected evidence):
Related merged work: #3155 squash-merged and closed #3149 (daemon declared-wait under busy pane); #3147 was the watcher handoff half. This PR does not close #3149. Daemon hunk here is comment-only. It still shares busy-declared-wait / pause-cadence territory with that landed pair. Leftover overlap with open #3160 (wjkawecki-jt): both rewrite Would close #2614 from body+code if this were the surviving leftover (author Attestation: First-time fork CI: after that full-diff review I approved workflow runs Not merging. Overlap leftover with #3160 (+ conceptual adjacency to merged #3147/#3155) and attestation mismatch. Not with the captain. Merge-eligible: NO. |
2ee8409 to
0a42e3a
Compare
0a42e3a to
14c46dd
Compare
…le; explain kept throttle
14c46dd to
20e0811
Compare
Intent
Publish the already-implemented declared-pause wedge correction as one upstream PR for kunchenguid/firstmate issue #2614 now that publication is authorized and the maintainer marked it ready for PR. On current origin/main, a current declared paused: external wait must use FM_PAUSE_RESURFACE_SECS and must never produce repeated idle/busy possible-wedge or demand-deep-inspection escalation in ordinary or away-mode supervision; a newer non-pause status must supersede the wait and restore normal bounded wedge detection so an old pause cannot hide a real wedge forever, while authoritative active-run precedence remains intact. Keep this policy in the existing shared classifier owner, preserve non-paused max-defer and wedge reliability, and do not broaden the change into speculative busy-pane damping. Preserve or improve executable regressions for paused suppression, pause supersession, ordinary escalation, away-mode watcher routing, and daemon rechecks; run focused classifier, watcher, daemon, and AFK suites, documentation audience checks, exact bin/fm-lint.sh, and the complete no-mistakes pipeline. Port only the intended correction from preserved commit d6f4659 onto current origin/main without altering the preserved source branch or old copy, and include no private data/state/config, credentials, local paths, project content, agent co-author, or unrelated changes. The external-healthy-watcher test was treated as branch-relevant, reproduced identically on this branch and clean origin/main, and proven to fail only under Node 22 TS import support while both pass unchanged under available Linux Node 24, so do not weaken or alter that unrelated assertion. Push only fm/firstmate-declared-pause-wedge-upstream-pr through the captain-authorized Omar-Nawaf/firstmate fork and open one PR against kunchenguid/firstmate:main referencing #2614; never merge.
What Changed
status_pause_damps_wedgetobin/fm-classify-lib.shand applied it inbin/fm-watch.sh'spause_state_class, so a crew whose latest status declares apaused:external wait takes the boundedFM_PAUSE_RESURFACE_SECSrecheck regardless of agent liveness or pane busy state, instead of surfacing immediately or climbing possible-wedge / demand-deep-inspection escalation; a newer non-pause status line restores ordinary wedge detection, and an authoritative active-run verdict keeps its existing precedence..paused-resurfaced-<key>) across a busy flicker:clear_pause_stateno longer deletes it, and the main loop drops it only once the latest status stops declaring a wait, so a short busy turn that appends no status line cannot trigger an off-cadence recheck while the bounded recheck still fires once the throttle ages past the cadence.tests/fm-watch-triage.test.shwith a busy-flicker cadence regression and reworked the live-pause test to pin the bounded cadence over the old liveness override, and aligneddocs/architecture.md,docs/configuration.md, and the classifier/watcher/daemon comments with the latest-status rule.Risk Assessment
✅ Low: The change is confined to the declared-pause branch of pause_state_class plus the resurface-throttle lifecycle, keeps the policy in the shared classifier, leaves authoritative run-step precedence and all non-paused wedge paths untouched, and the traced state transitions (supersession, busy flicker, away-mode handoff, daemon clear) all keep the pause bounded rather than silenced, with executable tests pinning each behavior.
Testing
Exercised the intent's full acceptance surface: proved both new pause-wedge regressions fail on base d71f4b9 and pass on target; ran the focused watcher, classifier, daemon, AFK, daemon-lifecycle, and documentation-audience suites (all green except one pre-existing, base-reproducible procevent environment failure unrelated to the change); and captured a live fm-watch.sh CLI transcript demonstrating pause absorb, the bounded FM_PAUSE_RESURFACE_SECS recheck without possible-wedge or deep-inspection escalation, supersession by a newer non-pause status, and restored wedge escalation afterward. Lint and full-pipeline phases are owned by the outer executor. No UI surface is involved, so the CLI transcript is the reviewer-visible artifact.
Evidence: Pause-lifecycle end-user watcher transcript (absorb → bounded recheck → supersede → wedge restored)
Source: Pause-lifecycle end-user watcher transcript (absorb → bounded recheck → supersede → wedge restored)
== Phase 1: crew declares a pause; pane sits idle and stale == watcher wake> (none - absorbed silently) state: .paused marker recorded (long-cadence tracking) state: no wedge timer started, no possible-wedge escalation == Phase 2: pause ages past FM_PAUSE_RESURFACE_SECS -> ONE bounded recheck == watcher wake> stale: test:fm-demo-crew (paused 501s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds) => no 'possible wedge', no 'demand-deep-inspection' in the wake == Phase 3: newer non-pause status supersedes the wait == state: pause tracking cleared - wait superseded state: wedge timer restarted - bounded wedge detection restored == Phase 4: pane still frozen past the wedge threshold -> real wedge escalates == watcher wake> stale: test:fm-demo-crew (idle 501s, possible wedge, escalation 1)Evidence: Before-fix reproduction: both regression tests fail on base commit d71f4b9
Source: Before-fix reproduction: both regression tests fail on base commit d71f4b9
=== BASE d71f4b9: test_declared_pause_and_exited_captain_hold_are_bounded === not ok - a live declared pause surfaced immediately: stale: test:fm-gate exit=1 === BASE d71f4b9: test_paused_busy_flicker_keeps_resurface_cadence === not ok - a busy pane erased the declared pause's re-surface throttle exit=1Evidence: Watcher triage suite on target (59 ok incl. all pause/wedge/AFK regressions; 1 pre-existing procevent env failure)
Source: Watcher triage suite on target (59 ok incl. all pause/wedge/AFK regressions; 1 pre-existing procevent env failure)
Evidence: Classifier, daemon, docs-audience, AFK, daemon-lifecycle suites on target (199 ok, 0 failed)
Source: Classifier, daemon, docs-audience, AFK, daemon-lifecycle suites on target (199 ok, 0 failed)
Evidence: Procevent failure diagnosis: pre-existing on clean base commit, environment-linked
Source: Procevent failure diagnosis: pre-existing on clean base commit, environment-linked
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
bin/fm-watch.sh:732- pause_state_class now short-circuits to 'paused' via status_pause_damps_wedge at lines 734-737 and 755-756, butkind=$(window_kind "$win")on line 732 (a meta-file scan through fm_backend_meta_for_window + grep) is still evaluated before that check and is only consumed by the captain-held liveness gates. Every stale poll of a paused: pane now pays a metadata scan whose result is never used, and the comment above it ("reused by both liveness gates below") no longer describes the paused path. Move the window_kind read below the first status_pause_damps_wedge return (or into the two captain-held branches) so the declared-pause path keeps the zero-cost profile the comment promises.no-mistakes(document): ...,no-mistakes(review): ...). No Co-Authored-By trailers, local paths, or private data are present, so the intent's forbidden-content constraints hold; this is only a note for the PR step that these pipeline-stage prefixes will be visible in the upstream kunchenguid/firstmate PR history unless the branch is squashed or the messages are reworded there.tests/fm-watch-triage.test.sh:2793- tests/fm-watch-triage.test.sh cannot go fully green in this environment: test_procevent_captured_result_surfaces_proactively fails because the detached process-event runner never publishes to the durable wake queue (register+reconcile report started=1 but no .wake-queue row ever appears). This is pre-existing and unrelated to the change: the identical test and the dedicated tests/fm-procevent.test.sh fail the same way on the clean base commit d71f4b9. All 59 other watcher-triage tests, including every pause/wedge regression, pass. Remote CI on its own runners owns this coverage; the user should decide whether the local procevent environment gap needs separate investigation.bin/fm-test-run.sh tests/fm-watch-triage.test.sh(target 0a42e3a: 59 ok; 1 pre-existing procevent env failure)bin/fm-test-run.sh tests/fm-classify-corr-token.test.sh tests/fm-daemon.test.sh tests/fm-documentation-audiences.test.sh tests/fm-afk-return.test.sh tests/fm-afk-launch.test.sh tests/fm-wake-daemon-lifecycle-e2e.test.sh(6 suites, 199 ok, 0 failed, 86s)Before-fix reproduction:test_declared_pause_and_exited_captain_hold_are_boundedandtest_paused_busy_flicker_keeps_resurface_cadencerun against base-commit d71f4b9 binaries via git-archive tree in /tmp — both fail there (not ok - a live declared pause surfaced immediately,not ok - a busy pane erased the declared pause's re-surface throttle) and pass on targetManual end-user demo driving realbin/fm-watch.sh: declared pause absorbed silently with no wedge timer; aged pause re-surfaces once labeledawaiting externalwith nopossible wedge/demand-deep-inspection; newerworking:status clears pause tracking and restarts the wedge timer; frozen pane past threshold escalatespossible wedgeagainIsolated re-runs of the failing procevent test on target and base trees plusbin/fm-procevent.sh register/reconcilemanual reproduction proving the failure is pre-existing and environment-linked, not introduced by this changeChecked commits/diff for private data, credentials, local paths, and agent co-author trailers: none present; diff confined to the intended classifier/watcher/daemon/docs/test files✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.