fix(pi): coalesce ordinary watcher wake rows in the follow-up dock - #3472
fix(pi): coalesce ordinary watcher wake rows in the follow-up dock#3472npayette84 wants to merge 8 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Reviews (3): Last reviewed commit: "fix(pi): rebase wake coalescing onto the..." | Re-trigger Greptile |
4a6d1b0 to
1596472
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 15964722f3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const urgent = presentation === "urgent" || isUrgentSupervisionFailure(message); | ||
| // The already-pending row directs the model to drain the durable queue, so | ||
| // this event is presented by that row; its durable record is untouched. | ||
| if (!urgent && owner.pendingOrdinaryWakeRow) return true; |
There was a problem hiding this comment.
Preserve wakes suppressed after an inline drain
Captain, when an ordinary event arrives after a busy run has consumed the docked row and drained the durable queue, but before agent_settled, this early return suppresses the event's only notification. The settle handler only clears the latch, so if no subsequent event or captain turn occurs, the idle session never drains that newly queued wake; retain that a wake was suppressed and enqueue a replacement row at settle, or clear the latch on the matching inline-consumption edge. The new busy-path test triggers its later event only after settling, so it misses this timing.
AGENTS.md reference: AGENTS.md:L401-L404
Useful? React with 👍 / 👎.
…ures Pi's fixed follow-up dock gained one queued row per ordinary watcher notification, so a burst of actionable closes during one long handling turn buried the conversation under rows even though every event was already durable in the wake queue. Presentation now separates from delivery: while one ordinary work-waiting row is pending, later ordinary wakes add no row. The per-generation latch is set before the delivery await, because on an idle agent Pi runs the whole handling turn inside sendUserMessage and emits agent_start before that call resolves; a latch set after the await would strand itself and suppress every later ordinary wake. It clears on agent_start, the edge where the pending row is consumed, and rolls back when delivery rejects before a row exists. Urgent supervision failures bypass the latch from both directions: call sites mark the paths they own, and any message carrying the `watcher: FAILED` marker is urgent whichever branch delivered it, so continuity-restoration exhaustion and restoration-time lock loss, which ride the ordinary delivery branch, still present their own row while a row is pending. The durable wake queue is untouched, and ownership, successor restoration, and no-redundant-arm behavior are unchanged. Regressions drive the real extension through a production-timing follow-up fake built from Pi 0.84.4's runtime ordering, and each was run against deliberately broken variants of the extension to prove it is not vacuous.
…ent contract Rebasing onto upstream main brought in the tokenized wake acknowledgement added by kunchenguid#3498: an actionable wake settles only when Pi reports the docked row back through before_agent_start, and delivery of the next pending close waits on that. The coalescing fakes now announce consumption the same way, so a coalesced ordinary wake reports as presented by the row already docked instead of stalling the pending-delivery loop. The discarded-dock regression is dropped on this base: a row that is never consumed leaves the pending-delivery loop waiting on its acknowledgement, so no later wake reaches presentation at all and the latch is no longer what the scenario observes. The agent_settled clearing edge stays covered by the busy-path re-arm test. docs/verification/supervision.md records the re-run counterfactual matrix on this base, including that the latch-after-send variant no longer fails any test there.
…contract Upstream main now settles a main delivery once Pi accepts the follow-up and observes consumption at before_agent_start for an idle main and at the user message_start for a streaming one (kunchenguid#3513), replacing the acknowledgement contract this change was built against. Presentation coalescing moves on top of that contract unchanged in behavior: sendWake keeps its WakePresentation argument next to the base's pending-record argument, sets the ordinary latch before the delivery await, rolls it back in the catch that already drops the unconsumed-wake record, and a coalesced wake never enters that map, so its pending record is finished rather than left waiting for a consumption that will never come. The base's streaming-time delivery regression drove two actionable closes through one streaming run and expected a docked row for each. Coalescing presents the second through the row already docked, so it now proves the successor chain advances without a second row and takes its unconsumed handoff wake from a later close raised after that row is consumed at message_start and its run settles. The verification record names the new base, and the counterfactual matrix was re-run there: unmodified upstream fails only the burst test, each single clearing edge fails only its own re-arm test, call-site-only urgency fails both restoration-failure tests, removing urgency fails all four failure-bypass tests, and a latch shared across generations fails only the session-replacement test.
1596472 to
f88f9d1
Compare
Intent
Implement the captain-approved corrected Pi Follow-up wake presentation coalescing patch identified by a judge report that rejected two prior candidate branches, shipping it as a fresh change on current upstream/main.
Background the reviewer needs: Pi's fixed follow-up dock gained one queued row per ordinary watcher notification, so a burst of actionable watcher closes during one long handling turn could stack more than sixty rows and bury the conversation, even though every event was already durable in the wake queue. Two earlier candidate branches each solved part of this and each failed a required criterion, so neither was promoted; this change is deliberately the union of what each got right, roughly 30 functional extension lines plus regressions, and NOT a port of either candidate wholesale.
Required behavior, all accepted and implemented:
watcher: FAILEDclassification IS combined with candidate B's explicit "urgent" call-site marks. The call-site marks are currently redundant with the content classifier for every existing failure text; that redundancy is intentional defence in depth so a future failure path that forgets the marker still surfaces. Do not collapse the WakePresentation parameter into the content check, and do not weaken urgent failure visibility.Known and accepted residual, documented in the extension header, not a defect to fix: Pi exposes no per-row consumption signal, so a captain prompt that starts a run while a row is still docked clears the latch early and a close during that run can dock a second row. That transient is bounded at two rows, both consumed in order, and is the accepted cost of using agent_start as the consumption edge.
Regression evidence required and delivered, all through the real extension boundary (no toy reimplementation): a production-timing fake sendUserMessage where an idle send fires agent_start inside the call and resolves only after the driver-controlled run completes, proving a genuinely later idle wake presents exactly one new row; a busy-agent case where one ordinary row is already docked and a second close hits restoration exhaustion, and a second where it hits restoration-time lock loss, proving each typed failure still presents separately; plus burst, re-arm, later-event, generation replacement, durable-record preservation, and delivery-rejection rollback coverage. Both prior candidates' test suites faked sendUserMessage as instantly resolving, which is exactly what hid their defects, so the production-timing fake is the point of these tests and must not be simplified back to an instant-resolve fake.
Every new test was additionally run against six deliberately broken variants of the extension to prove none is vacuous: unmodified upstream fails only the burst test; latch-after-await fails only the idle-consumed re-arm test; call-site-only urgency fails both restoration-failure tests; no rejection rollback fails only the rejected-delivery test; a latch shared across generations fails only the session-replacement test; and removing urgency entirely fails all four failure-bypass tests.
Scope decisions the reviewer should not second-guess:
Verified green before submission: tests/fm-pi-watch-extension.test.sh (45 tests), tests/fm-watch-arm.test.sh (14), tests/fm-watch-recovery-loop.test.sh (2), tests/fm-pi-branch-extension.test.sh (31), tests/fm-calm-pi-extension.test.sh (9), tests/fm-turnend-guard.test.sh (70), CI=true bin/fm-lint.sh under pinned ShellCheck 0.11.0 and actionlint 1.7.12, and bin/fm-doc-audience-check.sh.
The PR must target current upstream main and stop green for captain review; do not merge it.
What Changed
.pi/extensions/fm-primary-pi-watch.tsnow tracks a per-generationpendingOrdinaryWakeRowlatch: an ordinary watcher wake docks a follow-up row only when no earlier ordinary row is pending. The latch is set before callingpi.sendUserMessage(which isvoidon Pi 0.84.4'sExtensionAPI, so nothing can be awaited or observed) and cleared on bothagent_startandagent_settled, the two run boundaries that show a docked row was consumed or discarded. A replacement session activates a fresh generation with a clear latch. Delivery is untouched — every event is still enqueued durably by the watcher; only presentation coalesces.sendWaketakes an explicitWakePresentationmark from the call sites that own failure paths, and any message carrying thewatcher: FAILEDmarker is treated as urgent regardless of the branch that delivered it. Typed continuity-restoration failures ride the ordinary delivery branch, so the content check keeps them from being coalesced behind a pending row.tests/fm-pi-watch-extension.test.shgains nine regressions driving the real extension through a follow-up fake that models Pi's actualvoidsend surface and its run transitions — burst coalescing with durable-queue preservation, re-arm after idle-consumed, busy-path inline-drained, and discarded rows, four failure-bypass cases, and generation replacement.docs/watcher-continuity.mdstates the operator-facing guarantee and points at the extension header for the full contract;docs/verification/supervision.mdrecords the dated verification run, the runtime surfaces it was read from, the broken-variant counterfactuals, theskip: tsc not foundresult, and the applicability review across primaries and runtime backends.CONTRIBUTING.mdnotes the bash 3.2 heredoc-in-command-substitution apostrophe hazard hit while writing these tests.Risk Assessment
✅ Low: The functional change is ~30 lines confined to one Pi extension, its central invariant (both latch-clearing edges, set-before-send ordering, urgency backstop) checks out against the installed Pi 0.84.4 runtime, the nine new regressions are non-vacuous under hand-traced counterfactuals, and the only open items are a test tidiness nit and one intent divergence the captain already adjudicated with the runtime evidence in hand.
Testing
I ran the extension suite (
tests/fm-pi-watch-extension.test.sh, 46 ok, exit 0) plus the three other suites that load the same tracked extension (primary-types, which skips with notscpresent, branch-extension 31, calm-extension 9, all exit 0), then went past pass/fail for product-level proof. Two demo drivers exercise the real extension through the real ExtensionAPI boundary with a void-returningsendUserMessagefake matching Pi 0.84.4, and their transcripts show the actual dock the captain would see: 20 rows on upstream 355f46f versus 1 row on this change for the same 20-close burst, with all 20 durable wake-queue records intact in both, and a second transcript where restoration exhaustion still surfaces its ownwatcher: FAILEDrow while the rejected call-site-only-urgency shape swallows it. Becausetests/lib.shexits on first failure, I built a per-test runner and ran each of the 9 new coalescing tests separately against 7 deliberately broken extension variants; every variant fails exactly the set the change's verification record claims, so none of the new tests is vacuous. I also confirmed the CONTRIBUTING hazard empirically on this machine's bash 3.2 and read the installed Pi 0.84.4 runtime to check the void-send claim the design rests on. No screenshots apply: the affected surface is a text follow-up dock inside a Pi session, so the captured transcripts are the rendered end-user output.Evidence: Pi follow-up dock before/after: 20-close burst during one captain turn
Source: Pi follow-up dock before/after: 20-close burst during one captain turn
--- BEFORE: upstream main (355f46f) --- actionable watcher closes during the captain's turn : 20 durable wake-queue records (bin/fm-wake-drain.sh) : 20 Pi follow-up dock rows presented to the captain : 20 --- AFTER: this change (5a4babd) --- actionable watcher closes during the captain's turn : 20 durable wake-queue records (bin/fm-wake-drain.sh) : 20 Pi follow-up dock rows presented to the captain : 1 Pi follow-up dock as the captain would see it: [row 1] FIRSTMATE_OP: v1 watcher: FIRSTMATE WATCHER WAKE: signal: secondmate alpha finished, needs review Full text of dock row 1 (what the model actually receives): | FIRSTMATE_OP: v1 watcher: FIRSTMATE WATCHER WAKE: signal: secondmate alpha finished, needs review | | Run bin/fm-wake-drain.sh first and handle the queued wake. Watcher continuity is extension-owned.Evidence: Supervision failure still presents its own row past a pending ordinary row
Source: Supervision failure still presents its own row past a pending ordinary row
--- AFTER: this change (5a4babd) --- Pi follow-up dock rows presented to the captain: 2 [row 1] | FIRSTMATE WATCHER WAKE: signal: secondmate alpha finished, needs review [row 2] | FIRSTMATE WATCHER WAKE: signal: secondmate bravo finished, needs review | watcher: FAILED - Pi extension could not verify a ready successor watcher | watcher: FAILED - Pi extension could not restore watcher continuity after 1 retries --- COUNTEREXAMPLE: call-site-only urgency (rejected candidate B shape) --- Pi follow-up dock rows presented to the captain: 1 *** no supervision-failure row reached the captain: the failure was coalesced away ***Evidence: Non-vacuity matrix: 9 coalescing tests against 7 broken extension variants
Source: Non-vacuity matrix: 9 coalescing tests against 7 broken extension variants
v1-upstream -> FAIL burst only v2-latch-after-await -> FAIL idle-consumed re-arm only v3-callsite-only-urgency -> FAIL restoration exhaustion + restoration-time lock loss v4-shared-latch -> FAIL session replacement only v5-no-urgency -> FAIL all four failure-bypass tests v6-no-agent-start-clear -> FAIL idle-consumed re-arm only v7-no-agent-settled-clear -> FAIL busy-path re-arm + discarded-row re-armEvidence: Variant definitions and what each proves
Source: Variant definitions and what each proves
Evidence: tests/fm-pi-watch-extension.test.sh full run (46 ok, exit 0)
Source: tests/fm-pi-watch-extension.test.sh full run (46 ok, exit 0)
Evidence: Other suites loading the same extension (types/branch/calm)
Source: Other suites loading the same extension (types/branch/calm)
Evidence: Evidence README with reproduction steps
Source: Evidence README with reproduction steps
Evidence: Demo drivers that produced the two transcripts
Source: Demo drivers that produced the two transcripts
Pipeline
Updates from git push no-mistakes
... (4 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)
🔧 Fix: drop unreachable send rollback, align fakes to void API
2 issues (1 warning, 1 info) still open:
.pi/extensions/fm-primary-pi-watch.ts:634- The inline comment above theagent_starthandler still carries the superseded ordering claim that the fix rounds were required to remove everywhere. It reads "The idle-path consumption edge: an idle send starts its run inside sendUserMessage, so the row is consumed at run start", which directly contradicts the corrected header 21 lines above: "the await resumes on the very next microtask while the run it triggers starts from the runtime's own asynchronous chain and emits agent_start later" (lines 21-26), and the verification record's "every runtime effect happens afterwards, off that call stack" (docs/verification/supervision.md:477). The installed Pi 0.84.4 confirms the header, not the comment:ExtensionAPI.sendUserMessageis void (dist/core/extensions/types.d.ts:980, discarded at loader.js:300), and evenAgentSession.promptawaits the extensioninputemit, the auth check, and_checkCompaction(agent-session.js:842-812) before reaching_runAgentPrompt, so no run starts on the extension's call stack. Commit e67eed9 rewrote the header but left this comment, which commit 684d80f had introduced. Behavior is unaffected - agent_start remains the correct idle-path edge - but the file is the designated single owner of this contract and now states both orderings, and the captain's round-1 instruction 3 required correcting every place stating the old ordering. Fix: restate it as "an idle send makes the runtime start a run that takes the queued row and emits agent_start, re-arming presentation at run start rather than at run end".tests/fm-pi-watch-extension.test.sh:1598- The busy-path and cleared-dock drivers are near-identical copies and each carries the other's helper unused:clearQueuedRowsis defined at line 1598 but never called intest_pi_busy_consumed_row_rearms_for_genuinely_later_wake(which only callsdrainQueuedInlineat 1647), anddrainQueuedInlineis defined at line 1729 but never called intest_pi_cleared_dock_rearms_for_genuinely_later_wake(which only callsclearQueuedRowsat 1784). Nothing lints inside these node heredocs, so the dead halves will not be caught later. Dropping the unused helper from each driver removes the misleading suggestion that both transitions are exercised in both tests. The tests themselves are sound and non-vacuous: I traced that each genuinely fails under anagent_start-only clear (row docks, inline drain or discard emits no agent_start, settle is the only remaining edge), which is exactly the defect they guard.🔧 Fix: correct stale ordering comment, drop dead test helpers
2 issues (1 warning, 1 info) still open:
docs/watcher-continuity.md:38- The operator-facing guarantee states an absolute that the extension header contradicts in the same change. Line 38 reads "Coalescing never costs a wake: the run that consumes or discards the pending row re-arms presentation, so a genuinely later wake still presents one new row." But.pi/extensions/fm-primary-pi-watch.ts:39-41— the file this doc names as the contract owner — states "a wake arriving after an inline drain but before settle is suppressed for the remainder of that one run." That window is reachable and I traced it: a captain run is in flight, an ordinary close docks a row and sets the latch, Pi'srunLoopdrains that row inline (pi-agent-core/dist/agent-loop.js:161-165,pendingMessages = followUpMessages; continue;) with no secondagent_start, and a genuinely later actionable close during the remainder of that long run hits the early return at.pi/extensions/fm-primary-pi-watch.ts:311and presents no row of its own. Only atagent_settleddoes presentation re-arm. So coalescing does cost that wake its own row; the durable queue still holds the event and the bound is one run, which is exactly what the header says and what the doc should say. The neighbouring sentence at line 41, "nothing is acknowledged, truncated, or delayed by the latch", reads as an absolute too, while presentation delay is the deliberate mechanism (durable records genuinely are untouched — I confirmedfm_recovery_marker_begin_handlingonly movesdowntime->handlingand leaves thepending/announcedprefix, so no recovery announcement is consumed by a coalesced wake). Fix: keep the operator guarantee but scope it to the owned contract, e.g. that the run consuming or discarding the row re-arms presentation and a wake arriving after that still presents, with a wake arriving inside the drain-to-settle window of one run held only until that run settles..pi/extensions/fm-primary-pi-watch.ts:317- Recorded for the final acceptance decision only, not a defect and no action recommended. The frozen --intent text marks as required: "The ordinary latch rolls back if sendUserMessage rejects before a row is established, then the error propagates so the existing catch reports it as a failure", plus "Do not 'simplify' the set-before-await ordering or the try/catch around the send", a "production-timing fake sendUserMessage where an idle send fires agent_start inside the call and resolves only after the driver-controlled run completes", and "delivery-rejection rollback coverage". None of those are present now:sendWakecallsawait pi.sendUserMessage(...)bare with no try/catch,test_pi_rejected_delivery_rolls_back_ordinary_latchis gone, and every fake models avoidfire-and-forget send. That divergence is the captain's own round-2 instruction, and the evidence behind it checks out against the installed Pi 0.84.4:ExtensionAPI.sendUserMessageis declaredvoid(dist/core/extensions/types.d.ts:980),dist/core/extensions/loader.js:299-302discards the runtime promise, anddist/core/agent-session.js:2013routes every rejection torunner.emitError, so the rollback branch and its test could never run against real Pi. The set-before-send ordering the intent calls decisive is preserved; only its stated justification changed. The residual is documented at lines 51-57 and the record's superseded-claims note at docs/verification/supervision.md:480 is accurate. I am not re-raising this as a blocker since the user already decided it with the evidence in hand; it is listed so the intent-versus-shipped divergence is visible rather than silent.🔧 Fix: qualify coalescing guarantee to the bounded within-run window
2 infos still open:
tests/fm-pi-watch-extension.test.sh:1326- Residual dead halves of the shared fake, the same tidiness class round 3 cleaned from the busy-path and cleared-dock drivers. In the burst driver,runningis set true bystartCaptainRun()before any send and is only cleared by the finalsettleRun()immediately beforeprocess.exit(0), so theif (!running && !runScheduled)branch insendUserMessagenever fires:startRunis unreachable andconsumedis written by nothing and read by nothing. The assertionagentStarts !== 1in that driver is therefore guaranteed by the fake's own bookkeeping rather than by extension behavior. Separately, the idle re-arm driver at line 1452 importsexistsSyncandreadFileSyncand uses neither. Nothing lints inside these node heredocs, so these will not be caught later. Dropping the unused import pair and either removingconsumedfrom the burst driver or leavingstartRunwith a comment that it models the idle branch this driver never enters would keep each driver honest about what it exercises. No guarantee depends on any of this, and every coalescing assertion I traced remains non-vacuous..pi/extensions/fm-primary-pi-watch.ts:317- Recorded for the acceptance decision only; already adjudicated in round 4 and no action recommended. The frozen --intent marks as required "The ordinary latch rolls back if sendUserMessage rejects before a row is established, then the error propagates so the existing catch reports it as a failure", plus "Do not 'simplify' the set-before-await ordering or the try/catch around the send", a fake where "an idle send fires agent_start inside the call and resolves only after the driver-controlled run completes", and "delivery-rejection rollback coverage". None are present: line 317 is a bareawait pi.sendUserMessage(...)with no try/catch,test_pi_rejected_delivery_rolls_back_ordinary_latchis gone, and every fake models a void fire-and-forget send. That divergence is the captain's own round-2 instruction and the evidence holds against the installed Pi 0.84.4 (ExtensionAPI.sendUserMessagedeclared void atdist/core/extensions/types.d.ts:980, discarded atdist/core/extensions/loader.js:300, rejections routed torunner.emitErroratdist/core/agent-session.js:2013), so the rollback branch and its test could never execute against real Pi. The set-before-send ordering the intent calls decisive is preserved; only its stated justification changed, and the residual is documented at the extension header anddocs/verification/supervision.md:480. Listed so the intent-versus-shipped divergence stays visible rather than silent..pi/extensions/fm-primary-pi-watch.ts:317- The intent lists as required that "the ordinary latch rolls back if sendUserMessage rejects before a row is established, then the error propagates", with a rejected-delivery regression as required evidence. Neither exists in the shipped change: review commit e67eed9 removed the rollback and its test, and no rejected-delivery test is in the suite. I verified independently why, in the installed Pi 0.84.4 runtime: ExtensionAPI.sendUserMessage is declared void at dist/core/extensions/types.d.ts:980, dist/core/extensions/loader.js:302 calls the runtime method without returning its promise, and dist/core/agent-session.js:2013 catches every rejection into runner.emitError. The extension can never observe a failed send, so the required behavior is not implementable on this API surface and no test can demonstrate it. The extension header and docs/verification/supervision.md both record this as a residual. The code looks correct to me and the intent text looks stale on this one point, but confirming that supersession is the captain's call, not mine.bash tests/fm-pi-watch-extension.test.sh(46 ok, exit 0, includes the 9 new coalescing regressions)bash tests/fm-pi-branch-extension.test.sh(31 ok, exit 0)bash tests/fm-calm-pi-extension.test.sh(9 ok, exit 0)bash tests/fm-pi-primary-types.test.sh(exit 0, reportsskip: tsc not found for Pi extension typecheck, matching the verification record)Manual end-user demo: 20-close burst through the real extension and ExtensionAPI boundary, upstream 355f46f vs HEAD, capturing the rendered follow-up dock and the durable wake queue (pi-dock-burst-demo.sh)Manual end-user demo: pending ordinary row plus continuity-restoration exhaustion, HEAD vs call-site-only-urgency variant (pi-dock-failure-visibility-demo.sh)Non-vacuity matrix: each of the 9 coalescing tests run individually (subshell-isolated runner,EXT_OVERRIDE) against 7 broken variants (upstream, latch-after-await, call-site-only urgency, shared latch, no urgency, no agent_start clear, no agent_settled clear)/bin/bash -nreproduction of the bash 3.2 heredoc-apostrophe hazard recorded in CONTRIBUTING.md, plus/bin/bash -n tests/fm-pi-watch-extension.test.shRead the installed@earendil-works/pi-coding-agent0.84.4dist/core/extensions/types.d.ts,extensions/loader.js, andagent-session.jsto confirmExtensionAPI.sendUserMessagereturns void and rejections go torunner.emitError✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.