fix(bin): expire stale launch reservations - #76
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
CEO overview
What changed technically
Defect B is fixed on top of
origin/main, which already contains PR #75's Defect A exclusion of held, parked, blocked, and other non-counting backlog lanes.The reservation branch now counts an
in_flightworker with no endpoint only when its per-task launch-reservation timestamp is no older than 300 seconds and is not in the future.fm-pull.shrecords the timestamp immediately aftertasks-axi startand before releasing the task-set lock, preserving the concurrent-launch race protection.fm-spawn.shrecords the timestamp for direct fresh spawns while holding the task-set lock and removes it after endpoint metadata publication.Teardown removes any remaining reservation artifact.
The implementation does not use the backlog
sincedate because that date describes task creation rather than the current launch attempt.Regression coverage proves that a fresh no-endpoint reservation counts, a stale no-endpoint
in_flightrecord older than the window does not count, endpoint-backed workers remain counted, and PR #75's held or blocked exclusions remain intact.The heavy self-hosted CI lanes remain disabled by design and must not be enabled.
Validation
Module-boundary decision
Current module retained: reservation freshness remains owned by the fleet snapshot, pull, spawn, and teardown scripts that own attention accounting and launch publication.
Decision needed
No decision required.
What Changed
Risk Assessment
✅ Low: Captain, the bounded reservation logic preserves the 300-second cutoff, exclusion lanes, direct-launch coverage, and four-worker accounting.
Testing
Focused pull and teardown suites passed, and live isolated CLI runs produced persisted evidence for the 299/300-second boundary, exclusions, pull timestamp creation, and teardown cleanup; no lint, full suite, pipeline, push, or CI phase was run.
tests/fm-pull.test.shtests/fm-pull.test.shNO_MISTAKES_GATE, so the realfm-spawncommand refuses before launch. Bypassing that guard or allowing its external task-worktree allocation would exceed this…Evidence: Live reservation snapshot
Source: Live reservation snapshot
Evidence: Live pull timestamp record
Source: Live pull timestamp record
Evidence: Live teardown marker cleanup
Source: Live teardown marker cleanup
teardown_completed=true reservation_marker=absentPipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 3 issues found → auto-fixed (3) ✅
bin/fm-fleet-snapshot.sh:783- A fresh no-endpoint reservation that is later blocked or held still counts:tasks-axi block <id> --by <other>accepts an existing in-flight task, while the changed reservation predicates check only state/current_role. This contradicts the required “held or blocked exclusions remain intact.” Exclude held and unresolved-blocker records at the reservation-selection boundary and cover this path.tests/fm-pull.test.sh:217- The required deterministic “300-second boundary” is not tested. The fixture uses 1900 and 1000 against epoch 2000, so it proves neither inclusion at 1700 nor exclusion at 1699. Add observable snapshot assertions for both boundary values.bin/fm-spawn.sh:1024- The new reservation write also runs for--secondmate; its successful remote path returns after endpoint metadata publication and never reaches the cleanup at line 2893. This exceeds worker-slot accounting scope and contradicts “removes it after endpoint metadata publication.” Limit the marker to ordinary workers or remove it in the remote publication path.🔧 Fix applied.
2 warnings still open:
bin/fm-pull.sh:51- The newFM_LAUNCH_RESERVATION_NOW_EPOCHoverride is an unrequired launch path that can write a stale timestamp. For example, a pull launched with it set to0records an immediately expired reservation; if the process is then killed after it creates an endpoint but before metadata publication, the next snapshot excludes that live no-metadata worker and permits another launch. This contradicts the required criterion that a genuinely mid-spawn worker reserves its slot. Remove this test hook from both timestamp writers so they always record the current epoch.bin/fm-fleet-snapshot.sh:145-FM_ATTENTION_RESERVATION_WINDOW_SECSadds a runtime-configurable window not required by the intent. Setting it below 300 expires a genuine reservation early; setting it above 300 retains a stale one past the required fixed 300-second bound. This contradicts the stated “bounded 300-second freshness window”; remove the override and retain a fixed 300-second constant.🔧 Fix applied.
1 error still open:
bin/fm-fleet-snapshot.sh:785- Intent requires thatfm-spawn.shreserve direct fresh spawns so a genuinely mid-spawn worker keeps its slot, but the new scanner only admits marker IDs already present as unheld, unblockedIn flightbacklog workers. A documented directfm-spawn.shship/scout launch can create its backend endpoint and be interrupted before metadata publication without ever creating a backlog row; its marker is therefore ignored and a subsequent launch sees the slot as free. Make marker-backed ordinary direct launches participate in reservation selection while retaining the held/blocked exclusions.🔧 Fix applied.
✅ Re-checked - no issues remain.
tests/fm-pull.test.shtests/fm-pull.test.shNO_MISTAKES_GATE, so the realfm-spawncommand refuses before launch. Bypassing that guard or allowing its external task-worktree allocation would exceed this…bin/fm-session-start.shtests/fm-pull.test.shtests/fm-teardown.test.shFM_HOME=<isolated-home> FM_SNAPSHOT_NOW_EPOCH=2000 bin/fm-fleet-snapshot.sh --local-jsonFM_HOME=<isolated-home> bin/fm-pull.sh start pull-writer <missing-project> --mode local-only --yolo off --harness piFM_HOME=<isolated-home> bin/fm-teardown.sh task-cleanup✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.