Skip to content

fix(bin): expire stale launch reservations - #76

Merged
bingb0t5 merged 5 commits into
mainfrom
fm/fm-attention-stale-reservation-r1
Sep 15, 2026
Merged

bingb0t5 merged 5 commits into
mainfrom
fm/fm-attention-stale-reservation-r1

Conversation

@bingb0t5

Copy link
Copy Markdown
Owner

Intent

CEO overview

  • What is changing: Worker-slot accounting now expires backlog-only launch reservations after a bounded 300-second freshness window.
  • Why it matters: A finished or stale task left in flight without an endpoint will no longer consume one of the four worker slots forever.
  • Customer or business impact: Firstmate can dispatch queued work after an abandoned launch without requiring a manual backlog reopen, while a genuinely mid-spawn worker still reserves its slot.
  • Risk and rollout: The change is limited to the reservation branch, uses a per-task launch-reservation epoch record written when pull or spawn begins, preserves the existing four-worker cap, and rolls out through the required no-mistakes PR and CI checks.

What changed technically

Defect B is fixed on top of origin/main, which already contains PR #75's Defect A exclusion of held, parked, blocked, and other non-counting backlog lanes.

The reservation branch now counts an in_flight worker with no endpoint only when its per-task launch-reservation timestamp is no older than 300 seconds and is not in the future.

fm-pull.sh records the timestamp immediately after tasks-axi start and before releasing the task-set lock, preserving the concurrent-launch race protection.

fm-spawn.sh records the timestamp for direct fresh spawns while holding the task-set lock and removes it after endpoint metadata publication.

Teardown removes any remaining reservation artifact.

The implementation does not use the backlog since date because that date describes task creation rather than the current launch attempt.

Regression coverage proves that a fresh no-endpoint reservation counts, a stale no-endpoint in_flight record older than the window does not count, endpoint-backed workers remain counted, and PR #75's held or blocked exclusions remain intact.

The heavy self-hosted CI lanes remain disabled by design and must not be enabled.

Validation

  • Checks passed: Focused pull, fleet snapshot, and teardown test suites passed, fast lint passed on the changed shell and test files, and the diff check passed.
  • Checks not run: The no-mistakes pipeline and hosted CI have not run yet.
  • Evidence and limitations: The focused regression uses a deterministic snapshot epoch and verifies the 300-second boundary; hosted CI will provide the full required suite and platform coverage.

Module-boundary decision

Current module retained: reservation freshness remains owned by the fleet snapshot, pull, spawn, and teardown scripts that own attention accounting and launch publication.

Decision needed

No decision required.

What Changed

  • Record per-task launch-reservation epochs during pull and fresh direct spawns, then clear them after endpoint metadata publication or teardown.
  • Count no-endpoint ship and scout reservations only within the 300-second freshness window, while preserving held and blocked exclusions.
  • Add regression coverage for fresh, stale, direct, and excluded launch reservations.

Risk Assessment

✅ Low: Captain, the bounded reservation logic preserves the 300-second cutoff, exclusion lanes, direct-launch coverage, and four-worker accounting.

Testing

Focused pull and teardown suites passed, and live isolated CLI runs produced persisted evidence for the 299/300-second boundary, exclusions, pull timestamp creation, and teardown cleanup; no lint, full suite, pipeline, push, or CI phase was run.

  • Live validation: ⚠️ inconclusive - 4 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Fresh, stale, and direct reservations are correctly counted ✅ pass live Live reservation snapshot artifact and tests/fm-pull.test.sh
Held and blocked launch reservations do not consume slots ✅ pass live Live reservation snapshot artifact and tests/fm-pull.test.sh
Pull records a current reservation before failed endpoint publication ✅ pass live Live pull timestamp record
Teardown removes a remaining reservation marker ✅ pass live Live teardown marker cleanup artifact
Direct ship/scout spawn scope and endpoint-publication cleanup ⏸️ untested no The assigned gate environment exports NO_MISTAKES_GATE, so the real fm-spawn command refuses before launch. Bypassing that guard or allowing its external task-worktree allocation would exceed this…
Evidence: Live reservation snapshot

Source: Live reservation snapshot

{
  "backlog_inputs": [
    {
      "id": "row-plus-marker",
      "state": "in_flight",
      "current_role": "worker",
      "hold_kind": null,
      "hold_reason": null,
      "unresolved_blocker_ids": []
    },
    {
      "id": "held-reservation",
      "state": "in_flight",
      "current_role": "held",
      "hold_kind": "external",
      "hold_reason": "waiting externally",
      "unresolved_blocker_ids": []
    },
    {
      "id": "blocked-reservation",
      "state": "in_flight",
      "current_role": "worker",
      "hold_kind": null,
      "hold_reason": null,
      "unresolved_blocker_ids": [
        "blocker"
      ]
    },
    {
      "id": "stale-backlog",
      "state": "in_flight",
      "current_role": "worker",
      "hold_kind": null,
      "hold_reason": null,
      "unresolved_blocker_ids": []
    },
    {
      "id": "fresh-backlog",
      "state": "in_flight",
      "current_role": "worker",
      "hold_kind": null,
      "hold_reason": null,
      "unresolved_blocker_ids": []
    }
  ],
  "attention": {
    "limit": 4,
    "count": 3,
    "remaining": 1,
    "valid": true,
    "reservations": [
      {
        "id": "fresh-backlog",
        "reservation_at_epoch": 1701
      },
      {
        "id": "fresh-direct",
        "reservation_at_epoch": 1701
      },
      {
        "id": "row-plus-marker",
        "reservation_at_epoch": 1701
      }
    ]
  }
}
Evidence: Live pull timestamp record

Source: Live pull timestamp record

{
  "marker_epoch": 1789482456,
  "writer_window": {
    "before": 1789482456,
    "after": 1789482456
  },
  "snapshot": {
    "attention_count": 1,
    "reservations": [
      {
        "id": "pull-writer",
        "reservation_at_epoch": 1789482456
      }
    ]
  },
  "spawn_result": "ok: start pull-writer -> In flight\nhelp[1]:\n  - Run `tasks-axi done pull-writer --pr <url>` when it ships\nerror: no-mistakes gate agent must not drive the fleet (NO_MISTAKES_GATE set)\nerror: spawn for pull-writer failed; its In flight row remains as a visible unknown reservation\n"
}
Evidence: Live teardown marker cleanup

Source: Live teardown marker cleanup

teardown_completed=true reservation_marker=absent

teardown_completed=true
reservation_marker=absent
- Outcome: ⚠️ 1 warning across 1 run (7m49s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 3 issues found → auto-fixed (3) ✅
  • 🚨 bin/fm-fleet-snapshot.sh:783 - A fresh no-endpoint reservation that is later blocked or held still counts: tasks-axi block &lt;id&gt; --by &lt;other&gt; accepts an existing in-flight task, while the changed reservation predicates check only state/current_role. This contradicts the required “held or blocked exclusions remain intact.” Exclude held and unresolved-blocker records at the reservation-selection boundary and cover this path.
  • 🚨 tests/fm-pull.test.sh:217 - The required deterministic “300-second boundary” is not tested. The fixture uses 1900 and 1000 against epoch 2000, so it proves neither inclusion at 1700 nor exclusion at 1699. Add observable snapshot assertions for both boundary values.
  • ⚠️ bin/fm-spawn.sh:1024 - The new reservation write also runs for --secondmate; its successful remote path returns after endpoint metadata publication and never reaches the cleanup at line 2893. This exceeds worker-slot accounting scope and contradicts “removes it after endpoint metadata publication.” Limit the marker to ordinary workers or remove it in the remote publication path.

🔧 Fix applied.
2 warnings still open:

  • ⚠️ bin/fm-pull.sh:51 - The new FM_LAUNCH_RESERVATION_NOW_EPOCH override is an unrequired launch path that can write a stale timestamp. For example, a pull launched with it set to 0 records an immediately expired reservation; if the process is then killed after it creates an endpoint but before metadata publication, the next snapshot excludes that live no-metadata worker and permits another launch. This contradicts the required criterion that a genuinely mid-spawn worker reserves its slot. Remove this test hook from both timestamp writers so they always record the current epoch.
  • ⚠️ bin/fm-fleet-snapshot.sh:145 - FM_ATTENTION_RESERVATION_WINDOW_SECS adds a runtime-configurable window not required by the intent. Setting it below 300 expires a genuine reservation early; setting it above 300 retains a stale one past the required fixed 300-second bound. This contradicts the stated “bounded 300-second freshness window”; remove the override and retain a fixed 300-second constant.

🔧 Fix applied.
1 error still open:

  • 🚨 bin/fm-fleet-snapshot.sh:785 - Intent requires that fm-spawn.sh reserve direct fresh spawns so a genuinely mid-spawn worker keeps its slot, but the new scanner only admits marker IDs already present as unheld, unblocked In flight backlog workers. A documented direct fm-spawn.sh ship/scout launch can create its backend endpoint and be interrupted before metadata publication without ever creating a backlog row; its marker is therefore ignored and a subsequent launch sees the slot as free. Make marker-backed ordinary direct launches participate in reservation selection while retaining the held/blocked exclusions.

🔧 Fix applied.
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 warning
  • ⚠️ live validation verdict: inconclusive (4 of 5 scenarios were driven live against the product); untested: Direct ship/scout spawn scope and endpoint-publication cleanup
  • Live validation: ⚠️ inconclusive - 4 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Fresh, stale, and direct reservations are correctly counted ✅ pass live Live reservation snapshot artifact and tests/fm-pull.test.sh
Held and blocked launch reservations do not consume slots ✅ pass live Live reservation snapshot artifact and tests/fm-pull.test.sh
Pull records a current reservation before failed endpoint publication ✅ pass live Live pull timestamp record
Teardown removes a remaining reservation marker ✅ pass live Live teardown marker cleanup artifact
Direct ship/scout spawn scope and endpoint-publication cleanup ⏸️ untested no The assigned gate environment exports NO_MISTAKES_GATE, so the real fm-spawn command refuses before launch. Bypassing that guard or allowing its external task-worktree allocation would exceed this…
  • bin/fm-session-start.sh
  • tests/fm-pull.test.sh
  • tests/fm-teardown.test.sh
  • FM_HOME=<isolated-home> FM_SNAPSHOT_NOW_EPOCH=2000 bin/fm-fleet-snapshot.sh --local-json
  • FM_HOME=<isolated-home> bin/fm-pull.sh start pull-writer <missing-project> --mode local-only --yolo off --harness pi
  • FM_HOME=<isolated-home> bin/fm-teardown.sh task-cleanup
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@bingb0t5
bingb0t5 merged commit 5d726e2 into main Sep 15, 2026
13 checks passed
@bingb0t5
bingb0t5 deleted the fm/fm-attention-stale-reservation-r1 branch September 15, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant