fix: accelerate local Bearings snapshot composition

[kunchenguid]

Intent

Cut the cost of composing the LOCAL Bearings fleet snapshot so that on a large home a same-home run is fast in both the cold and the warm case, targeting under 5 seconds. Before this change local composition took roughly 13-16 seconds on a large home. This is specifically the LOCAL composition path and is a separate concern from the remote ledger collector, which already returns in about 2 seconds and is not what this work is about.

The hard acceptance criterion is that the fm-bearings.v1 projection output must stay IDENTICAL apart from timestamps. This is a pure cost fix: no field may change value, appear, or disappear, and no row may be reordered or dropped. Speed must not be bought with any behavior or output change.

Accepted scope decision, deliberately narrow. Only the real hot paths are addressed:

1. The per-task local observations. For each task the snapshot needs a current-state read and an endpoint-existence probe, and for secondmates an agent-liveness read. These are independent observations, and the previous code performed them strictly in series across every task, so a home with N local workers paid N slow reads back to back. They are now prefetched per task, with each task's current-state read and endpoint probe started together, across a bounded worker pool. The pool size is the new FM_SNAPSHOT_LOCAL_READ_CONCURRENCY knob, default 8, validated by the same positive-bound validator as the other snapshot knobs.
2. fm_meta_get in bin/fm-backend.sh, which is called many times per task and per snapshot and previously spawned a grep | tail | cut subprocess chain on every single call. It is now a pure-shell read loop. Its existing semantics are preserved exactly: last value wins for a repeated key, and a file with no trailing newline still yields its final value.

Broader ideas that were considered and deliberately NOT pursued in this change: rewriting the large-backlog parse, changing the scout-report scan, and a general reduction of the many small jq invocations. Those were dropped in favour of this narrower, lower-risk fix against the paths that actually dominate the measured cost. Reviewers should not treat their absence as an oversight.

Constraints held throughout:

• bin/fm-timeout-lib.sh remains the single owner of bounded execution. This change introduces no new timeout, deadline, or kill mechanism; every command bound, including the existing FM_SNAPSHOT_CREW_STATE_TIMEOUT per-task bound, is still owned there. The concurrency added here only decides how many bounded reads may be in flight at once.
• Everything stays bash 3.2 safe, since that is the system bash on macOS.
• Temporary per-task observation files are created under a mktemp -d directory with umask 077, and are cleaned up both on the normal path and through the EXIT trap, which now chains the task cleanup alongside the pre-existing collection cleanup.
• Tests are behavioral: they drive the real command and assert observable output and timing, never implementation source text.

Test coverage added:

• A behavioral regression that builds a synthetic large home (five local workers plus a 300-entry Done backlog), runs the real snapshot twice, and asserts three things: composition stays under the five second target, the serial and concurrent projections are byte-identical so concurrency introduces no drift, and every worker row survives.
• An extension of the existing fm_meta_get test covering the last-value-wins and missing-final-newline edge cases that the pure-shell rewrite has to preserve.

Context for this particular run: the change was already implemented, committed, and opened as a pull request, but that pull request then fell behind and conflicted with the default branch, and its earlier validation run was aborted for a tooling upgrade. The branch has now been rebased onto the current default branch. The one conflict was in bin/fm-fleet-snapshot.sh, where the default branch had meanwhile removed the remote_home_present variable entirely as part of removing legacy remote snapshot reads. The resolution keeps the default branch's removal of that variable while preserving the concurrent prefetch, and carries the default branch's updated rationale comment about not probing remote endpoints into the prefetch function, which is where that decision now lives. A full fresh validation is wanted on the rebased head.

End-to-end verification already performed on a copy of a real large home (8 local workers, 887-line backlog, real worktrees and real backends), measured with the pre-change and post-change code interleaved so that host load affects both equally: the snapshot went from roughly 18-26 seconds to roughly 6-10 seconds, and the bearings projection from roughly 14-20 seconds to roughly 6-9 seconds, a consistent 2.5-3x reduction. Absolute numbers were inflated because the host was heavily oversubscribed at the time (load average 16-26 on 14 CPUs) by unrelated concurrent work; the eight per-task reads alone accounted for 4.6-6.0 seconds of that on their own, and a single external per-task status call cost 1.0-1.4 seconds under the same load, so the snapshot's own composition is now a small remainder on top of that external term. Output identity was confirmed on the same real home: the fm-bearings.v1 projection compared 218 leaves with 217 byte-identical and the single difference being the generated timestamp, and the underlying fm-fleet-snapshot.v1 projection compared 5153 leaves with 5135 byte-identical, the differences being only observed_at timestamps.

Do not merge the pull request; merge authority is not granted for this work.

Two corrections were made on top of the recovered pull-request head before this run. First, the large-local-snapshot regression originally asserted that a whole snapshot composed in under five seconds; that bound measures host load rather than whether the per-task reads actually overlap, and it failed roughly one run in six on a contended machine, landing exactly on the five second boundary. It now times a serialized run and a concurrent run of the same workload and requires the concurrent one to save at least two seconds, so the two runs' shared composition overhead cancels and the assertion still fails loudly when the concurrency regresses, which was confirmed by forcing the concurrent run back to serial. Second, the pinned Bearings test count in CI moved from 47 to 48 because rebasing onto the current default branch picked up its captain-hold test.

What Changed

• Prefetch local task state, endpoint, and secondmate-liveness observations through a bounded, configurable worker pool while keeping captured observations generation-coherent.
• Replace fm_meta_get subprocess chains with a semantics-preserving shell loop and stream large snapshot JSON values through stdin.
• Add behavioral coverage for concurrent-read projection identity, metadata and status races, generation changes, teardown, and metadata edge cases.

Risk Assessment

✅ Low: The concurrent observation path is bounded, generation-aware, Bash 3.2-compatible, and preserves the documented snapshot projection semantics without introducing a substantiated reachable defect.

Testing

The supplied changed-test baseline and focused backend/Bearings suites passed; end-to-end macOS Bash 3.2 evidence showed the real Bearings CLI reducing the delayed local snapshot from 7s serial to 2s concurrent while preserving a byte-identical fm-bearings.v1 projection and all five worker rows.

Evidence: Bash 3.2 large-local snapshot timing and projection identity evidence

Source: Bash 3.2 large-local snapshot timing and projection identity evidence

{
  "scenario": "real fm-bearings-snapshot CLI against synthetic large local home",
  "serial_seconds": 7,
  "concurrent_seconds": 2,
  "seconds_saved": 5,
  "serial_projection_sha256": "e4ce870bdc0e78b22fcd03da2e143d6a122f5354839ddfaeb6f4850b8d74fd78",
  "concurrent_projection_sha256": "e4ce870bdc0e78b22fcd03da2e143d6a122f5354839ddfaeb6f4850b8d74fd78",
  "projections_byte_identical": true,
  "worker_rows": 5,
  "expected_worker_rows": 5,
  "done_backlog_rows": 300
}
ok - large local snapshot overlaps local reads with byte-identical serial and concurrent projections

Evidence: Generated fm-bearings.v1 projection containing all five workers

Source: Generated fm-bearings.v1 projection containing all five workers

{
  "schema": "fm-bearings.v1",
  "home": "fm-bearings.8Mti7y/large-local-snapshot",
  "generated": "2026-07-11T18:00:00Z",
  "prs": "not_requested (run: /bearings include PRs)",
  "in_flight": [
    {
      "id": "local-1",
      "kind": "ship",
      "state": "unknown",
      "repo": "firstmate",
      "doing": "harness state unavailable (unknown missing)"
    },
    {
      "id": "local-2",
      "kind": "ship",
      "state": "unknown",
      "repo": "firstmate",
      "doing": "harness state unavailable (unknown missing)"
    },
    {
      "id": "local-3",
      "kind": "ship",
      "state": "unknown",
      "repo": "firstmate",
      "doing": "harness state unavailable (unknown missing)"
    },
    {
      "id": "local-4",
      "kind": "ship",
      "state": "unknown",
      "repo": "firstmate",
      "doing": "harness state unavailable (unknown missing)"
    },
    {
      "id": "local-5",
      "kind": "ship",
      "state": "unknown",
      "repo": "firstmate",
      "doing": "harness state unavailable (unknown missing)"
    }
  ],
  "secondmates": [],
  "secondmate_reconcile": [],
  "decisions_open": [],
  "landed": [
    {
      "id": "history-99",
      "what": "Historical completed item 99",
      "artifact": "https://github.com/acme/firstmate/pull/99",
      "owner": "(main)"
    },
    {
      "id": "history-98",
      "what": "Historical completed item 98",
      "artifact": "https://github.com/acme/firstmate/pull/98",
      "owner": "(main)"
    },
    {
      "id": "history-97",
      "what": "Historical completed item 97",
      "artifact": "https://github.com/acme/firstmate/pull/97",
      "owner": "(main)"
    },
    {
      "id": "history-96",
      "what": "Historical completed item 96",
      "artifact": "https://github.com/acme/firstmate/pull/96",
      "owner": "(main)"
    },
    {
      "id": "history-95",
      "what": "Historical completed item 95",
      "artifact": "https://github.com/acme/firstmate/pull/95",
      "owner": "(main)"
    },
    {
      "id": "history-94",
      "what": "Historical completed item 94",
      "artifact": "https://github.com/acme/firstmate/pull/94",
      "owner": "(main)"
    }
  ],
  "gates": [],
  "reports": [],
  "recorded_prs": [],
  "omitted": [
    {
      "surface": "backlog item bodies",
      "reveal": "--fields bodies"
    },
    {
      "surface": "task paths",
      "reveal": "--fields paths"
    },
    {
      "surface": "watch/steer actions",
      "reveal": "--fields actions"
    },
    {
      "surface": "healthy endpoint detail",
      "reveal": "--fields endpoints"
    },
    {
      "surface": "full scout-report inventory",
      "reveal": "--all-reports"
    },
    {
      "surface": "superseded or prose-deferred queued items",
      "reveal": "--all-queued"
    },
    {
      "surface": "landed per-home capped at 6 for 1 home(s)",
      "reveal": "--all-landed"
    },
    {
      "surface": "live PR discovery + checks",
      "reveal": "--include-prs"
    }
  ]
}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (2) ✅

• ⚠️ bin/fm-fleet-snapshot.sh:716 - The new shell loop independently parses the same open-decision TSV grammar already parsed into open_decisions_json immediately above, solely replacing two jq queries. No stated requirement needs this second parser, and the accepted scope says the broader reduction of small jq invocations was not pursued. Remove this loop and derive the booleans from open_decisions_json as before, unless this extra optimization is explicitly intended.

🔧 Fix: Restore JSON-derived decision flags
1 error still open:

• 🚨 bin/fm-fleet-snapshot.sh:545 - The status log is copied at line 536, but the subsequently launched fm-crew-state process still reads the live $STATE/$id.status. If an idle task appends needs-decision or blocked after the copy but before fm-crew-state reads it, current_state reports parked/blocked while last_event and open_decisions come from the older copy and hide the actionable decision. Generation checks do not catch same-generation status updates. Pass the captured status path to fm-crew-state through a status-log override so all status-derived fields use the same observation.

🔧 Fix: Unify status-derived snapshot observations
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• bin/fm-test-run.sh --changed --exclude-family real-herdr-gated
• Baseline: bin/fm-test-run.sh --changed --exclude-family real-herdr-gated
• bash tests/fm-backend.test.sh
• bash tests/fm-bearings-snapshot.test.sh
• /bin/bash tests/fm-backend.test.sh using macOS Bash 3.2
• /bin/bash .../run-large-local-evidence.sh against a five-worker, 300-entry synthetic home

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the previously reported generation-coherence and teardown paths.

The current implementation captures stable task metadata and status observations, handles teardown during manifest capture, and brackets reusable endpoint reads with generation checks, resolving the prior snapshot race findings.

_{Reviews (9): Last reviewed commit: "no-mistakes(ci): Updated the stock macOS..." | Re-trigger Greptile}