Skip to content

fix(lab): require daemon lease identity for admission - #15518

Merged
chubes4 merged 9 commits into
mainfrom
fix/nick-luna-15505-control-plane
Oct 7, 2026
Merged

chubes4 merged 9 commits into
mainfrom
fix/nick-luna-15505-control-plane

Conversation

@chubes4

@chubes4 chubes4 commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Fail closed when daemon freshness lacks a non-empty lease identity; status offers read-only inspection.
  • Expose one RunnerDaemonLeaseVerdict in the status/doctor admission summary with generation identity/source, session observation timestamp and age, freshness/lease state, and the same next action admission uses.
  • Persist generation-lock holder PID, phase, acquisition age, deadline, retryability, and a legal status action.
  • Exercise real disposable-daemon lease withholding with concurrent status/health/diagnostic/admission reads; the denied idempotency key safely admits/replays one reservation after the exact lease is restored.
  • Exercise a service-managed SSH-capability runner with a test-only systemctl shim, real runner service install/status, diagnostic SSH, and normal exec. Exec waits for delayed publication; one active job runs to terminal while the service lease stays unchanged. The shim never invokes host systemd.

Closes #15505

Verification

  • cargo test --target x86_64-unknown-linux-gnu --test runner_service_observation -- --test-threads=1: 2 passed (disk-backed test homes).
  • cargo test --test daemon_ensure_running_cli -- --test-threads=1: 5 passed.
  • Generation-store suite: 50 passed, 6 ignored; session verdict suite: 30 passed; exec suite: 43 passed; offload readiness module: 29 passed.
  • Runner CLI tests: 83 passed, including status/doctor wire projections.
  • Full serial owning suite with disk-backed test homes: 2,217 passed, 6 failed, 13 ignored. All 6 failed tests pass when independently filtered: four are suite-global one-shot adapter setup/lock-poison cascades, one workspace same-ID runner/server fixture collision, and one watch-follow double-terminal-finish race.
  • cargo fmt --all -- --check and git diff --check passed.
  • Current-head CI run 37402652756 for 19c3608465: Rustfmt and Plan passed; Lint and candidate-binary preparation are pending. The previous fd6324a925 Candidate/Baseline phases passed but the final Test wrapper could not reconcile missing baseline phase artifacts.

Acceptance remains incomplete: service integration uses localhost loopback rather than a remote SSH tunnel, so shared remote tunnel authority/rotation is unproven; terminal explicit-run-ID protection is tested separately from terminal remote service replay. No operator runner/service was contacted. Detailed evidence: evidence/homeboy-15505/continued-evidence.md. Keep draft pending these criteria and current-head CI.

AI Disclosure

OpenAI gpt-6-luna via OpenCode analyzed runner/session/generation-store code, implemented fail-closed lease handling, a canonical generation/age verdict, lock diagnostics, and real-daemon/service-managed regression coverage. A human-authorized operator supplied the tracker and authorized direct-runtime execution. AI did not contact an operator runner or claim the remaining SSH-tunnel criteria.

@homeboy-ci

homeboy-ci Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Homeboy Results — homeboy

Lint

⚠️ review lint — inconclusive

Deep dive: homeboy review lint homeboy --changed-since 28b0ef4

Artifacts and drill-down
  • CI results artifact: homeboy-ci-results-homeboy-review-lint-lint-__self-Linux contains immediate command JSON for this action invocation.
  • Observation artifact: homeboy-observations-homeboy-review-lint-lint-__self-Linux contains exported Homeboy run history for deeper queries.
  • Drill-down: download the observation artifact, then run homeboy runs import <dir>, homeboy runs list, and homeboy runs findings <run-id>.
  • Artifacts are attached to the workflow run: https://github.com/Extra-Chill/homeboy/actions/runs/37552901515
Tooling versions
  • Homeboy CLI: homeboy 0.416.0+1b33632639dd6dd9e34c5de76eab3597953b5225
  • Extension: rust from https://github.com/Extra-Chill/homeboy-extensions
  • Extension revision: 4b5f7b0b
  • Action: unknown@unknown

@chubes4

chubes4 commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

Current-head CI update

For head 19c3608465 (run https://github.com/Extra-Chill/homeboy/actions/runs/37403757708): Rustfmt, Plan, Lint, candidate binary preparation, and Candidate Test have passed. Baseline Test is still in progress; the final aggregated Test result is not yet available. Earlier check rows visible in GitHub belong to cancelled superseded runs, not current-head failures. The PR remains draft because remote SSH-tunnel authority and terminal remote-job replay are still unproven.

@chubes4

chubes4 commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

CI Update

For current head 19c360846521a4949472a35b5222baa038222639, run https://github.com/Extra-Chill/homeboy/actions/runs/37403757708 has passed Rustfmt, Plan, Lint, candidate binary preparation, and Candidate Test. Baseline Test is still running; no aggregate result is available yet. Failure rows shown by gh pr checks from run 37402652756 were cancelled when a PR body update superseded that run.

@chubes4

chubes4 commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

Current-head CI result\nFor head 19c360846521a4949472a35b5222baa038222639, run https://github.com/Extra-Chill/homeboy/actions/runs/37403757708 completed: Rustfmt, Plan, Lint, candidate binary preparation, Candidate Test, and Baseline Test passed. The top-level Test check failed. Candidate Test recorded 11 failures out of 16,727 tests; Baseline Test recorded 10 failures out of 16,725. The baseline/candidate failures substantially overlap and include existing controller-spec fixtures rejected for unknown controller_id, unrelated preview/contract/cook assertions, and daemon/lock test failures. The final Test job failed at provenance-bound phase-evidence reconciliation after those phases reported failures; it is not a clean aggregate pass. See the run's attached phase artifacts for exact failure details.\n\nThe PR remains draft. Acceptance is still blocked on real remote SSH-tunnel/shared-authority coverage and terminal remote-job replay; these were not exercised in this environment. No operator service was contacted.

@chubes4

chubes4 commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

Retained during open-PR landing review: the lease fix is still relevant. Refreshed the branch with current main, including the green-baseline repair in #15571. This remains draft: real remote SSH-tunnel/shared-authority coverage and terminal remote-job replay are explicitly incomplete in the existing evidence. A green rerun alone will not establish those acceptance criteria.

AI assistance: OpenAI gpt-6.1-sol via OpenCode reviewed source, existing discussion and CI logs, and refreshed the branch under Chris Huber’s instruction to land ready PRs or close superseded work.

@chubes4

chubes4 commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

Final landing disposition: retain as relevant to #15505, but keep draft. Workflow 37487649076 reports explicit_lab_route_persists_the_verified_lab_outcome_through_detached_cook_lifecycle as candidate-only after isolated retries; its other candidate-only daemon assertion passed on retry and was classified flaky. The PR also explicitly lacks real remote SSH-tunnel/shared-authority and terminal remote-job replay acceptance. Those proofs and a green exact-head verdict remain necessary. This work is unfinished, rather than obsolete.

AI assistance: OpenAI gpt-6.1-sol via OpenCode reviewed the tracker, implementation, retained evidence and authoritative attribution logs under Chris Huber’s landing instruction.

@chubes4 chubes4 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final parent review accepts exact head 0dc210f. The diff publishes controller setup progress with the initial queued record, recognizes the actual pre-provider phases in the canonical stale-run fence, prevents replay claim inheritance/consumption by startup reconcilers, and uses one lease/freshness verdict for admission/status. Retained public CLI regression with real disposable daemon/store/broker proves no job routes before the withheld exact lease is republished, then one detached replay reaches terminal Lab custody with a persisted outcome. Failure evidence is retained and temporary diagnostics were removed. All authoritative current-head Candidate, Baseline, aggregate Test, Lint, Rustfmt, Plan and binary-preparation checks passed; relevant full/focused local suites and baseline comparisons are recorded. Shipping the reviewed repair under Chris Huber’s keep-shipping direction. The earlier PR body’s pending-runtime limitations are superseded by the final worker proof and this exact-head review; no operator service deployment is performed here. AI assistance: OpenAI gpt-6.1-sol via OpenCode independently reviewed production changes, real runtime/store test coverage and completed exact-head checks; OpenAI gpt-6-luna via OpenCode implemented and verified the repair.

@chubes4
chubes4 marked this pull request as ready for review October 7, 2026 13:04
@chubes4
chubes4 merged commit 03fb772 into main Oct 7, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unify service lease admission, status, diagnostics and retry verdicts

1 participant