Pin managed Linux bootstrap installers

[TurboTheTurtle]

Closes #715.

Summary

• Replace managed Linux Code and Tailscale root curl | sh installer pipelines with pinned, checksum-verified archives.
• Install code-server under /usr/local/lib/code-server with a stable /usr/local/bin/code-server symlink.
• Treat legacy CRABBOX_TAILSCALE_INSTALL_MODE=package as pinned static mode; preserving package-mode behavior would require a separate signed-repository implementation.
• Remove downloaded archives and temporary install directories after success while retaining EXIT cleanup on failure.
• Preserve current-main Google Linux signing-key and source-scoped APT hardening.
• Update operator docs and add an Unreleased changelog entry thanking @TurboTheTurtle.

Verification

Exact head: 45cd2f1f5ac8199543a7890188705b20283635db (rebased onto the 0.34.1 development head). Exact-head CI is terminal green: https://github.com/openclaw/crabbox/actions/runs/28615364564

• go vet ./... and go test -race ./... — pass.
• go test -race ./internal/cli -run '^TestWindowsWSL2BootstrapCompleteProbeUsesWindowsMarker$' -count=3 — pass; confirms the unrelated full-matrix timeout is load-sensitive.
• npm run format:check --prefix worker
• npm run lint --prefix worker
• npm run check --prefix worker
• npm test --prefix worker — 25 files, 753 tests passed.
• npm run build --prefix worker
• scripts/check-docs.sh — 51 command docs, 69-provider matrix, 202 Markdown files, and docs build passed.
• Downloaded the official code-server 4.126.0 amd64 and Tailscale 1.98.4 amd64 archives; both matched the pinned SHA-256 values.
• Autoreview against origin/main — clean, no accepted/actionable findings.
• A first real AWS canary exposed that cp -a preserved the private mktemp root mode and made code-server non-executable by the lease user. The current head restores mode 0755 after the copy in both CLI- and coordinator-generated bootstrap paths, with ordering regression coverage and clean focused/whole-branch autoreviews.
• Exact-head AWS canary from binary SHA-256 60fc8745b73d0f4401fd5f5b7647309609e106561b8812f4e833f6809806be72 — code-server 4.126.0 executed as the lease user, install root mode was 0755, archive residue was absent, readiness passed, and the disposable EC2 instance and provider key both returned to zero.

Remaining live gate

Boot one managed Linux canary with Tailscale enabled, verify Tailscale 1.98.4, prove a 100.x tailnet address, and confirm provider/key/node cleanup. The one-off ephemeral Tailscale credential remains private and is injected only for that validation run.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed July 2, 2026, 4:16 PM ET / 20:16 UTC.

Summary
Replaces managed Linux Code and Tailscale bootstrap installer scripts in the Go and Worker paths with pinned checksum-verified archives, updates Tailscale install-mode defaults/docs/tests, and adds release-note text.

Reproducibility: yes. for source-level reproduction: current main and v0.34.0 still render the Code and package-mode Tailscale root curl-to-shell installer paths. I did not run a live installer compromise, and after-fix Tailscale runtime proof remains incomplete.

Review metrics: 3 noteworthy metrics.

• Changed surface: 12 files, +167/-36. The diff spans Go bootstrap, Worker bootstrap, config, tests, docs, and release notes, so parity and upgrade behavior matter before merge.
• Installer types hardened: 2 installer types across 2 generators. The central security change replaces both Code and Tailscale root installer-script paths in CLI and coordinator bootstrap generation.
• Remaining live gate: 1 Tailscale canary. The PR body and comments still name a managed Linux Tailscale canary as required proof before landing.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #715
Summary: This PR is the active candidate fix for the canonical managed Linux Code and Tailscale installer-integrity issue; related reports were consolidated around that issue.

Members:

• canonical: [security] Linux bootstrap executes unpinned installer scripts as root #715 - The PR body uses closing syntax for this open issue, and GitHub lists it as the closing issue reference.
• duplicate: [security] Linux bootstrap executes remote install scripts without pinning or digest verification #711 - The broader closed security report covers the same root installer-script integrity problem and cross-references this PR path.
• duplicate: [security] Unpinned code-server bootstrap executes upstream installer as root #723 - The Code-only report is a subset of the same managed Linux installer-script root cause.
• superseded: [security] Pin and verify managed code-server bootstrap downloads #724 - The earlier code-server pinning request was closed in favor of later consolidated installer-integrity tracking.
• partial_overlap: Make Tailscale bootstrap installation more deterministic #311 - The deterministic Tailscale install issue overlaps the pinned Tailscale direction but preserved package-mode compatibility as a distinct concern.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🦪 silver shellfish
Patch quality: 🧂 unranked krab
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• [P1] Add redacted latest-head managed Linux Code+Tailscale canary proof showing Tailscale 1.98.4, a 100.x address, and cleanup; redact IPs, keys, endpoints, and other private details.
• Resolve the package-mode upgrade decision by preserving signed package semantics or recording explicit maintainer acceptance of the pinned alias.
• Remove Worker code-server archive residue and move release-note context out of CHANGELOG.md unless release ownership is explicit.

Proof guidance:

• [P1] Needs stronger real behavior proof before merge: Partial archive-install logs and a Code canary are useful, but latest-head Tailscale tailnet proof with a redacted 100.x address and cleanup evidence is still missing; update the PR body with redacted live output so ClawSweeper can re-review.

Risk before merge

• [P1] Existing operators using CRABBOX_TAILSCALE_INSTALL_MODE=package will silently receive pinned static archive behavior unless maintainers explicitly accept that upgrade behavior or preserve signed package-mode semantics.
• [P1] The latest PR head still lacks a Tailscale-enabled managed Linux canary proving Tailscale 1.98.4, a 100.x address, and provider/key/node cleanup with a real injected credential path.
• [P1] The Worker coordinator Code path appears to copy code-server.tgz into /usr/local/lib/code-server because CS_ARCHIVE is not removed before cp -a copies the temp directory.
• [P1] CHANGELOG.md is release-owned under this review policy, so the release-note text needs release-owner ownership or should move to PR-body/commit-message context.

Maintainer options:

1. Finish the guarded landing path (recommended)
   Resolve the package-mode compatibility decision, remove Worker archive residue and release-owned changelog churn, then add redacted latest-head managed Linux Code+Tailscale canary proof before merge.
2. Accept package as a pinned alias
   Maintainers may intentionally accept that legacy package mode now uses the pinned static archive path, but the upgrade behavior should be explicit and backed by runtime proof.
3. Pause until Tailscale proof is possible
   If no approved ephemeral Tailscale credential is available, keep the PR paused rather than landing an unproven fail-closed bootstrap path.

Next step before merge

• [P1] Human review remains because package-mode compatibility and live Tailscale credential proof are maintainer-controlled, though Worker archive cleanup and release-note removal are concrete local fixes.

Security
Cleared: No concrete new security or supply-chain regression was found; the diff improves bootstrap integrity by removing privileged remote installer-script execution, while remaining blockers are compatibility, cleanup, proof, and release ownership.

Review findings

• [P1] Preserve package mode or record approval — worker/src/tailscale.ts:73
• [P2] Remove the copied code-server archive — worker/src/bootstrap.ts:1497
• [P3] Move release-note text out of the changelog — CHANGELOG.md:7

Review details

Best possible solution:

Land the pinned-archive hardening after package-mode upgrade behavior is preserved or explicitly accepted, Worker archive residue is fixed, release notes are handled outside normal PR-owned changelog edits, and redacted latest-head Code+Tailscale canary proof is posted.

Do we have a high-confidence way to reproduce the issue?

Yes for source-level reproduction: current main and v0.34.0 still render the Code and package-mode Tailscale root curl-to-shell installer paths. I did not run a live installer compromise, and after-fix Tailscale runtime proof remains incomplete.

Is this the best way to solve the issue?

No, not merge-ready yet. Pinned verified archives are the right direction, but package-mode compatibility, Worker code-server archive residue, release-owned changelog handling, and live Tailscale canary proof need resolution first.

Full review comments:

• [P1] Preserve package mode or record approval — worker/src/tailscale.ts:73
  Always returning pinned makes CRABBOX_TAILSCALE_INSTALL_MODE=package a pinned-static alias, so existing operators who selected package-repository semantics lose that behavior on upgrade. Preserve a signed package path or record explicit maintainer acceptance with upgrade proof before merge.
  Confidence: 0.86
• [P2] Remove the copied code-server archive — worker/src/bootstrap.ts:1497
  The Worker path leaves CS_ARCHIVE inside CS_INSTALL_DIR, then cp -a copies the whole temp directory into /usr/local/lib/code-server. That contradicts the no-archive-residue behavior and differs from the Go path, which removes the archive before copying.
  Confidence: 0.9
• [P3] Move release-note text out of the changelog — CHANGELOG.md:7
  This workflow treats CHANGELOG.md as release-owned, so this normal PR should keep release-note context in the PR body or commit message unless a release owner explicitly takes the changelog edit.
  Confidence: 0.78

Overall correctness: patch is incorrect
Overall confidence: 0.9

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against ab1c29f3fd40.

Label changes

Label justifications:

• P1: The PR addresses high-impact managed-bootstrap supply-chain hardening, but merge still carries compatibility and availability blockers for documented managed Linux workflows.
• merge-risk: 🚨 compatibility: Merging changes legacy CRABBOX_TAILSCALE_INSTALL_MODE=package behavior to the pinned static archive path for existing operators.
• merge-risk: 🚨 availability: Merging a fail-closed pinned bootstrap path without latest-head Code+Tailscale canary proof could break managed Linux readiness at runtime.
• rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🦪 silver shellfish and patch quality is 🧂 unranked krab.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Partial archive-install logs and a Code canary are useful, but latest-head Tailscale tailnet proof with a redacted 100.x address and cleanup evidence is still missing; update the PR body with redacted live output so ClawSweeper can re-review.

Evidence reviewed

Acceptance criteria:

• [P1] go test -race ./internal/cli -run 'TestCloudInit(Code|Tailscale)'.
• [P1] npm test --prefix worker -- bootstrap.test.ts tailscale.test.ts.
• [P1] Boot a managed Linux lease with Code and Tailscale enabled from the latest PR head; verify code-server 4.126.0, Tailscale 1.98.4, a redacted 100.x tailnet address, no archive residue, and provider/key/node cleanup.

What I checked:

• Repository policy read: AGENTS.md was read fully and applied as review context for the Go CLI, Worker, docs, and testing surfaces touched by this PR. (AGENTS.md:1, ab1c29f3fd40)
• Current main still has the reported behavior: Current main still renders the managed Linux code-server root curl-to-shell installer and package-mode Tailscale installer-script path, so the PR is not obsolete. (worker/src/bootstrap.ts:1467, ab1c29f3fd40)
• Latest release has not shipped the hardening: v0.34.0 points at ef898c2 and still contains the same code-server and Tailscale installer-script paths. (worker/src/bootstrap.ts:1467, ef898c27570a)
• PR head adds pinned archive installs: The latest PR head uses versioned code-server and Tailscale archive URLs with SHA-256 checks before extraction in the Worker bootstrap path. (worker/src/bootstrap.ts:1492, 61bb275a9f65)
• Compatibility-sensitive mode change: The PR head always returns Tailscale install mode as pinned, making legacy CRABBOX_TAILSCALE_INSTALL_MODE=package a static-archive alias instead of preserving package semantics. (worker/src/tailscale.ts:73, 61bb275a9f65)
• Worker code-server archive residue: The Worker path extracts code-server into CS_INSTALL_DIR and then copies the entire directory into /usr/local/lib/code-server without removing CS_ARCHIVE first; the Go path removes the archive before copying. (worker/src/bootstrap.ts:1497, 61bb275a9f65)

Likely related people:

• steipete: Recent main history shows repeated bootstrap hardening work, and the current PR branch includes follow-up commits from steipete for code-server traversal, release-note placement, and installer cleanup. (role: recent area contributor and PR repair author; confidence: high; commits: a6e328cdfbe6, 79164fd14001, 1206ead310c2; files: worker/src/bootstrap.ts, internal/cli/bootstrap.go, worker/src/tailscale.ts)
• vincentkoc: GitHub history shows vincentkoc authored Tailscale bootstrap and cleanup hardening adjacent to this install path and committed the prior blamed current-main installer behavior. (role: feature history contributor; confidence: medium; commits: 938df1d9dfd5, 4b97692bae03; files: worker/src/tailscale.ts, worker/src/bootstrap.ts, internal/cli/bootstrap.go)
• coygeek: A prior available provenance trail ties the current Code and Tailscale installer-script stanzas to commit 4b97692 authored by coygeek and committed by vincentkoc. (role: introduced affected behavior in available history; confidence: medium; commits: 4b97692bae03; files: worker/src/bootstrap.ts, internal/cli/bootstrap.go, worker/src/tailscale.ts)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[TurboTheTurtle]

Partial runtime evidence is available from a disposable Ubuntu runner, but this still does not satisfy the full tailnet-readiness proof ClawSweeper requested.

Proof run: https://github.com/TurboTheTurtle/crabbox/actions/runs/28345365585/job/83967514604

Log highlights from artifact pr742-linux-archive-proof-log:

• Checked out PR head 22ef2e87778e2b3c033ff2958179a58f03080b4d.
• Confirmed no remaining code-server.dev/install.sh or tailscale.com/install.sh refs in worker/src or internal/cli.
• Downloaded code-server 4.126.0 from the pinned GitHub release archive; sha256sum -c - returned OK; /usr/local/bin/code-server --version returned 4.126.0 ...; code_server_archive_proof=success.
• Downloaded Tailscale 1.98.4 from the pinned static archive path; sha256sum -c - returned OK; both tailscale version and tailscaled --version returned 1.98.4; tailscale_archive_proof=success.
• tailnet_join_proof=not_run_no_auth_key_or_control_plane_secret_available.

So the archive-install pieces are proven live, but the requested real managed Linux Tailscale lease proof still needs an actual auth key/control-plane credential path. I’m not requesting ClawSweeper re-review for this PR yet because that missing tailnet join is still a real proof gap.

[steipete]

Maintainer proof update: exact head 7d3028c4166a4c8a7b26b62b39a9e9e6987bfea2 is now green across Go, Apple VZ, Worker, Scripts, Docs, and Release Check; GitHub reports the branch clean.

The only remaining landing gate is the Code + Tailscale canary already described in the PR body: use a one-off ephemeral key authorized for tag:crabbox, boot the sandbox, and prove its 100.x tailnet address. No approved key is currently available, so this remains unmerged.

[TurboTheTurtle]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

[steipete]

Superseded by the maintainer replacement #792 after GitHub rejected pushes to this fork despite maintainer edits being enabled. The replacement preserves Andy Ye as co-author and retains @TurboTheTurtle’s changelog credit while adding the reviewed Go/Worker parity fixes and live-gate plan. Thank you for the original hardening work.