feat: add AGX microVM SSH-lease provider

[zozo123]

Closes #341

Summary

Adds an early-access AGX Linux SSH-lease provider using AGX's documented SSH gateway shape.

• Registers provider: agx as a direct, Linux-only SSH-lease backend.
• Builds <user>+<instance>@<workspace> SSH targets from the Crabbox slug and configured AGX workspace gateway.
• Uses the operator's AGX-registered SSH key; there is no AGX API token or AGX control-plane client.
• Keeps normal Crabbox SSH behavior: local claims, rsync sync, command execution, ssh, status, list, and local release.
• Documents the current AGX boundary: no broker, no remote inventory, no remote delete/cleanup API, and no desktop/browser/code flags.
• Adds provider docs, config/flags/env handling, registration, tests, generated provider metadata, and shared live-smoke wiring.

Maintainer repairs on top of the contributor branch hardened the security boundary:

• agx.user must be a plain SSH login name, rejecting option-shaped values, whitespace/control characters, and gateway separators.
• agx.workspace must be a bare hostname, rejecting URLs, ports, userinfo, paths, whitespace/control characters, and option-shaped values.
• AGX provider flags are applied before AGX validation runs.
• Release-owned changelog entries were removed from the feature branch.

Current Verification

Local validation on current head abeb2cba3e8ae2509eb48c98dc338da191ec46d4:

go test ./internal/providers/agx
node scripts/check-docs-links.mjs
node scripts/check-command-docs.mjs
git diff --check

Earlier validation on the repaired branch also passed:

go test ./internal/providers/agx ./internal/providers/all ./internal/cli ./cmd/crabbox
node --test scripts/live-smoke.test.js
node scripts/check-docs-links.mjs
node scripts/check-command-docs.mjs
go vet ./...
go build -trimpath -o bin/crabbox ./cmd/crabbox
git diff --check

Public CI is green on current head:
https://github.com/openclaw/crabbox/actions/runs/28109944795

Remaining Merge Gate

Do not merge while status: needs proof, merge-risk: auth-provider, or merge-risk: security-boundary remain.

This still needs redacted live AGX proof against the real workspace gateway:

CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=agx CRABBOX_LIVE_COORDINATOR=0 \
  CRABBOX_LIVE_REPO=<repo> scripts/live-smoke.sh

Required proof should show doctor, warmup, status --wait, SSH command rendering, one synced command, history/log capture, stop, and expected local-release/AGX-idle-reclaim behavior. Maintainers also need to accept the early-access SSH-only provider contract before merge.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed June 24, 2026, 12:08 PM ET / 16:08 UTC.

Summary
Adds a built-in AGX Linux SSH-lease provider with config, flags, env handling, docs, registration, tests, generated provider metadata, and shared live-smoke wiring.

Reproducibility: not applicable. this is a new provider feature, not a bug report. The relevant validation path is the documented live AGX smoke run, and that proof has not been posted.

Review metrics: 3 noteworthy metrics.

• Diff size: 16 files, +1064/-3. The branch spans provider code, core config, generated metadata/docs, and live-smoke wiring.
• Provider surface: 1 built-in SSH-lease provider added. Maintainers need to accept a new provider/auth contract, not only a local code path.
• Live AGX artifacts: 0 posted. No real gateway output or recording is present even though live provider behavior is the merge gate.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #341
Summary: This PR is the active implementation candidate for the open AGX provider request; adjacent microVM provider PRs do not replace AGX support.

Members:

• canonical: New sandbox kiddo in the block add support #341 - The issue is the open AGX support request and this PR uses closing linkage for it.
• adjacent_distinct: feat(firecracker): add direct microVM provider #358 - The merged Firecracker provider covers a different local/self-hosted microVM backend, not AGX's workspace SSH gateway.
• adjacent_distinct: feat: add AWS Lambda MicroVM provider #674 - The merged AWS Lambda MicroVM provider covers a different provider surface and does not implement AGX support.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Post redacted live AGX smoke output covering doctor, warmup, status, SSH rendering, synced run/history, stop, and local release.
• Get explicit maintainer acceptance or rejection of the early-access SSH-only, local-claim lifecycle contract.

Proof guidance:

• [P1] Needs real behavior proof before merge: No redacted live AGX terminal output, logs, recording, or linked artifact is present; screenshots, terminal output, recordings, or logs are acceptable when private details like IPs, keys, endpoints, and other sensitive data are redacted. Updating the PR body after adding proof should trigger a fresh ClawSweeper review, or a maintainer can comment @clawsweeper re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

• [P1] No redacted live AGX terminal output, logs, recording, or linked artifact proves that the real workspace gateway accepts the configured SSH key, routes <user>+<instance>, supports sync/run/history, and behaves correctly after local release.
• [P1] The provider intentionally has no remote inventory or delete API, so stop removes only the local claim while AGX idle reclaim remains part of the provider contract maintainers must accept.
• [P1] AGX is early access and the PR relies on the public SSH shape, so onboarding, auth, routing, or lifecycle behavior may change after merge.
• [P1] This adds new config that becomes an SSH destination; unit tests cover string validation, but the auth/security boundary still needs live-provider proof or explicit maintainer acceptance.

Maintainer options:

1. Collect live AGX proof before merge (recommended)
   Have someone with an AGX-registered SSH key run the documented live-smoke path and post redacted terminal/log proof for doctor, warmup, status, SSH rendering, run/history, stop, and local release.
2. Accept the early-access provider contract
   A maintainer may explicitly accept the SSH-only, local-claim AGX lifecycle without broader provider APIs, but that decision should be visible before the risk labels clear.
3. Pause until AGX stabilizes
   If no one can prove the real gateway or AGX changes its published contract, pause or close this branch and revisit with a narrower provider PR later.

Next step before merge

• [P1] Automation cannot supply AGX credentials, live gateway proof, or maintainer acceptance of the early-access provider contract.

Security
Needs attention: No narrow code-injection defect is evident after the validation repairs, but the new AGX SSH auth boundary still needs live proof or explicit maintainer acceptance.

Review details

Best possible solution:

Keep the PR open until someone with AGX access posts redacted live-smoke proof and a maintainer explicitly accepts or rejects the SSH-only, local-release provider contract.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a new provider feature, not a bug report. The relevant validation path is the documented live AGX smoke run, and that proof has not been posted.

Is this the best way to solve the issue?

Unclear until signoff: the SSH-only adapter matches the public AGX shape and is reasonably hardened, but it should not merge without live AGX proof and maintainer acceptance of the local-only lifecycle contract.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 0ec69d642764.

Label changes

Label justifications:

• P2: This is a normal-priority provider feature with auth/security impact but no urgent regression or data-loss report.
• merge-risk: 🚨 auth-provider: The PR adds a new provider routing/auth path that depends on an operator AGX-registered SSH key.
• merge-risk: 🚨 security-boundary: The PR turns user/workspace config into an SSH destination and lacks live proof of the real gateway security boundary.
• rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: No redacted live AGX terminal output, logs, recording, or linked artifact is present; screenshots, terminal output, recordings, or logs are acceptable when private details like IPs, keys, endpoints, and other sensitive data are redacted. Updating the PR body after adding proof should trigger a fresh ClawSweeper review, or a maintainer can comment @clawsweeper re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

Security concerns:

• [medium] Prove the AGX SSH auth boundary before merge — internal/providers/agx/backend.go:269
  The PR constructs real SSH destinations from AGX config and uses the operator key; unit tests cover sanitization, but no live gateway proof shows the intended auth and host-key behavior against AGX.
  Confidence: 0.84

Acceptance criteria:

• [P1] CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=agx CRABBOX_LIVE_COORDINATOR=0 CRABBOX_LIVE_REPO= scripts/live-smoke.sh.
• [P1] go test ./internal/providers/agx.
• [P1] node --test scripts/live-smoke.test.js.
• [P1] node scripts/check-docs-links.mjs.
• [P1] node scripts/check-command-docs.mjs.

What I checked:

• Repository policy read: AGENTS.md was read fully; its provider-neutral architecture and secrets guidance apply to this provider/auth change. (AGENTS.md:1, 0ec69d642764)
• Current main lacks AGX support: A current-main search found no AGX provider references in docs, internal code, scripts, or workflows. (0ec69d642764)
• Latest release lacks AGX registry entry: The v0.33.0 provider registry has no AGX import, so the requested provider is not shipped in the latest release. (internal/providers/all/all.go:1, 966e99599db4)
• PR registers AGX provider: PR head defines provider=agx as a Linux SSH-lease provider with SSH and Crabbox sync features and no coordinator. (internal/providers/agx/provider.go:18, abeb2cba3e8a)
• PR uses SSH-only local-claim lifecycle: The backend waits for SSH readiness, records local lease claims, and removes only local state on release because AGX has no published control-plane delete API. (internal/providers/agx/backend.go:106, abeb2cba3e8a)
• SSH destination is security-sensitive: The provider builds <user>+<instance>@<workspace> SSH targets from config and uses cfg.SSHKey, so real auth/gateway proof matters before merge. (internal/providers/agx/backend.go:269, abeb2cba3e8a)

Likely related people:

• vincentkoc: Pushed host-key, docs, live-smoke, validation, and config-hardening repairs on this branch, and current-main history shows recent provider/live-smoke maintenance. (role: recent PR repair author and adjacent provider/live-smoke contributor; confidence: high; commits: 6e6187290f03, ed232fa1d4fb, ca4fc8841553; files: internal/providers/agx/backend.go, internal/providers/agx/backend_test.go, scripts/live-smoke.sh)
• coygeek: Current-main blame and log show substantial recent provider registration, config, and live-smoke work in the same SSH-lease/provider surface. (role: adjacent provider foundation contributor; confidence: high; commits: beb0dabaa97e, 1b5e52ddaf66, 4b5849d2036d; files: internal/providers/all/all.go, internal/cli/config.go, scripts/live-smoke.sh)
• steipete: The latest merged AWS Lambda MicroVM provider touched similar provider registration, config, docs, and metadata surfaces. (role: recent adjacent microVM provider contributor; confidence: medium; commits: 0301236b2752; files: internal/providers/all/all.go, internal/cli/config.go, docs/providers/README.md)
• zozo123: The branch author introduced the AGX implementation and has prior merged provider-area commits, making them useful for AGX-specific context beyond authorship alone. (role: AGX branch proposer and adjacent provider contributor; confidence: medium; commits: 4d2d7924e815, 50b22f0ba913, 973efedd2c87; files: internal/providers/agx/backend.go, internal/providers/agx/provider.go, docs/providers/agx.md)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[zozo123]

Updated the provider to match AGX's published interface after checking their docs.

Research finding: AGX (agx.so) currently publishes no control-plane API, auth contract, or CLI — only the SSH connection shape ssh <user>+<instance>@workspace.agx.so and an explicit stance: "No SDK required, no custom client — if it can ssh, it can work on AGX." It ships Summer 2026 and is built on Loophole Labs' Firecracker/CRIU/Drafter stack.

Change: the first draft invented a /v1/instances REST control plane + API key, which contradicts that "no SDK / no custom client" model. This revision commits only to the documented SSH interface:

• Removed the REST client and API-key requirement; auth uses the operator's own SSH key (cfg.SSHKey, registered with AGX during onboarding), like the Static SSH provider.
• Provision-on-connect: build the <user>+<instance> target and wait for SSH readiness; the slug is the stable instance name.
• List/Resolve are backed by local lease claims; release is local-only (AGX reclaims idle sandboxes). Dropped FeatureCleanup since there's no inventory API.
• AGXConfig trimmed to workspace / user / workRoot.

Still pending a maintainer/product decision (the original issue is tagged needs-product-decision): provisioning lifecycle, key-registration, and any future management API remain inferred from the homepage because AGX hasn't published them. Happy to align to whatever AGX ships once early access opens.

Verified: gofmt, go vet ./..., go test ./internal/cli ./internal/providers/..., go test -race ./internal/providers/agx, and scripts/check-docs.sh all pass. Tests need no live credentials.

[vincentkoc]

Updated this PR to current main and pushed maintainer fixes to zozo123:feat/agx-provider.

Head: eae03061b20855d8473e7814e1ae10c2df283ae7
CI: https://github.com/openclaw/crabbox/actions/runs/28079591126

Changes made:

• rebased onto current main
• removed the direct CHANGELOG.md edit from the rebased branch
• preserved normal SSH host-key verification for the AGX workspace gateway
• updated registry coverage for the new built-in provider
• updated the PR body to describe the current SSH-only, no-control-plane design

Local verification run from the repaired branch:

go test ./internal/providers/agx ./internal/providers/all ./internal/cli ./cmd/crabbox
node scripts/generate-provider-matrix.mjs --check
node scripts/check-command-docs.mjs
node scripts/check-docs-links.mjs
go run golang.org/x/tools/cmd/deadcode@v0.45.0 -test ./...
go vet ./...
go build -trimpath -o bin/crabbox ./cmd/crabbox
git diff --check

Public CI is green across Go, Apple VZ, Worker, Scripts, Docs, and Release Check.

Remaining blocker: I did not merge this because the PR still needs real AGX behavior proof and provider-contract/product signoff. I do not have live AGX access to prove warmup/status/run/stop against the real workspace gateway.

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

[vincentkoc]

@clawsweeper re-review

Pushed the remaining docs repair for the concrete review finding.

Head: ff6fb8cc6f2f04bdf78e60af11f4139e5dd82d0e

Change made:

• AGX docs now point users at shared SSH key configuration: --ssh-key, ssh.key, or SSH agent
• the YAML example now uses:

ssh:
  key: ~/.ssh/id_ed25519

Local validation passed:

node scripts/check-docs-links.mjs
node scripts/check-command-docs.mjs
git diff --check

Still not merging: real AGX warmup/status/run/stop proof and provider-contract/product signoff remain missing.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

[vincentkoc]

Public CI is now green on current head ff6fb8ccae76a175456dd162a467c720c4c5e14f: https://github.com/openclaw/crabbox/actions/runs/28084598113.

Still not merging: real AGX warmup/status/run/stop proof and provider-contract/product signoff remain missing, with status: 📣 needs proof, merge-risk: 🚨 auth-provider, and merge-risk: 🚨 security-boundary still present.

[vincentkoc]

Rebased this PR onto current main after #674 landed.

New head: b14dfcaac73333cadfe062a848874d9869516083

Conflict resolution kept AWS Lambda MicroVM and AGX in generated provider docs. Provider matrix now reports 67 providers.

Local validation on the rebased head:

go test ./internal/providers/agx ./internal/providers/all ./internal/cli ./cmd/crabbox
node scripts/generate-provider-matrix.mjs --check
node scripts/check-command-docs.mjs
node scripts/check-docs-links.mjs
go vet ./...
go run golang.org/x/tools/cmd/deadcode@v0.45.0 -test ./...
go build -trimpath -o bin/crabbox ./cmd/crabbox
git diff --check

Still not merging: live AGX warmup/status/run/stop proof and auth/security-boundary gates are still required.

[vincentkoc]

Public CI is green on rebased head b14dfcaac73333cadfe062a848874d9869516083: https://github.com/openclaw/crabbox/actions/runs/28087365291

Green checks: Go, Apple VZ, Worker, Scripts, Docs, and Release Check. Merge state is clean.

Still not merging: status: 📣 needs proof plus auth-provider/security-boundary labels remain, and this still needs redacted live AGX warmup/status/run/stop proof plus maintainer acceptance of the early-access SSH-only provider contract.

[vincentkoc]

Public CI is green on current head ed232fa1d4fbd1efdc9fb86237dc031cc41961c3: https://github.com/openclaw/crabbox/actions/runs/28096459945

Green checks: Go, Apple VZ, Worker, Scripts, Docs, and Release Check.

This includes the new AGX live-smoke harness coverage. Still not merging: status: 📣 needs proof, merge-risk: 🚨 auth-provider, and merge-risk: 🚨 security-boundary remain. The PR still needs redacted live AGX warmup/status/run/stop/local-release proof against the real workspace gateway plus maintainer acceptance of the early-access SSH-only provider contract.

[vincentkoc]

@clawsweeper re-review

Updated the PR body with the canonical summary, validation evidence, and remaining merge gate for current head. No code changes in this update.

Still not merging unless the live proof labels clear and the provider-specific proof/contract gate is satisfied.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

[vincentkoc]

@clawsweeper re-review

Follow-up repair pushed at ca4fc884adf5dc134ba4f329e23496db2eddef9e.

What changed:

• Hardened agx.user so it must be a plain SSH login name, rejecting option-shaped values, whitespace/control chars, and gateway separators before Crabbox builds <user>+<instance>@<workspace>.
• Hardened agx.workspace so it must be a bare hostname, rejecting URLs, ports, userinfo, paths, whitespace/control chars, and option-shaped values before SSH destination construction.
• Fixed AGX provider flag application order so --agx-user, --agx-workspace, and --agx-work-root are applied before AGX validation runs.
• Documented the AGX destination constraints and added an Unreleased changelog entry.

Validation run locally on current head:

• go test ./internal/providers/agx
• go test ./internal/providers/agx ./internal/providers/all ./internal/cli ./cmd/crabbox
• node --test scripts/live-smoke.test.js
• node scripts/check-docs-links.mjs
• node scripts/check-command-docs.mjs
• go vet ./...
• go build -trimpath -o bin/crabbox ./cmd/crabbox
• git diff --check

I still cannot provide live AGX gateway proof without an AGX-registered SSH key, so this should not be treated as live-provider validation. The security-boundary/auth-provider labels should stay until reviewer acceptance decides the credentials-free proof is sufficient or someone can run the AGX live smoke.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

[vincentkoc]

@clawsweeper re-review

Follow-up maintainer repair pushed in abeb2cba43083224966d94bdf1e08ef490f2809a.

What changed:

• Removed the release-owned CHANGELOG.md entry from this feature branch.
• Kept the AGX validation/security details in the PR discussion instead of the changelog.

Validation:

• go test ./internal/providers/agx
• node scripts/check-docs-links.mjs
• node scripts/check-command-docs.mjs
• git diff --check

Still not merge-ready: I still cannot provide live AGX gateway proof without an AGX-registered SSH key, so status: needs proof, auth-provider, and security-boundary should remain until live provider proof or explicit maintainer acceptance clears that gate.

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

[vincentkoc]

@clawsweeper re-review

Updated the PR body to reflect that public CI is green on current head abeb2cba3e8ae2509eb48c98dc338da191ec46d4: https://github.com/openclaw/crabbox/actions/runs/28109944795

No code changes in this update. Remaining gate is still redacted live AGX gateway proof plus maintainer acceptance of the early-access SSH-only provider contract.

[steipete]

Closing this implementation PR while keeping #341 open for a future stable AGX integration.

AGX remains an early-access SSH gateway rather than a provider lifecycle API. This branch can synthesize a gateway target, but it cannot inventory, create, verify ownership of, or delete a remote instance; stop only removes Crabbox's local claim and relies on undocumented idle reclamation. That is not a sufficient ownership and cleanup contract for a built-in SSH-lease provider, and there is no real gateway proof on the final branch head.

Thanks for the focused provider work and the later validation hardening. This lane can reopen when AGX publishes a stable onboarding/control-plane contract—or a documented gateway lifecycle with authoritative remote status and cleanup—and we can prove it end to end against a real workspace.