Add first-class Firecracker microVM provider support

[coygeek]

Add first-class Firecracker microVM provider support

Summary

Crabbox should support a first-class firecracker provider that provisions local or self-hosted Linux microVMs with Firecracker, exposes them as normal Crabbox SSH leases, and then reuses Crabbox's existing sync/run/ssh/stop/cleanup workflow.

Today Crabbox has several Firecracker-adjacent providers, but no raw Firecracker provider:

• tensorlake is documented as Firecracker-backed delegated execution, but does not provision raw Firecracker microVMs.
• e2b, smolvm, vercel-sandbox, and similar providers hide the microVM lifecycle behind hosted sandbox APIs.
• apple-vz, multipass, kubevirt, proxmox, and xcp-ng prove that Crabbox already has the provider model for local or self-hosted VM substrates exposed as SSH leases.

The missing piece is a direct Firecracker adapter for users who operate a Linux KVM host and want Crabbox to create short-lived microVM test boxes directly.

Motivation

Firecracker is purpose-built for fast, minimal-overhead, strongly isolated microVMs. That maps well to Crabbox's core job: lease disposable compute, sync the current checkout, run a command, collect output, and clean up.

This would give Crabbox a local/self-hosted microVM option that sits between:

• local-container: fastest path, weaker isolation because it shares the host kernel;
• multipass/incus/kubevirt: VM isolation, but through a larger runtime or cluster layer;
• hosted delegated sandboxes: convenient, but provider-owned sync/run semantics and no raw Firecracker lifecycle.

A Firecracker provider would be especially useful for:

• local Linux hosts with /dev/kvm;
• bare-metal CI runners;
• self-hosted agent fleets that need fast disposable Linux leases;
• testing workloads where container isolation is not enough but full cloud VM provisioning is too slow or expensive.

Evidence From Firecracker Projects

The official firecracker-microvm/firecracker project exposes a VMM API over a Unix socket. It requires Linux with KVM access, a guest kernel image, a root filesystem image, and explicit network setup through tap devices or equivalent CNI-backed plumbing.

The official firecracker-microvm/firecracker-go-sdk project provides a Go library for launching and managing Firecracker machines. It supports:

• firecracker.Config with SocketPath, KernelImagePath, KernelArgs, Drives, NetworkInterfaces, MachineCfg, JailerCfg, VMID, and snapshot-related fields;
• Machine.Start, Machine.StopVMM, Machine.Wait, and metadata APIs;
• static tap interfaces and CNI-configured interfaces;
• optional jailer configuration for stronger host-side isolation.

The official firecracker-microvm/firectl project is a small CLI on top of the Go SDK. It is useful as an implementation reference, but a Crabbox built-in provider should likely use the Go SDK directly so Crabbox can own leases, labels/claims, cleanup, structured diagnostics, and future snapshot support.

The firecracker-microvm/firecracker-containerd project targets containerd integration. It is valuable background, but it is probably not the right first Crabbox adapter surface because Crabbox wants a provider-neutral SSH lease, not a containerd-specific runtime path.

Proposed Provider Shape

Add a built-in direct provider:

provider: firecracker
target: linux
firecracker:
  binary: firecracker
  jailer: ""
  kernel: /var/lib/crabbox/firecracker/vmlinux
  rootfs: /var/lib/crabbox/firecracker/rootfs.ext4
  user: crabbox
  workRoot: /work/crabbox
  cpus: 4
  memoryMiB: 4096
  diskMiB: 16384
  network: cni
  cniNetwork: crabbox-firecracker
  cniConfDir: /etc/cni/conf.d
  cniBinDir: /opt/cni/bin
  launchTimeout: 2m
  deleteOnRelease: true

Provider metadata:

• Name: firecracker
• Aliases: none initially, unless maintainers prefer fc
• Family: local-vm or self-hosted-virtualization
• Kind: ssh-lease
• Targets: linux
• Features, first pass: ssh, crabbox-sync, cleanup
• Coordinator: never

The first implementation should keep scope tight:

• Direct mode only; no coordinator support.
• Linux host only.
• Linux guest only.
• SSH lease only; no delegated-run mode.
• No desktop/browser/code support in the first pass.
• No Firecracker snapshot/checkpoint support in the first pass, but leave the config and state layout compatible with adding native checkpoint support later.

Lifecycle Requirements

warmup / fresh run should:

1. Verify host requirements:
   • Linux host.
   • Firecracker binary is present and executable.
   • /dev/kvm exists and is readable/writable.
   • configured kernel and rootfs image paths exist.
   • network mode prerequisites are present.
2. Generate a Crabbox lease ID, slug, per-lease SSH key, socket path, log path, and state directory.
3. Prepare a per-lease writable rootfs overlay or copy so concurrent leases do not mutate the same base rootfs.
4. Inject or otherwise arrange the Crabbox SSH public key for the configured guest user.
5. Start a Firecracker microVM through the Go SDK.
6. Configure networking so Crabbox can reach guest SSH from the host.
7. Wait for SSH readiness with the standard Crabbox ready check.
8. Persist a local claim with enough Firecracker-specific metadata to resolve and stop the lease later.
9. Return a normal LeaseTarget so Crabbox core sync/run/ssh behavior remains unchanged.

run --id, ssh --id, status, and inspect should resolve by lease ID, slug, or provider VM name using local claims plus live process/socket/network state.

stop should terminate the VMM, remove local claim state, remove per-lease SSH keys, and delete per-lease rootfs/network/socket/log artifacts when deleteOnRelease is true.

cleanup --provider firecracker should delete stale Crabbox-owned Firecracker leases and ignore non-Crabbox Firecracker processes or state directories.

Networking

The first provider should support one reliable SSH path. Preferred initial direction:

• Use the Firecracker Go SDK CNI support where possible.
• Document the expected CNI chain and recommend tc-redirect-tap because the SDK already documents it as the practical bridge from CNI-created interfaces to Firecracker tap devices.
• Record the guest IP assigned by CNI in the local lease claim.

If CNI is too much for the first implementation, a narrower static tap mode is acceptable as long as it is documented and deterministic:

• create a per-lease tap device;
• assign a host-side gateway IP;
• configure guest static IP through kernel args, MMDS, cloud-init-compatible rootfs preparation, or documented image requirements;
• tear down tap/NAT rules on release.

Whichever mode is chosen, cleanup must not remove user-owned network resources. Firecracker state should use Crabbox-owned names/prefixes and persisted metadata.

Image And Guest Preparation

The provider needs a clear guest contract. The guest image must boot with:

• OpenSSH server;
• git;
• rsync;
• tar;
• python3;
• a configured SSH user;
• a writable Crabbox work root.

For v1, it is acceptable to require a prebuilt rootfs image that already satisfies this contract, plus a documented helper path for creating one. The issue should not require Crabbox to solve a full image-building pipeline in the first PR.

Future follow-up could add:

• crabbox firecracker image build helper;
• cloud-image conversion support;
• rootfs patching for authorized keys;
• snapshot/fork support from a warmed base image.

Implementation Touchpoints

Likely code/docs changes:

• internal/providers/firecracker/
  • provider registration and spec;
  • flags;
  • config validation;
  • backend lifecycle;
  • network helpers;
  • state/process cleanup helpers;
  • unit tests with fake Firecracker/SDK runner abstractions.
• internal/providers/all/all.go
  • blank import for the provider.
• internal/cli/config.go
  • FirecrackerConfig;
  • YAML config parsing;
  • environment overrides;
  • defaults and explicit markers where needed.
• internal/cli/config_cmd.go
  • crabbox config show rendering.
• docs/providers/firecracker.md
  • full provider documentation.
• docs/providers/README.md
  • generated provider matrix entry and notes.
• README.md
  • add provider row after docs matrix generation if the README provider table is generated or manually curated there.
• scripts/live-smoke.sh or a new focused live smoke script
  • optional host-gated live smoke when /dev/kvm and Firecracker prerequisites exist.

Acceptance Criteria

• crabbox providers --json includes firecracker with kind ssh-lease, target linux, features ssh, crabbox-sync, and cleanup, and coordinator never.
• crabbox doctor --provider firecracker --json reports actionable host readiness:
  • host OS;
  • /dev/kvm availability;
  • Firecracker binary version or presence;
  • kernel/rootfs path status;
  • configured network mode readiness.
• crabbox warmup --provider firecracker --slug fc-smoke creates a microVM and returns a normal lease ID/slug.
• crabbox run --provider firecracker --id fc-smoke -- uname -a executes through normal Crabbox SSH sync/run.
• crabbox ssh --provider firecracker --id fc-smoke opens the normal Crabbox SSH session.
• crabbox stop --provider firecracker fc-smoke terminates the VMM and removes Crabbox-owned local artifacts.
• crabbox cleanup --provider firecracker --dry-run reports stale owned leases without touching user-owned Firecracker resources.
• Unit tests cover config defaults, flag/env parsing, validation failures, acquire rollback, resolve, release, and cleanup ownership checks.
• Docs include host prerequisites, guest image contract, network setup, config, flags, environment overrides, lifecycle behavior, troubleshooting, and live smoke instructions.

Non-Goals For The First PR

• Coordinator/broker support.
• Hosted Firecracker products or Firecracker-compatible cloud APIs.
• firecracker-containerd runtime integration.
• Kubernetes orchestration.
• Native snapshots/checkpoints/forks.
• Desktop/browser/code provisioning.
• Multi-tenant production hardening beyond a clear local/self-hosted operator warning.
• Automatic kernel/rootfs building unless maintainers want that included.

Open Questions

• Should the provider be classified as local-vm or self-hosted-virtualization in the provider matrix?
• Should the first implementation require CNI, or should it start with a static tap mode?
• Should rootfs mutation be implemented in v1, or should v1 require a preprepared image with the Crabbox SSH user already present?
• Should jailer support be included in v1 as optional configuration, or deferred until the basic non-jailer lifecycle is stable?
• Is fc an acceptable alias, or should the first release use only the explicit firecracker provider name?

Suggested First PR Slice

Implement the smallest useful SSH-lease provider:

• direct Linux-only provider;
• SDK-driven VMM lifecycle;
• preprepared kernel/rootfs inputs;
• one documented network mode;
• normal Crabbox per-lease SSH key and local claim handling;
• doctor, warmup, run, ssh, stop, list, cleanup;
• provider docs and matrix regeneration;
• host-gated live smoke documentation.

This keeps the first PR reviewable while preserving a clean path to stronger guest image tooling, jailer hardening, and native snapshot/fork support later.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed June 14, 2026, 7:35 PM ET / 23:35 UTC.

Summary
Keep this issue open: current main still lacks a direct firecracker SSH-lease provider, and the same work is already owned by an open linked implementation PR that has not merged yet.

Reproducibility: no. reproduction is not applicable because this is a new provider request rather than broken existing behavior. Source and docs inspection confirm current main lacks the requested direct Firecracker provider surface.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.
• Include redacted logs or terminal output.

Next step
A same-author open PR with closing syntax already owns the implementation path, so ClawSweeper should not queue a separate fix PR for this issue.

Review details

Best possible solution:

Review and either land #358 as the provider implementation or replace it with a narrower maintainer-approved Firecracker adapter slice.

Do we have a high-confidence way to reproduce the issue?

No; reproduction is not applicable because this is a new provider request rather than broken existing behavior. Source and docs inspection confirm current main lacks the requested direct Firecracker provider surface.

Is this the best way to solve the issue?

Yes in direction, but not complete yet; a provider adapter matches Crabbox's architecture, and the open linked PR is the current implementation candidate to review before closing this issue.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• The linked PR adds host execution, KVM, CNI, rootfs, and cleanup behavior, so the implementation needs normal PR review and proof before this issue can close.
• The issue leaves product choices open around provider category, networking mode, rootfs preparation, jailer support, and aliases.

Codex review notes: model internal, reasoning high; reviewed against d3d1891aafa7.

Label changes

Label changes:

• add P2: This is a substantial provider expansion with normal maintainer priority and limited immediate impact until an implementation merges.
• add impact:auth-provider: The issue requests a new provider selection and routing surface for Crabbox leases.
• add issue-rating: 🌊 off-meta tidepool: Current issue advisory state selects this label.
• add clawsweeper:no-new-fix-pr: Current issue advisory state selects this label.
• add clawsweeper:needs-product-decision: Current issue advisory state selects this label.

Label justifications:

• P2: This is a substantial provider expansion with normal maintainer priority and limited immediate impact until an implementation merges.
• impact:auth-provider: The issue requests a new provider selection and routing surface for Crabbox leases.

Evidence reviewed

What I checked:

• Repository policy read: AGENTS.md was read fully; its provider-neutral boundary guidance is relevant because this request adds a provider adapter and config surface. (AGENTS.md:1, d3d1891aafa7)
• Current provider registry lacks Firecracker: The built-in provider blank-import registry lists the current provider packages through xcp-ng and has no firecracker provider import on main. (internal/providers/all/all.go:4, d3d1891aafa7)
• Current docs say raw Firecracker is missing: The generated provider matrix lists Tensorlake as a delegated Firecracker-backed provider and explicitly says Crabbox does not provision raw Firecracker instances directly. (docs/providers/README.md:142, d3d1891aafa7)
• Existing Firecracker-adjacent provider is delegated-run: Tensorlake's provider spec uses family firecracker but kind delegated-run, not the requested SSH-lease raw Firecracker lifecycle. (internal/providers/tensorlake/provider.go:21, d3d1891aafa7)
• Provider contract supports the requested shape: The core provider spec has ProviderKindSSHLease and the ssh/crabbox-sync/cleanup feature vocabulary that the issue asks the new adapter to expose. (internal/cli/provider_backend.go:303, d3d1891aafa7)
• Open linked implementation PR: GitHub reports feat(firecracker): add direct microVM provider #358 as OPEN, base d3d1891, head 47db201, and closing this issue. (47db2019dda2)

Likely related people:

• Peter Steinberger: The current provider registry, provider matrix, and provider-boundary docs in this checkout are blamed to the visible release-base commit authored by Peter Steinberger. (role: provider architecture and docs provenance; confidence: medium; commits: f6b4a9765285; files: internal/providers/all/all.go, docs/providers/README.md, docs/vision.md)
• coygeek: coygeek authored the open implementation PR that GitHub reports as closing this issue, so they are the immediate contributor context for the Firecracker provider branch. (role: linked implementation candidate; confidence: medium; commits: 47db2019dda2; files: internal/providers/firecracker/, internal/cli/config.go, docs/providers/firecracker.md)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.