diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c38a2084..243e280bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -94,6 +94,7 @@ - Added `sync-plan --json` with candidate and dirty-delta sizes, configured guardrail status, deleted-path counts, and ranked file and directory hotspots for automation. Thanks @zozo123. - Added a CubeSandbox delegated-run provider with E2B-compatible lifecycle and envd execution, archive sync, CubeProxy routing, exact API-endpoint/sandbox-bound ownership claims, conflict-safe explicit adoption, and guarded cleanup. Thanks @zozo123. - Added coordinator-managed Daytona Linux leases with a Worker-held API key, exact ownership cleanup, expiring SSH-token refresh, CLI secret redaction, and production Cloudflare configuration. Thanks @vincentkoc. +- Added fal Compute direct Linux GPU SSH leases with per-lease keys, API-endpoint-scoped and claim-locked cleanup, bounded ambiguous-create recovery, and guarded zero-residue live-smoke coverage. Thanks @coygeek. ### Fixed diff --git a/README.md b/README.md index ad35b727b..921bffce5 100644 --- a/README.md +++ b/README.md @@ -226,6 +226,7 @@ from the CLI. | [Daytona](docs/providers/daytona.md) — `daytona` | Linux · direct | Daytona-managed dev sandbox over SSH. | | [Morph](docs/providers/morph.md) — `morph` | Linux · direct | Morph Cloud snapshot-backed instances over the shared SSH gateway. | | [RunPod](docs/providers/runpod.md) — `runpod` (`run-pod`, `runpodio`) | Linux · direct | RunPod GPU pods with public SSH. | +| [fal](docs/providers/fal.md) — `fal` (`fal-ai`) | Linux · direct | fal Compute GPU instances with public SSH and local-claim cleanup. | | [ASCII Box](docs/providers/ascii-box.md) — `ascii-box` (`ascii`, `asciibox`) | Linux · direct | ASCII Box Ubuntu sandboxes exposed as SSH leases. | XCP-ng itself can host Linux, Windows, and BSD guests, but Crabbox's current diff --git a/docs/features/provider-live-smoke.md b/docs/features/provider-live-smoke.md index cd71ffe7f..e2c3f0b45 100644 --- a/docs/features/provider-live-smoke.md +++ b/docs/features/provider-live-smoke.md @@ -155,7 +155,7 @@ hermetic lifecycle tests, `scripts/live-smoke.sh`, dedicated live runners, and `//go:build smoke` tests. Regenerate it with `node scripts/generate-provider-matrix.mjs`; docs CI rejects drift. -Current coverage: 75 providers; 4 with convention-named hermetic lifecycle tests, 55 with a live runner, 4 with tagged Go smoke tests, and 19 with none of those lifecycle surfaces. +Current coverage: 76 providers; 4 with convention-named hermetic lifecycle tests, 56 with a live runner, 4 with tagged Go smoke tests, and 19 with none of those lifecycle surfaces. | Provider | Hermetic lifecycle | Live runner | Tagged Go smoke | | --- | --- | --- | --- | @@ -184,6 +184,7 @@ Current coverage: 75 providers; 4 with convention-named hermetic lifecycle tests | [e2b](../providers/e2b.md) | — | matrix | — | | [exe-dev](../providers/exe-dev.md) | — | — | — | | [external](../providers/external.md) | — | matrix | — | +| [fal](../providers/fal.md) | — | dedicated + matrix | — | | [fastapi-cloud](../providers/fastapi-cloud.md) | — | — | — | | [firecracker](../providers/firecracker.md) | yes (`firecracker`) | dedicated | — | | [freestyle](../providers/freestyle.md) | — | — | — | diff --git a/docs/providers/README.md b/docs/providers/README.md index 76c1e5888..5e1337128 100644 --- a/docs/providers/README.md +++ b/docs/providers/README.md @@ -85,7 +85,7 @@ selection metadata. Regenerate it with `node scripts/generate-provider-matrix.mj `scripts/check-docs.sh` fails when provider registration, metadata, docs paths, or this generated table drift. -Current built-in surface: 75 providers (42 SSH lease, 30 delegated run, 3 service control). +Current built-in surface: 76 providers (43 SSH lease, 30 delegated run, 3 service control). Access terms: @@ -120,6 +120,7 @@ Access terms: | [e2b](e2b.md) | built-in; `delegated-run` · delegated-sandbox | No SSH; `provider-owned` · direct only; features: `url-bridge`, `run-session` | `linux`; E2B Firecracker sandbox | `provider-managed`; GPU: no | E2B; sandbox kill or expiry | Hosted ephemeral code sandbox | URL bridge is provider-specific; no normal SSH lease | | [exe-dev](exe-dev.md) (`exe`, `exedev`) | built-in; `ssh-lease` · direct-cloud | Crabbox-managed SSH; `crabbox-sync` · direct only; features: `ssh`, `crabbox-sync` | `linux`; exe.dev managed VM | `provider-managed`; GPU: unknown | exe.dev; provider lifecycle | Fast managed Linux VM exposed over SSH | Public SSH only; provider CLI owns auth | | [external](external.md) (`exec-provider`) | built-in; `ssh-lease` · external-provider | Crabbox-managed SSH; `crabbox-sync` · direct only; features: `ssh`, `crabbox-sync`, `cleanup`, `desktop`, `browser`, `code` | `linux`; Configured executable contract | `byo`; GPU: unknown | external executable; contract-defined | Private or organization-specific provider integration | Safety and semantics depend on the configured executable | +| [fal](fal.md) (`fal-ai`) | built-in; `ssh-lease` · gpu-cloud | Crabbox-managed SSH; `crabbox-sync` · direct only; features: `ssh`, `crabbox-sync`, `cleanup` | `linux`; fal Compute instance | `cloud`; GPU: yes | Crabbox; local-claim-owned instance delete | Direct GPU-backed Linux workload over SSH | Direct-only; live creates are billable and account-gated | | [fastapi-cloud](fastapi-cloud.md) (`fastapicloud`, `fastapi`) | specialized; `service-control` · service-control | SSH not applicable; `none` · direct only; features: none | `linux`; FastAPI Cloud app | `cloud`; GPU: unknown | FastAPI Cloud; not exposed | Inspecting FastAPI Cloud app deployment readiness | Cannot execute arbitrary Crabbox run commands or stop apps | | [firecracker](firecracker.md) | built-in; `ssh-lease` · self-hosted-virtualization | Crabbox-managed SSH; `crabbox-sync` · direct only; features: `ssh`, `crabbox-sync`, `cleanup` | `linux`; Firecracker microVM | `self-hosted`; GPU: no | Crabbox direct lifecycle; microVM and local artifact cleanup | Self-hosted Linux KVM host with prepared Firecracker kernel, rootfs, and CNI | Requires Linux, /dev/kvm, Firecracker assets, and a working CNI setup on the host | | [freestyle](freestyle.md) | built-in; `delegated-run` · delegated-sandbox | No SSH; `archive-sync` · direct only; features: `archive-sync`, `run-session` | `linux`; Freestyle VM | `provider-managed`; GPU: unknown | Freestyle; provider VM cleanup | Hosted delegated Linux VM execution | No Crabbox-managed SSH path | diff --git a/docs/providers/fal.md b/docs/providers/fal.md new file mode 100644 index 000000000..6a68fb0b0 --- /dev/null +++ b/docs/providers/fal.md @@ -0,0 +1,262 @@ +# fal Provider + +Read this when you are: + +- choosing `provider: fal` or the `fal-ai` alias; +- validating a direct fal Compute SSH lease; +- changing `internal/providers/fal` or the guarded live smoke. + +fal is a Linux-only **SSH lease** provider for fal Compute instances. Crabbox +creates one Compute instance through the fal API, sends a per-lease SSH public +key in the create request, records the instance in a local Crabbox claim, waits +for public SSH readiness, and then uses the normal Crabbox SSH sync, run, ssh, +stop, and cleanup paths. + +fal is **direct-only** in this release. It does not run through the coordinator, +so the local CLI must have a fal API key and direct cleanup remains the +operator's responsibility. fal Compute instances are billable while they are +running; stopping a Crabbox lease deletes the instance instead of pausing it. + +## When To Use It + +Use fal when you need a direct Linux GPU instance and local fal credentials are +acceptable. Prefer AWS, Azure, GCP, or Hetzner when you need a brokered team +path, coordinator-side credentials, or cloud-specific cost accounting. Prefer a +non-GPU direct provider when a CPU-only Linux box is enough. fal Serverless and +Model APIs are not the first Crabbox provider path here because this provider +needs a long-lived SSH-reachable box for rsync and command execution. + +## Commands + +```sh +crabbox doctor --provider fal +crabbox warmup --provider fal --fal-instance-type gpu_1x_h100_sxm5 +crabbox warmup --provider fal --type gpu_1x_h100_sxm5 +crabbox run --provider fal --fal-instance-type gpu_1x_h100_sxm5 -- go test ./... +crabbox ssh --provider fal --id my-app +crabbox list --provider fal --json +crabbox stop --provider fal my-app +crabbox cleanup --provider fal --dry-run +``` + +`--id` accepts the canonical lease id (`cbx_...`), the friendly slug, or the fal +instance id when that instance is backed by a local Crabbox fal claim. The +`fal-ai` provider alias is accepted for compatibility, but examples use the +canonical `fal` name. + +## Configuration + +```yaml +provider: fal +target: linux +fal: + apiUrl: https://api.fal.ai/v1 + instanceType: gpu_1x_h100_sxm5 + user: ubuntu + workRoot: /home/ubuntu/crabbox +``` + +Config keys under `fal:`: + +| Key | Maps to | Default | Notes | +| --- | --- | --- | --- | +| `apiUrl` | `cfg.Fal.APIURL` | `https://api.fal.ai/v1` | fal Platform API base URL. HTTPS is required unless targeting localhost for tests. | +| `instanceType` | `cfg.Fal.InstanceType` | `gpu_1x_h100_sxm5` | fal Compute instance type. | +| `sector` | `cfg.Fal.Sector` | unset | fal Compute sector; set only for supported 8× H100 multi-node instance types. | +| `user` | `cfg.Fal.User` | `ubuntu` | SSH user for the instance. | +| `workRoot` | `cfg.Fal.WorkRoot` | `/home/ubuntu/crabbox` | Remote Crabbox work root. | + +The generic `--type` flag and fal-specific `--fal-instance-type` flag both +select the Compute instance type. If both are present, explicit `--type` wins. + +Provider flags: + +```text +--fal-api-url +--fal-instance-type +--fal-sector +--fal-user +--fal-work-root +``` + +Environment overrides: + +```text +CRABBOX_FAL_KEY fal API key, preferred +FAL_KEY fal API key fallback +CRABBOX_FAL_API_URL Override the fal API URL +CRABBOX_FAL_INSTANCE_TYPE Override the Compute instance type +CRABBOX_FAL_SECTOR Override the Compute sector +CRABBOX_FAL_USER Override the SSH user +CRABBOX_FAL_WORK_ROOT Override the remote work root +``` + +Do not pass the fal API key as a command-line argument. Crabbox intentionally +has no fal key flag, so the key cannot leak through shell history or process +listings. + +Fal persists crash-recovery claims and SSH keys beneath the normal local +Crabbox state directories. On Windows clients, configure those directories +with ordinary drive or UNC paths; extended device paths such as `\\?\` and +`\??\` are rejected because they cannot provide the same stable durability +boundary. + +## Token Scope + +Crabbox sends the API key only in the `Authorization: Key ...` header to the fal +Compute endpoints used for instance list, get, create, and delete. Cross-origin +redirects are rejected before credentials or create bodies can be replayed to +another destination. + +`crabbox doctor --provider fal` is non-mutating. It checks that credentials are +available and that the Compute instance list API is reachable. Missing API +keys, authorization failures, billing, quota, and capacity issues should be +treated as account or provider readiness problems, not as proof that a live +lease can be created. + +## Lifecycle + +1. Generate a per-lease SSH key under the Crabbox testbox key directory. +2. Create one fal Compute instance with `instanceType`, `sector`, and the + per-lease public key. +3. Wait for the instance to report `ready` and expose an SSH host. +4. Wait for Crabbox SSH/bootstrap readiness over public SSH. +5. Claim the lease locally with fal instance id and SSH endpoint metadata. +6. Run normal Crabbox sync, run, ssh, status, and list workflows over SSH. +7. Delete the fal instance and remove the local claim and stored key on `stop`. + +If creation or post-create readiness becomes indeterminate, Crabbox preserves a +local recovery claim for `crabbox stop --provider fal `. A create +whose response was lost is recovered only by replaying the exact request and +idempotency key inside fal's bounded idempotency window; Crabbox then binds the +returned instance id before deletion. After that window, it fails closed and +retains the claim for manual provider reconciliation rather than risking a +second billable instance. + +If an exact instance id is known but rollback deletion fails, the retained +`rollback-cleanup` claim bypasses normal TTL/idle eligibility so the next +`cleanup` retries deletion immediately. + +## Ownership And Cleanup + +fal cleanup is intentionally local-claim based. Crabbox will not delete a fal +instance unless the local claim proves the provider, lease id, slug, target, and +fal instance id line up with the requested lease. This protects foreign fal +Compute instances and Crabbox-like resources that were not claimed by this CLI. + +Use: + +```sh +crabbox list --provider fal --json +crabbox cleanup --provider fal --dry-run +crabbox cleanup --provider fal +``` + +`cleanup --dry-run` prints what would be deleted without mutating fal or local +claims. Crabbox binds each claim to the creating credential. With that binding +intact, repeated instance absence around a complete workspace inventory proves +provider deletion and finalizes the claim and SSH key. After key rotation, +Crabbox rebinds only when the replacement credential can read the exact claimed +instance and see it in the workspace inventory; otherwise it retains the claim +for manual reconciliation. Identityless create recovery still requires the +original credential, so keep that credential active until every pending lease +has been stopped. A per-lease process and filesystem lock keeps +recovery-pending claims intact while their acquisition is still live. After an +acquiring process exits, non-kept `create-intent` claims are removed locally, +non-kept provisioning instances are deleted, and non-kept ambiguous creates are +recovered with the exact request and idempotency key, then deleted. Dry-run +reports those actions without replaying or deleting; kept claims and recoveries +beyond fal's idempotency window remain listed for explicit stop or manual +reconciliation. Unmarked transitional acquisition claims (`create-intent`, +`ambiguous-create*`, and `provisioning`) written by Crabbox versions predating +the acquisition-lifetime lock are also retained conservatively, because a +missing new-format lock cannot prove that an older acquisition process has +exited. Durable rollback and deletion markers still complete, while ordinary +ready claims retain normal keep, TTL, and idle cleanup behavior. If credentials +or the control plane are unavailable, claimed instances remain visible as +`provider-verification-unavailable` instead of being mistaken for an empty +inventory. + +## Cost Discipline + +fal Compute instances are billable while active. Keep live checks short, prefer +`--ttl` and `--idle-timeout`, use `cleanup --dry-run` before destructive +cleanup, and run live smoke only with explicit opt-in environment gates. If +cleanup fails, keep the reported slug and inspect local claims with +`crabbox list --provider fal --json` before using the fal console or API to +delete any remaining instance. + +## Guarded Live Smoke + +The repeatable live check is opt-in: + +```sh +CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=fal scripts/live-fal-smoke.sh +``` + +The script builds `bin/crabbox`, reads `CRABBOX_FAL_KEY` or `FAL_KEY` from the +environment, requires an empty Crabbox-owned fal inventory, creates a +short-lived Compute lease, waits for readiness, runs `echo ok`, verifies +`list --json`, stops the lease, runs dry-run cleanup, and verifies the provider +inventory, local claim store, and per-lease key store are empty afterward. The +lease is not kept, so TTL and idle cleanup remain available if the runner dies. +If create returns without a recoverable claim, the script requires a sustained +pre-create inventory baseline for at least nine minutes before it accepts zero +residue. Production timing overrides cannot shorten that recovery window. + +Optional live-smoke overrides: + +```text +CRABBOX_LIVE_FAL_INSTANCE_TYPE Instance type for the smoke, default gpu_1x_h100_sxm5 +CRABBOX_LIVE_FAL_SECTOR Sector for the smoke; unset by default +CRABBOX_LIVE_FAL_API_URL API URL for the smoke, default https://api.fal.ai/v1 +CRABBOX_LIVE_FAL_INVENTORY_POLL_ATTEMPTS Maximum ordinary inventory polls, default 65 +CRABBOX_LIVE_FAL_INVENTORY_POLL_SECONDS Delay between inventory polls, default 2 +CRABBOX_LIVE_FAL_AMBIGUOUS_BASELINE_OBSERVATIONS Consecutive baseline checks after an unclaimed create; values below 271 are clamped +``` + +Final classifications include: + +```text +classification=live_fal_smoke_passed +classification=environment_blocked +classification=billing_blocked +classification=quota_blocked +classification=capacity_blocked +classification=validation_failed +classification=cleanup_failed +``` + +External blockers such as missing credentials, inactive billing, quota, or +capacity are reported as classified blocked outcomes. The smoke script redacts +fal keys, SSH key payload fields, private key material, and token-like URLs from +diagnostic output. + +## Capabilities + +- **SSH** and **Crabbox sync**: yes. +- **Tailscale**: no; fal Compute exposes public SSH for this provider path. +- **Desktop / browser / code**: not advertised in this phase. +- **Cleanup**: yes, local-claim-owned fal Compute instances only. +- **Coordinator**: never; direct CLI only. + +## Gotchas + +- fal is direct-only. Coordinator secrets, team scheduling, and cost accounting + do not cover these instances. +- Running Compute instances are billable. Use short TTLs, dry-run cleanup, and + explicit live-smoke gates. +- `--fal-instance-type` must be a fal Compute instance type such as + `gpu_1x_h100_sxm5`. +- `--tailscale` and non-Linux targets are rejected for this provider. +- The first provider path is fal Compute over SSH. fal Serverless and Model APIs + are separate fal products and are not Crabbox lease backends here. + +## Related Docs + +- [fal Compute](https://fal.ai/docs/documentation/compute) +- [Create an instance API](https://fal.ai/docs/platform-apis/v1/compute/instances/create) +- [Provider reference](README.md) +- [Provider backends](../provider-backends.md) +- [Provider feature overview](../features/providers.md) +- [Operations](../operations.md) diff --git a/docs/providers/provider-metadata.json b/docs/providers/provider-metadata.json index dbabe1cfc..44304b6f7 100644 --- a/docs/providers/provider-metadata.json +++ b/docs/providers/provider-metadata.json @@ -365,6 +365,20 @@ "caveat": "Safety and semantics depend on the configured executable", "docs": "external.md" }, + "fal": { + "status": "built-in", + "category": "gpu-cloud", + "substrate": "fal Compute instance", + "location": "cloud", + "ssh": "crabbox-managed", + "sync": "crabbox-sync", + "gpu": "yes", + "lifecycle": "Crabbox", + "cleanup": "local-claim-owned instance delete", + "bestFit": "Direct GPU-backed Linux workload over SSH", + "caveat": "Direct-only; live creates are billable and account-gated", + "docs": "fal.md" + }, "fastapi-cloud": { "status": "specialized", "category": "service-control", diff --git a/internal/cli/claim.go b/internal/cli/claim.go index 612fd0b2c..d43e5a90a 100644 --- a/internal/cli/claim.go +++ b/internal/cli/claim.go @@ -306,6 +306,14 @@ func claimLeaseTargetForConfigIfUnchanged(leaseID, slug string, cfg Config, serv } func claimLeaseTargetForConfigScopeIfUnchanged(leaseID, slug string, cfg Config, providerScope string, server Server, target SSHTarget, idleTimeout time.Duration, expected leaseClaim, expectedExists bool) (leaseClaim, error) { + return claimLeaseTargetForConfigScopeIfUnchangedMode(leaseID, slug, cfg, providerScope, server, target, idleTimeout, expected, expectedExists, false) +} + +func claimLeaseTargetForConfigScopeIfUnchangedDurable(leaseID, slug string, cfg Config, providerScope string, server Server, target SSHTarget, idleTimeout time.Duration, expected leaseClaim, expectedExists bool) (leaseClaim, error) { + return claimLeaseTargetForConfigScopeIfUnchangedMode(leaseID, slug, cfg, providerScope, server, target, idleTimeout, expected, expectedExists, true) +} + +func claimLeaseTargetForConfigScopeIfUnchangedMode(leaseID, slug string, cfg Config, providerScope string, server Server, target SSHTarget, idleTimeout time.Duration, expected leaseClaim, expectedExists, durable bool) (leaseClaim, error) { provider, staticDetails := claimProviderDetailsForConfig(cfg) var updated leaseClaim err := claimLeaseForRepoProviderScopePondDetailsMetadata(leaseID, slug, provider, providerScope, cfg.Pond, staticDetails, "", idleTimeout, false, claimMetadata{ @@ -317,6 +325,7 @@ func claimLeaseTargetForConfigScopeIfUnchanged(leaseID, slug string, cfg Config, allowEmptyRepoRoot: true, guard: unchangedLeaseClaimGuard(leaseID, expected, expectedExists), result: &updated, + durable: durable, }) return updated, err } @@ -898,6 +907,11 @@ func canonicalClaimProvider(provider string) string { } func providerClaimScope(provider string, cfg Config) string { + if resolved, err := ProviderFor(provider); err == nil { + if scoped, ok := resolved.(ProviderClaimScopeProvider); ok { + return scoped.ProviderClaimScope(cfg) + } + } switch provider { case "azure": return azureLeaseClaimScope(cfg.AzureSubscription, cfg.AzureResourceGroup) @@ -906,11 +920,11 @@ func providerClaimScope(provider string, cfg Config) string { return "project:" + cfg.GCPProject } case "cubesandbox": - if endpoint := normalizedCubeSandboxClaimEndpoint(cfg.CubeSandbox.APIURL); endpoint != "" { + if endpoint := normalizedProviderClaimEndpoint(cfg.CubeSandbox.APIURL); endpoint != "" { return "endpoint:" + endpoint } case "e2b": - if endpoint := normalizedCubeSandboxClaimEndpoint(cfg.E2B.APIURL); endpoint != "" { + if endpoint := normalizedProviderClaimEndpoint(cfg.E2B.APIURL); endpoint != "" { return "endpoint:" + endpoint } case "namespace-instance": @@ -946,7 +960,7 @@ func providerClaimScope(provider string, cfg Config) string { return "" } -func normalizedCubeSandboxClaimEndpoint(raw string) string { +func normalizedProviderClaimEndpoint(raw string) string { endpoint := strings.TrimSpace(routingSafeURL(raw)) parsed, err := url.Parse(endpoint) if err != nil || parsed.Scheme == "" || parsed.Host == "" { @@ -1306,6 +1320,20 @@ func cleanupLeaseClaimIfUnchangedAfterWithSync(leaseID string, expected leaseCla if err := unchangedLeaseClaimGuard(leaseID, expected, expectedExists)(claim, exists); err != nil { return err } + if !exists { + // Windows may retain a write-through tombstone even when the source is + // absent. Prove the namespace state is durable before provider cleanup. + if err := removeControllerFile(path); err != nil && !errors.Is(err, os.ErrNotExist) { + return exit(2, "remove absent claim %s: %v", path, err) + } + if err := syncNearestExistingDirectoryWithSync(filepath.Dir(path), syncDirectory); err != nil { + return exit(2, "sync absent claim namespace %s: %v", filepath.Dir(path), err) + } + if action != nil { + return action() + } + return nil + } if action != nil { if err := action(); err != nil { return err @@ -1313,19 +1341,49 @@ func cleanupLeaseClaimIfUnchangedAfterWithSync(leaseID string, expected leaseCla } // Even when the source is absent, Windows may still have the // deterministic tombstone left by an interrupted write-through remove. - if err := removeControllerFile(path); err != nil && (exists || !errors.Is(err, os.ErrNotExist)) { + if err := removeControllerFile(path); err != nil && !errors.Is(err, os.ErrNotExist) { return exit(2, "remove claim %s: %v", path, err) } if err := syncDirectory(filepath.Dir(path)); err != nil { - if !exists && errors.Is(err, os.ErrNotExist) { - return nil - } return exit(2, "sync removed claim directory %s: %v", filepath.Dir(path), err) } return nil }) } +func finalizeAbsentLeaseClaimAfterSync(leaseID string, action func() error) error { + return finalizeAbsentLeaseClaimAfterSyncWithSync(leaseID, action, syncControllerDirectory) +} + +func finalizeAbsentLeaseClaimAfterSyncWithSync(leaseID string, action func() error, syncDirectory func(string) error) error { + path, err := leaseClaimPath(leaseID) + if err != nil { + return err + } + return withLeaseClaimLock(path, func() error { + claim, exists, err := readLeaseClaimPathWithPresence(path) + if err != nil { + return err + } + if err := validateLeaseClaimFileIdentity(leaseID, claim, exists); err != nil { + return err + } + if err := unchangedLeaseClaimGuard(leaseID, leaseClaim{}, false)(claim, exists); err != nil { + return err + } + if err := removeControllerFile(path); err != nil && !errors.Is(err, os.ErrNotExist) { + return exit(2, "remove absent claim %s: %v", path, err) + } + if err := syncNearestExistingDirectoryWithSync(filepath.Dir(path), syncDirectory); err != nil { + return exit(2, "sync absent claim namespace %s: %v", filepath.Dir(path), err) + } + if action != nil { + return action() + } + return nil + }) +} + func restoreLeaseClaimIfUnchanged(leaseID string, current, previous leaseClaim, previousExists bool) error { if !previousExists { return removeLeaseClaimIfUnchanged(leaseID, current) @@ -1532,3 +1590,14 @@ func crabboxStateDir() (string, error) { } return filepath.Join(dir, "crabbox", "state"), nil } + +func crabboxStateRootDir() (string, error) { + if dir := os.Getenv("XDG_STATE_HOME"); dir != "" { + return filepath.Clean(dir), nil + } + dir, err := os.UserConfigDir() + if err != nil { + return "", exit(2, "user state directory is unavailable") + } + return filepath.Clean(dir), nil +} diff --git a/internal/cli/claim_test.go b/internal/cli/claim_test.go index cb40fa209..eab7edcb2 100644 --- a/internal/cli/claim_test.go +++ b/internal/cli/claim_test.go @@ -46,6 +46,46 @@ func TestConfirmedAbsentClaimRemovalRequiresDirectorySyncAndRetriesAfterDeletion } } +func TestAbsentClaimCleanupSyncFailureSkipsAction(t *testing.T) { + t.Setenv("XDG_STATE_HOME", t.TempDir()) + called := false + syncErr := errors.New("claim namespace sync unavailable") + err := cleanupLeaseClaimIfUnchangedAfterWithSync("cbx_absent_cleanup_sync", leaseClaim{}, false, func() error { + called = true + return nil + }, func(string) error { return syncErr }) + if err == nil || !strings.Contains(err.Error(), syncErr.Error()) { + t.Fatalf("cleanup error=%v want %v", err, syncErr) + } + if called { + t.Fatal("cleanup action ran before durable absence was proven") + } +} + +func TestAbsentClaimCleanupSyncsBeforeAction(t *testing.T) { + t.Setenv("XDG_STATE_HOME", t.TempDir()) + const leaseID = "cbx_absent_cleanup_order" + path, err := leaseClaimPath(leaseID) + if err != nil { + t.Fatal(err) + } + var events []string + err = cleanupLeaseClaimIfUnchangedAfterWithSync(leaseID, leaseClaim{}, false, func() error { + events = append(events, "action") + return nil + }, func(dir string) error { + events = append(events, "sync:"+filepath.Clean(dir)) + return nil + }) + if err != nil { + t.Fatal(err) + } + want := []string{"sync:" + filepath.Clean(filepath.Dir(filepath.Dir(path))), "action"} + if !reflect.DeepEqual(events, want) { + t.Fatalf("events=%q want %q", events, want) + } +} + func TestWriteLeaseClaimAtomicWithSyncPropagatesDirectorySyncFailure(t *testing.T) { t.Setenv("XDG_STATE_HOME", t.TempDir()) const leaseID = "cbx_sync_failure" @@ -150,6 +190,285 @@ func TestDurableGuardedClaimWritePropagatesRequiredBoundarySyncFailure(t *testin } } +func TestEnsurePrivateDirectoryDurableSyncsFreshAncestorChain(t *testing.T) { + base := t.TempDir() + dir := filepath.Join(base, "new-state", "crabbox", "claim-locks") + var synced []string + if err := ensurePrivateDirectoryDurableWithinWithSync(dir, base, func(path string) error { + synced = append(synced, filepath.Clean(path)) + return nil + }); err != nil { + t.Fatal(err) + } + want := []string{ + filepath.Join(base, "new-state", "crabbox"), + filepath.Join(base, "new-state"), + base, + } + if !reflect.DeepEqual(synced, want) { + t.Fatalf("synced=%q want=%q", synced, want) + } +} + +func TestEnsurePrivateDirectoryDurableRejectsFilesystemRootBoundary(t *testing.T) { + base := t.TempDir() + root := filepath.VolumeName(base) + string(filepath.Separator) + dir := filepath.Join(root, "crabbox-missing-"+filepath.Base(base), "private") + err := ensurePrivateDirectoryDurableWithinWithSync(dir, root, func(string) error { + t.Fatal("filesystem root boundary must fail before syncing") + return nil + }) + if err == nil || !strings.Contains(err.Error(), "not a safe existing directory") { + t.Fatalf("error=%v", err) + } +} + +func TestEnsurePrivateDirectoryDurableRetryRepeatsFixedBoundary(t *testing.T) { + base := t.TempDir() + dir := filepath.Join(base, "new-state", "crabbox", "claim-locks") + syncErr := errors.New("boundary sync unavailable") + err := ensurePrivateDirectoryDurableWithinWithSync(dir, base, func(path string) error { + if filepath.Clean(path) == filepath.Clean(base) { + return syncErr + } + return nil + }) + if err == nil || !strings.Contains(err.Error(), syncErr.Error()) { + t.Fatalf("first attempt error=%v", err) + } + var synced []string + if err := ensurePrivateDirectoryDurableWithinWithSync(dir, base, func(path string) error { + synced = append(synced, filepath.Clean(path)) + return nil + }); err != nil { + t.Fatal(err) + } + if len(synced) == 0 || synced[len(synced)-1] != filepath.Clean(base) { + t.Fatalf("retry syncs=%q want original boundary %q", synced, base) + } +} + +func TestEnsurePrivateDirectoryDurableLogicalRootRetryKeepsOriginalBoundary(t *testing.T) { + base := t.TempDir() + logicalRoot := filepath.Join(base, "new-state") + dir := filepath.Join(logicalRoot, "crabbox", "claim-locks") + boundary, err := privateDirectoryDurabilityBoundary(dir, logicalRoot) + if err != nil { + t.Fatal(err) + } + syncErr := errors.New("stable boundary sync unavailable") + err = ensurePrivateDirectoryDurableWithSync(dir, logicalRoot, func(path string) error { + if filepath.Clean(path) == boundary { + return syncErr + } + return nil + }) + if err == nil || !strings.Contains(err.Error(), syncErr.Error()) { + t.Fatalf("first attempt error=%v", err) + } + var synced []string + if err := ensurePrivateDirectoryDurableWithSync(dir, logicalRoot, func(path string) error { + synced = append(synced, filepath.Clean(path)) + return nil + }); err != nil { + t.Fatal(err) + } + if len(synced) == 0 || synced[len(synced)-1] != boundary { + t.Fatalf("retry syncs=%q want original boundary %q", synced, boundary) + } +} + +func TestEnsurePrivateDirectoryDurableCreatesCustomXDGRootFromStableTopLevel(t *testing.T) { + base := t.TempDir() + alternateTemp := filepath.Join(base, "alternate-temp") + alternateHome := filepath.Join(base, "alternate-home") + if err := os.MkdirAll(alternateTemp, 0o700); err != nil { + t.Fatal(err) + } + t.Setenv("TMPDIR", alternateTemp) + t.Setenv("TEMP", alternateTemp) + t.Setenv("TMP", alternateTemp) + t.Setenv("HOME", alternateHome) + logicalRoot := filepath.Join(base, "custom-xdg", "state") + dir := filepath.Join(logicalRoot, "crabbox", "claims") + boundary, err := privateDirectoryDurabilityBoundary(dir, logicalRoot) + if err != nil { + t.Fatal(err) + } + if boundary == filepath.Clean(logicalRoot) || filepath.Dir(boundary) == boundary { + t.Fatalf("boundary=%q is not a stable non-root ancestor", boundary) + } + syncErr := errors.New("top-level sync unavailable") + err = ensurePrivateDirectoryDurableWithSync(dir, logicalRoot, func(path string) error { + if filepath.Clean(path) == boundary { + return syncErr + } + return nil + }) + if err == nil || !strings.Contains(err.Error(), syncErr.Error()) { + t.Fatalf("first attempt error=%v", err) + } + var synced []string + if err := ensurePrivateDirectoryDurableWithSync(dir, logicalRoot, func(path string) error { + synced = append(synced, filepath.Clean(path)) + return nil + }); err != nil { + t.Fatal(err) + } + if len(synced) == 0 || synced[len(synced)-1] != boundary { + t.Fatalf("retry syncs=%q want stable top-level boundary %q", synced, boundary) + } +} + +func TestEnsureCrabboxClaimNamespaceDurableRejectsRelativeXDGRoot(t *testing.T) { + t.Setenv("XDG_STATE_HOME", filepath.Join("relative-home", "state")) + called := false + err := ensureCrabboxClaimNamespaceDurableWithSync(func(string) error { + called = true + return nil + }) + if err == nil || !strings.Contains(err.Error(), "must be absolute") { + t.Fatalf("error=%v", err) + } + if called { + t.Fatal("relative state root reached directory sync") + } +} + +func TestIsWindowsDevicePathRecognizesSeparatorVariants(t *testing.T) { + for _, path := range []string{ + `\\?\UNC\server\share\state`, + `//?/UNC/server/share/state`, + `/\?\UNC/server/share/state`, + `\??\UNC\server\share\state`, + `/??/UNC/server/share/state`, + `\\.\C:\state`, + `//./UNC/server/share/state`, + } { + t.Run(path, func(t *testing.T) { + if !isWindowsDevicePath(path) { + t.Fatalf("isWindowsDevicePath(%q)=false", path) + } + if filepath.Separator == '\\' { + _, err := privateDirectoryDurabilityBoundary(path+`\crabbox\claims`, path) + if err == nil || !strings.Contains(err.Error(), "Windows device paths are not supported") { + t.Fatalf("error=%v", err) + } + } + }) + } + if filepath.Separator == '/' { + _, err := privateDirectoryDurabilityBoundary(`/??/state/crabbox/claims`, `/??/state`) + if err != nil && strings.Contains(err.Error(), "Windows device paths are not supported") { + t.Fatalf("valid POSIX path was classified as a Windows device path: %v", err) + } + } +} + +func TestEnsureCrabboxClaimNamespaceDurableSyncsPreexistingClaimsDirectory(t *testing.T) { + base := t.TempDir() + stateRoot := filepath.Join(base, "fresh", "state") + t.Setenv("XDG_STATE_HOME", stateRoot) + stateDir, err := crabboxStateDir() + if err != nil { + t.Fatal(err) + } + claimDir := filepath.Join(stateDir, "claims") + if err := os.MkdirAll(claimDir, 0o700); err != nil { + t.Fatal(err) + } + boundary, err := privateDirectoryDurabilityBoundary(claimDir, stateRoot) + if err != nil { + t.Fatal(err) + } + var synced []string + if err := ensureCrabboxClaimNamespaceDurableWithSync(func(path string) error { + synced = append(synced, filepath.Clean(path)) + return nil + }); err != nil { + t.Fatal(err) + } + if info, err := os.Stat(claimDir); err != nil || !info.IsDir() { + t.Fatalf("claim namespace stat=%v info=%v", err, info) + } + if len(synced) == 0 || synced[0] != filepath.Clean(stateDir) { + t.Fatalf("claim namespace syncs=%q want state parent first %q", synced, stateDir) + } + if len(synced) == 0 || synced[len(synced)-1] != boundary { + t.Fatalf("state namespace syncs=%q want stable boundary %q", synced, boundary) + } +} + +func TestFinalizeAbsentLeaseClaimSyncFailureSkipsAction(t *testing.T) { + t.Setenv("XDG_STATE_HOME", t.TempDir()) + called := false + syncErr := errors.New("directory sync unavailable") + err := finalizeAbsentLeaseClaimAfterSyncWithSync("cbx_finalize_absent", func() error { + called = true + return nil + }, func(string) error { return syncErr }) + if err == nil || !strings.Contains(err.Error(), syncErr.Error()) { + t.Fatalf("finalize error=%v", err) + } + if called { + t.Fatal("action ran before durable absence was proven") + } +} + +func TestFinalizeAbsentLeaseClaimSyncsNearestExistingAncestor(t *testing.T) { + t.Setenv("XDG_STATE_HOME", t.TempDir()) + const leaseID = "cbx_finalize_missing_directory" + path, err := leaseClaimPath(leaseID) + if err != nil { + t.Fatal(err) + } + called := false + var synced string + err = finalizeAbsentLeaseClaimAfterSyncWithSync(leaseID, func() error { + called = true + return nil + }, func(dir string) error { + synced = filepath.Clean(dir) + return nil + }) + if err != nil { + t.Fatal(err) + } + if !called { + t.Fatal("action did not run after durable absence") + } + want := filepath.Clean(filepath.Dir(filepath.Dir(path))) + if synced != want { + t.Fatalf("synced=%q want nearest existing ancestor %q", synced, want) + } +} + +func TestFinalizeAbsentLeaseClaimPresentClaimSkipsAction(t *testing.T) { + t.Setenv("XDG_STATE_HOME", t.TempDir()) + const leaseID = "cbx_finalize_present" + if err := claimLeaseForRepoProvider(leaseID, "finalize-present", "external", "/repo", time.Minute, false); err != nil { + t.Fatal(err) + } + called := false + synced := false + err := finalizeAbsentLeaseClaimAfterSyncWithSync(leaseID, func() error { + called = true + return nil + }, func(string) error { + synced = true + return nil + }) + if err == nil || !strings.Contains(err.Error(), "claim changed") { + t.Fatalf("finalize error=%v", err) + } + if called || synced { + t.Fatalf("called=%v synced=%v", called, synced) + } + if _, exists, readErr := readLeaseClaimWithPresence(leaseID); readErr != nil || !exists { + t.Fatalf("claim exists=%v err=%v", exists, readErr) + } +} + func TestDurableGuardedClaimWriteHoldsLockThroughDirectorySync(t *testing.T) { t.Setenv("XDG_STATE_HOME", t.TempDir()) const leaseID = "cbx_durable_lock" diff --git a/internal/cli/config.go b/internal/cli/config.go index 59de2ecbf..8d776f422 100644 --- a/internal/cli/config.go +++ b/internal/cli/config.go @@ -181,6 +181,7 @@ type Config struct { Runpod RunpodConfig Vast VastConfig vastWorkRootExplicit bool + Fal FalConfig NvidiaBrev NvidiaBrevConfig nvidiaBrevWorkRootExplicit bool Hostinger HostingerConfig @@ -710,6 +711,20 @@ type VastConfig struct { ReleaseAction string } +type FalConfig struct { + APIKey string + APIURL string + InstanceType string + Sector string + User string + WorkRoot string +} + +const ( + defaultFalUser = "ubuntu" + defaultFalWorkRoot = "/home/ubuntu/crabbox" +) + // NvidiaBrevConfig is intentionally non-secret. Authentication stays in the // NVIDIA Brev CLI's own credential store and is never accepted as Crabbox // config or argv. @@ -2044,6 +2059,75 @@ func applyProviderConfigDefaults(cfg *Config) error { normalizeTargetConfig(cfg) return validateTargetConfig(*cfg) } + if cfg.Provider == "fal" { + for _, field := range []struct { + name string + value string + }{ + {name: "fal.apiUrl", value: cfg.Fal.APIURL}, + {name: "fal.instanceType", value: cfg.Fal.InstanceType}, + {name: "fal.user", value: cfg.Fal.User}, + {name: "fal.workRoot", value: cfg.Fal.WorkRoot}, + {name: "ssh.user", value: cfg.explicitSSHUser}, + {name: "workRoot", value: cfg.explicitWorkRoot}, + } { + if field.value != "" && strings.TrimSpace(field.value) == "" { + return exit(2, "%s must not be blank", field.name) + } + } + cfg.Fal.APIURL = strings.TrimSpace(cfg.Fal.APIURL) + cfg.Fal.InstanceType = strings.TrimSpace(cfg.Fal.InstanceType) + cfg.Fal.Sector = strings.TrimSpace(cfg.Fal.Sector) + cfg.Fal.User = strings.TrimSpace(cfg.Fal.User) + cfg.Fal.WorkRoot = strings.TrimSpace(cfg.Fal.WorkRoot) + cfg.explicitSSHUser = strings.TrimSpace(cfg.explicitSSHUser) + cfg.explicitWorkRoot = strings.TrimSpace(cfg.explicitWorkRoot) + if cfg.ServerTypeExplicit && strings.TrimSpace(cfg.ServerType) != "" { + cfg.Fal.InstanceType = strings.TrimSpace(cfg.ServerType) + } + if cfg.Fal.APIURL == "" { + cfg.Fal.APIURL = "https://api.fal.ai/v1" + } + if cfg.Fal.InstanceType == "" { + cfg.Fal.InstanceType = "gpu_1x_h100_sxm5" + } + if cfg.Fal.User == "" { + cfg.Fal.User = defaultFalUser + } + if cfg.Fal.WorkRoot == "" { + cfg.Fal.WorkRoot = defaultFalWorkRoot + } + if !IsTargetExplicit(cfg) { + cfg.TargetOS = targetLinux + } + if cfg.explicitWindowsMode != "" { + cfg.WindowsMode = cfg.explicitWindowsMode + } else { + cfg.WindowsMode = windowsModeNormal + } + if cfg.explicitWorkRoot != "" { + cfg.WorkRoot = cfg.explicitWorkRoot + if cfg.Fal.WorkRoot == "" || cfg.Fal.WorkRoot == defaultFalWorkRoot { + cfg.Fal.WorkRoot = cfg.explicitWorkRoot + } + } else { + cfg.WorkRoot = cfg.Fal.WorkRoot + } + if cfg.explicitSSHUser != "" { + cfg.SSHUser = cfg.explicitSSHUser + } else { + cfg.SSHUser = cfg.Fal.User + } + if cfg.explicitSSHPort != "" { + cfg.SSHPort = cfg.explicitSSHPort + } else { + cfg.SSHPort = "22" + } + cfg.SSHFallbackPorts = nil + cfg.ServerType = cfg.Fal.InstanceType + normalizeTargetConfig(cfg) + return validateTargetConfig(*cfg) + } if cfg.Provider == "hyperv" { if !IsTargetExplicit(cfg) { cfg.TargetOS = targetWindows @@ -2918,6 +3002,12 @@ func baseConfig() Config { WorkRoot: defaultPOSIXWorkRoot, ReleaseAction: "destroy", }, + Fal: FalConfig{ + APIURL: "https://api.fal.ai/v1", + InstanceType: "gpu_1x_h100_sxm5", + User: defaultFalUser, + WorkRoot: defaultFalWorkRoot, + }, NvidiaBrev: NvidiaBrevConfig{ CLI: "brev", GPUName: "A100", @@ -3253,6 +3343,7 @@ type fileConfig struct { UnikraftCloud *fileUnikraftCloudConfig `yaml:"unikraftCloud,omitempty"` Runpod *fileRunpodConfig `yaml:"runpod,omitempty"` Vast *fileVastConfig `yaml:"vast,omitempty"` + Fal *fileFalConfig `yaml:"fal,omitempty"` NvidiaBrev *fileNvidiaBrevConfig `yaml:"nvidiaBrev,omitempty"` Hostinger *fileHostingerConfig `yaml:"hostinger,omitempty"` Wandb *fileWandbConfig `yaml:"wandb,omitempty"` @@ -3894,6 +3985,14 @@ type fileVastConfig struct { ReleaseAction string `yaml:"releaseAction,omitempty"` } +type fileFalConfig struct { + APIURL string `yaml:"apiUrl,omitempty"` + InstanceType string `yaml:"instanceType,omitempty"` + Sector string `yaml:"sector,omitempty"` + User string `yaml:"user,omitempty"` + WorkRoot string `yaml:"workRoot,omitempty"` +} + type fileNvidiaBrevConfig struct { CLI string `yaml:"cli,omitempty"` Org string `yaml:"org,omitempty"` @@ -6385,6 +6484,24 @@ func applyFileConfigWithTrust(cfg *Config, file fileConfig, trusted bool) error MarkDeleteOnReleaseExplicit(cfg, "vast") } } + if file.Fal != nil { + if file.Fal.APIURL != "" { + cfg.Fal.APIURL = file.Fal.APIURL + cfg.credentialProvenance.falAPIURL = credentialSource + } + if file.Fal.InstanceType != "" { + cfg.Fal.InstanceType = file.Fal.InstanceType + } + if file.Fal.Sector != "" { + cfg.Fal.Sector = file.Fal.Sector + } + if file.Fal.User != "" { + cfg.Fal.User = file.Fal.User + } + if file.Fal.WorkRoot != "" { + cfg.Fal.WorkRoot = file.Fal.WorkRoot + } + } if file.NvidiaBrev != nil { if trusted && file.NvidiaBrev.CLI != "" { cfg.NvidiaBrev.CLI = file.NvidiaBrev.CLI @@ -8593,6 +8710,18 @@ func applyEnv(cfg *Config) error { cfg.Vast.ReleaseAction = value MarkDeleteOnReleaseExplicit(cfg, "vast") } + if value, ok := firstNonEmptyEnv("CRABBOX_FAL_KEY", "FAL_KEY"); ok { + cfg.Fal.APIKey = value + cfg.credentialProvenance.falAPIKey = credentialSourceEnvironment + } + if value, ok := firstNonEmptyEnv("CRABBOX_FAL_API_URL"); ok { + cfg.Fal.APIURL = value + cfg.credentialProvenance.falAPIURL = credentialSourceEnvironment + } + cfg.Fal.InstanceType = getenv("CRABBOX_FAL_INSTANCE_TYPE", cfg.Fal.InstanceType) + cfg.Fal.Sector = getenv("CRABBOX_FAL_SECTOR", cfg.Fal.Sector) + cfg.Fal.User = getenv("CRABBOX_FAL_USER", cfg.Fal.User) + cfg.Fal.WorkRoot = getenv("CRABBOX_FAL_WORK_ROOT", cfg.Fal.WorkRoot) cfg.NvidiaBrev.CLI = getenv("CRABBOX_NVIDIA_BREV_CLI", cfg.NvidiaBrev.CLI) cfg.NvidiaBrev.Org = getenv("CRABBOX_NVIDIA_BREV_ORG", cfg.NvidiaBrev.Org) cfg.NvidiaBrev.Type = getenv("CRABBOX_NVIDIA_BREV_TYPE", cfg.NvidiaBrev.Type) @@ -9372,6 +9501,9 @@ func serverTypeForConfig(cfg Config) string { if cfg.Provider == "cloudflare" { return cloudflareContainerInstanceTypeForClass(cfg.Class) } + if cfg.Provider == "fal" { + return blank(cfg.Fal.InstanceType, "gpu_1x_h100_sxm5") + } if cfg.Provider == "aws" { return awsInstanceTypeCandidatesForConfig(cfg)[0] } @@ -9424,6 +9556,9 @@ func serverTypeForProviderClass(provider, class string) string { if provider == "cloudflare" { return cloudflareContainerInstanceTypeForClass(class) } + if provider == "fal" { + return "gpu_1x_h100_sxm5" + } if provider == "aws" { return awsInstanceTypeCandidatesForClass(class)[0] } diff --git a/internal/cli/config_cmd.go b/internal/cli/config_cmd.go index 40dd69669..7f37e3834 100644 --- a/internal/cli/config_cmd.go +++ b/internal/cli/config_cmd.go @@ -101,6 +101,19 @@ func effectiveConfigForShow(cfg Config) Config { } cfg.SSHFallbackPorts = nil } + if cfg.Provider == "fal" { + base := baseConfig() + if !IsSSHUserExplicit(&cfg) && (cfg.SSHUser == "" || cfg.SSHUser == base.SSHUser) { + cfg.SSHUser = blank(cfg.Fal.User, "root") + } + if !IsSSHPortExplicit(&cfg) && (cfg.SSHPort == "" || cfg.SSHPort == base.SSHPort) { + cfg.SSHPort = "22" + } + if !IsWorkRootExplicit(&cfg) { + cfg.WorkRoot = blank(cfg.Fal.WorkRoot, defaultPOSIXWorkRoot) + } + cfg.SSHFallbackPorts = nil + } if cfg.Provider == "scaleway" { base := baseConfig() if !IsSSHUserExplicit(&cfg) && (cfg.SSHUser == "" || cfg.SSHUser == base.SSHUser) { @@ -246,6 +259,14 @@ func configShowView(cfg Config) map[string]any { "filesystemMounts": cfg.Lambda.FilesystemMounts, "auth": lambdaAuthState(), }, + "fal": map[string]any{ + "apiUrl": redactedConfigURL(cfg.Fal.APIURL), + "instanceType": cfg.Fal.InstanceType, + "sector": cfg.Fal.Sector, + "user": cfg.Fal.User, + "workRoot": cfg.Fal.WorkRoot, + "auth": falAuthState(cfg), + }, "nvidiaBrev": map[string]any{ "cli": cfg.NvidiaBrev.CLI, "auth": "cli", @@ -773,6 +794,7 @@ func writeConfigShowText(w io.Writer, cfg Config) { fmt.Fprintf(w, "linode region=%s image=%s type=%s firewall=%s ssh_cidrs=%s\n", cfg.Linode.Region, cfg.Linode.Image, cfg.Linode.Type, blank(cfg.Linode.FirewallID, "-"), blank(strings.Join(cfg.Linode.SSHCIDRs, ","), "-")) fmt.Fprintf(w, "lambda region=%s type=%s image=%s image_family=%s firewall_ruleset=%s ssh_cidrs=%s filesystems=%s mounts=%d auth=%s\n", cfg.Lambda.Region, cfg.Lambda.Type, blank(cfg.Lambda.Image, "-"), blank(cfg.Lambda.ImageFamily, "-"), blank(cfg.Lambda.FirewallRuleset, "-"), blank(strings.Join(cfg.Lambda.SSHCIDRs, ","), "-"), blank(strings.Join(cfg.Lambda.FilesystemNames, ","), "-"), len(cfg.Lambda.FilesystemMounts), lambdaAuthState()) fmt.Fprintf(w, "vast api_url=%s instance_type=%s gpu_name=%s gpu_count=%d image=%s template_id=%s runtype=%s disk_gb=%d max_dph_total=%.4g min_reliability=%.4g order=%s user=%s work_root=%s release_action=%s auth=%s\n", blank(redactedConfigURL(cfg.Vast.APIURL), "-"), blank(cfg.Vast.InstanceType, "-"), blank(cfg.Vast.GPUName, "-"), cfg.Vast.GPUCount, blank(cfg.Vast.Image, "-"), blank(cfg.Vast.TemplateID, "-"), blank(cfg.Vast.Runtype, "-"), cfg.Vast.DiskGB, cfg.Vast.MaxDphTotal, cfg.Vast.MinReliability, blank(cfg.Vast.Order, "-"), blank(cfg.Vast.User, "-"), blank(cfg.Vast.WorkRoot, "-"), blank(cfg.Vast.ReleaseAction, "-"), tokenState(cfg.Vast.APIKey)) + fmt.Fprintf(w, "fal api_url=%s instance_type=%s sector=%s user=%s work_root=%s auth=%s\n", blank(redactedConfigURL(cfg.Fal.APIURL), "-"), blank(cfg.Fal.InstanceType, "-"), blank(cfg.Fal.Sector, "-"), blank(cfg.Fal.User, "-"), blank(cfg.Fal.WorkRoot, "-"), falAuthState(cfg)) fmt.Fprintf(w, "nvidia_brev cli=%s org=%s type=%s gpu_name=%s provider=%s mode=%s launchable=%s startup_script=%s release_action=%s target=%s user=%s work_root=%s auth=cli\n", blank(cfg.NvidiaBrev.CLI, "-"), blank(cfg.NvidiaBrev.Org, "-"), blank(cfg.NvidiaBrev.Type, "-"), blank(cfg.NvidiaBrev.GPUName, "-"), blank(cfg.NvidiaBrev.Provider, "-"), blank(cfg.NvidiaBrev.Mode, "-"), blank(cfg.NvidiaBrev.Launchable, "-"), blank(cfg.NvidiaBrev.StartupScript, "-"), blank(cfg.NvidiaBrev.ReleaseAction, "-"), blank(cfg.NvidiaBrev.Target, "-"), blank(cfg.NvidiaBrev.User, "-"), blank(cfg.NvidiaBrev.WorkRoot, "-")) fmt.Fprintf(w, "nebius cli=%s profile=%s parent_id=%s subnet_id=%s platform=%s preset=%s image_family=%s disk_type=%s disk_size_gib=%d user=%s public_ip=%s security_group_ids=%s service_account_id=%s recovery_policy=%s auth=cli\n", blank(cfg.Nebius.CLI, "-"), blank(cfg.Nebius.Profile, "-"), blank(cfg.Nebius.ParentID, "-"), blank(cfg.Nebius.SubnetID, "-"), blank(cfg.Nebius.Platform, "-"), blank(cfg.Nebius.Preset, "-"), blank(cfg.Nebius.ImageFamily, "-"), blank(cfg.Nebius.DiskType, "-"), cfg.Nebius.DiskSizeGiB, blank(cfg.Nebius.User, "-"), blank(cfg.Nebius.PublicIP, "-"), blank(strings.Join(cfg.Nebius.SecurityGroupIDs, ","), "-"), blank(cfg.Nebius.ServiceAccountID, "-"), blank(cfg.Nebius.RecoveryPolicy, "-")) fmt.Fprintf(w, "hostinger api_url=%s item_id=%s payment_method_id=%s template_id=%s data_center_id=%s hostname_prefix=%s user=%s work_root=%s allow_purchase=%t release_action=%s auth=%s\n", blank(redactedConfigURL(cfg.Hostinger.APIURL), "-"), blank(cfg.Hostinger.ItemID, "-"), blank(cfg.Hostinger.PaymentMethodID, "-"), blank(cfg.Hostinger.TemplateID, "-"), blank(cfg.Hostinger.DataCenterID, "-"), blank(cfg.Hostinger.HostnamePrefix, "-"), blank(cfg.Hostinger.User, "-"), blank(cfg.Hostinger.WorkRoot, "-"), cfg.Hostinger.AllowPurchase, blank(cfg.Hostinger.ReleaseAction, "-"), tokenState(cfg.Hostinger.APIToken)) @@ -822,6 +844,13 @@ func lambdaAuthState() string { return "missing" } +func falAuthState(cfg Config) string { + if strings.TrimSpace(cfg.Fal.APIKey) != "" { + return "env" + } + return "missing" +} + // sanitizedMalformedConfigURL strips any userinfo from a malformed URL so // url.Parse error messages and downstream diagnostics cannot echo the // original credentials. diff --git a/internal/cli/config_cmd_test.go b/internal/cli/config_cmd_test.go index f7136691b..d66cd2c71 100644 --- a/internal/cli/config_cmd_test.go +++ b/internal/cli/config_cmd_test.go @@ -167,6 +167,119 @@ func TestConfigShowIncludesFirecrackerConfig(t *testing.T) { } } +func TestConfigShowIncludesFalConfigWithoutCredentialDetails(t *testing.T) { + cfg := baseConfig() + cfg.Provider = "fal" + cfg.Fal.APIKey = "secret-fal-key" + cfg.Fal.APIURL = "https://api.fal.example.test/v1" + cfg.Fal.InstanceType = "gpu_8x_h100_sxm5" + cfg.Fal.Sector = "sector_2" + cfg.Fal.User = "root" + cfg.Fal.WorkRoot = "/work/fal" + if err := applyProviderConfigDefaults(&cfg); err != nil { + t.Fatal(err) + } + + view := configShowView(cfg) + fal, ok := view["fal"].(map[string]any) + if !ok || fal["apiUrl"] != "https://api.fal.example.test/v1" || + fal["instanceType"] != "gpu_8x_h100_sxm5" || + fal["sector"] != "sector_2" || + fal["user"] != "root" || + fal["workRoot"] != "/work/fal" || + fal["auth"] != "env" { + t.Fatalf("fal view=%#v", fal) + } + data, err := json.Marshal(view) + if err != nil { + t.Fatal(err) + } + var text bytes.Buffer + writeConfigShowText(&text, cfg) + for name, output := range map[string]string{"json": string(data), "text": text.String()} { + if strings.Contains(output, "secret-fal-key") || strings.Contains(output, "FAL_KEY") || strings.Contains(output, "CRABBOX_FAL_KEY") { + t.Fatalf("%s output leaked fal credential details: %s", name, output) + } + for _, want := range []string{ + "https://api.fal.example.test/v1", + "gpu_8x_h100_sxm5", + "sector_2", + "/work/fal", + "auth", + "env", + } { + if !strings.Contains(output, want) { + t.Fatalf("%s output missing %q: %s", name, want, output) + } + } + } +} + +func TestConfigShowRejectsBlankFalIdentityFields(t *testing.T) { + for _, tc := range []struct { + name string + config string + want string + }{ + {name: "api url", config: "fal:\n apiUrl: ' '\n", want: "fal.apiUrl must not be blank"}, + {name: "user", config: "fal:\n user: ' '\n", want: "fal.user must not be blank"}, + {name: "ssh user", config: "ssh:\n user: ' '\n", want: "ssh.user must not be blank"}, + {name: "work root", config: "workRoot: ' '\n", want: "workRoot must not be blank"}, + } { + t.Run(tc.name, func(t *testing.T) { + clearConfigEnv(t) + home := t.TempDir() + configPath := filepath.Join(home, "config.yaml") + t.Setenv("HOME", home) + t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config")) + t.Setenv("CRABBOX_CONFIG", configPath) + config := "provider: fal\n" + tc.config + if err := os.WriteFile(configPath, []byte(config), 0o600); err != nil { + t.Fatal(err) + } + app := App{Stdout: &bytes.Buffer{}, Stderr: &bytes.Buffer{}} + for _, args := range [][]string{nil, {"--json"}} { + if err := app.configShow(args); err == nil || !strings.Contains(err.Error(), tc.want) { + t.Fatalf("configShow(%v) err=%v want %q", args, err, tc.want) + } + } + }) + } +} + +func TestConfigShowNormalizesFalIdentityFields(t *testing.T) { + clearConfigEnv(t) + home := t.TempDir() + configPath := filepath.Join(home, "config.yaml") + t.Setenv("HOME", home) + t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config")) + t.Setenv("CRABBOX_CONFIG", configPath) + config := "provider: fal\nworkRoot: ' /srv/crabbox '\nssh:\n user: ' runner '\nfal:\n apiUrl: ' https://api.fal.example.test/v1 '\n user: ' root '\n" + if err := os.WriteFile(configPath, []byte(config), 0o600); err != nil { + t.Fatal(err) + } + var stdout bytes.Buffer + app := App{Stdout: &stdout, Stderr: &bytes.Buffer{}} + if err := app.configShow([]string{"--json"}); err != nil { + t.Fatal(err) + } + var got struct { + SSHUser string `json:"sshUser"` + WorkRoot string `json:"workRoot"` + Fal struct { + APIURL string `json:"apiUrl"` + User string `json:"user"` + WorkRoot string `json:"workRoot"` + } `json:"fal"` + } + if err := json.Unmarshal(stdout.Bytes(), &got); err != nil { + t.Fatal(err) + } + if got.Fal.APIURL != "https://api.fal.example.test/v1" || got.Fal.User != "root" || got.Fal.WorkRoot != "/srv/crabbox" || got.SSHUser != "runner" || got.WorkRoot != "/srv/crabbox" { + t.Fatalf("normalized config=%#v", got) + } +} + func TestConfigSetBrokerRegisteredMode(t *testing.T) { clearConfigEnv(t) home := t.TempDir() diff --git a/internal/cli/config_test.go b/internal/cli/config_test.go index 510fe7e4b..7c3d2b5c3 100644 --- a/internal/cli/config_test.go +++ b/internal/cli/config_test.go @@ -528,6 +528,13 @@ func clearConfigEnv(t *testing.T) { "CRABBOX_VAST_USER", "CRABBOX_VAST_WORK_ROOT", "CRABBOX_VAST_RELEASE_ACTION", + "CRABBOX_FAL_KEY", + "FAL_KEY", + "CRABBOX_FAL_API_URL", + "CRABBOX_FAL_INSTANCE_TYPE", + "CRABBOX_FAL_SECTOR", + "CRABBOX_FAL_USER", + "CRABBOX_FAL_WORK_ROOT", "CRABBOX_NVIDIA_BREV_CLI", "CRABBOX_NVIDIA_BREV_ORG", "CRABBOX_NVIDIA_BREV_TYPE", @@ -2201,6 +2208,103 @@ func TestLinodeConfigFileAndEnv(t *testing.T) { } } +func TestFalProviderConfigDefaultsFileAndEnv(t *testing.T) { + clearConfigEnv(t) + cfg := baseConfig() + cfg.Provider = "fal" + if err := applyFileConfig(&cfg, fileConfig{Fal: &fileFalConfig{ + APIURL: "https://file-api.example.test/v1", + InstanceType: "gpu_8x_h100_sxm5", + Sector: "sector_2", + User: "ubuntu", + WorkRoot: "/file/fal", + }}); err != nil { + t.Fatal(err) + } + if cfg.Fal.APIURL != "https://file-api.example.test/v1" || + cfg.Fal.InstanceType != "gpu_8x_h100_sxm5" || + cfg.Fal.Sector != "sector_2" || + cfg.Fal.User != "ubuntu" || + cfg.Fal.WorkRoot != "/file/fal" || + cfg.Fal.APIKey != "" { + t.Fatalf("file fal config not applied or accepted a secret: %#v", cfg.Fal) + } + + t.Setenv("FAL_KEY", "fallback-key") + t.Setenv("CRABBOX_FAL_KEY", "crabbox-key") + t.Setenv("CRABBOX_FAL_API_URL", "https://env-api.example.test/v1") + t.Setenv("CRABBOX_FAL_INSTANCE_TYPE", "gpu_1x_h100_sxm5") + t.Setenv("CRABBOX_FAL_SECTOR", "sector_3") + t.Setenv("CRABBOX_FAL_USER", "root") + t.Setenv("CRABBOX_FAL_WORK_ROOT", "/env/fal") + if err := applyEnv(&cfg); err != nil { + t.Fatal(err) + } + if cfg.Fal.APIKey != "crabbox-key" || + cfg.Fal.APIURL != "https://env-api.example.test/v1" || + cfg.Fal.InstanceType != "gpu_1x_h100_sxm5" || + cfg.Fal.Sector != "sector_3" || + cfg.Fal.User != "root" || + cfg.Fal.WorkRoot != "/env/fal" { + t.Fatalf("env fal config not applied: %#v", cfg.Fal) + } + if err := applyProviderConfigDefaults(&cfg); err != nil { + t.Fatal(err) + } + if cfg.TargetOS != targetLinux || cfg.SSHUser != "root" || cfg.SSHPort != "22" || cfg.WorkRoot != "/env/fal" || len(cfg.SSHFallbackPorts) != 0 { + t.Fatalf("fal defaults not applied: target=%q user=%q port=%q workRoot=%q fallback=%v", cfg.TargetOS, cfg.SSHUser, cfg.SSHPort, cfg.WorkRoot, cfg.SSHFallbackPorts) + } + if got := serverTypeForConfig(cfg); got != "gpu_1x_h100_sxm5" { + t.Fatalf("serverTypeForConfig=%q", got) + } +} + +func TestFalProviderEmptyDefaultsAndExplicitOverrides(t *testing.T) { + cfg := Config{Provider: "fal"} + if err := applyProviderConfigDefaults(&cfg); err != nil { + t.Fatal(err) + } + if cfg.Fal.APIURL != "https://api.fal.ai/v1" || + cfg.Fal.InstanceType != "gpu_1x_h100_sxm5" || + cfg.Fal.User != defaultFalUser || + cfg.Fal.WorkRoot != defaultFalWorkRoot || + cfg.TargetOS != targetLinux || + cfg.WindowsMode != windowsModeNormal || + cfg.SSHUser != defaultFalUser || + cfg.SSHPort != "22" || + cfg.WorkRoot != defaultFalWorkRoot || + cfg.ServerType != "gpu_1x_h100_sxm5" { + t.Fatalf("empty fal defaults=%#v", cfg) + } + + explicit := Config{ + Provider: "fal", + ServerType: " gpu_8x_h100_sxm5 ", + ServerTypeExplicit: true, + TargetOS: targetLinux, + targetExplicit: true, + explicitWindowsMode: windowsModeNormal, + explicitWorkRoot: "/explicit/work", + explicitSSHUser: "ubuntu", + explicitSSHPort: "2222", + Fal: FalConfig{ + WorkRoot: defaultFalWorkRoot, + }, + } + if err := applyProviderConfigDefaults(&explicit); err != nil { + t.Fatal(err) + } + if explicit.WorkRoot != "/explicit/work" || explicit.Fal.WorkRoot != "/explicit/work" || explicit.SSHUser != "ubuntu" || explicit.SSHPort != "2222" || explicit.ServerType != "gpu_8x_h100_sxm5" || explicit.Fal.InstanceType != "gpu_8x_h100_sxm5" { + t.Fatalf("explicit fal defaults=%#v", explicit) + } + if got := serverTypeForConfig(Config{Provider: "fal"}); got != "gpu_1x_h100_sxm5" { + t.Fatalf("empty serverTypeForConfig=%q", got) + } + if got := serverTypeForProviderClass("fal", "beast"); got != "gpu_1x_h100_sxm5" { + t.Fatalf("serverTypeForProviderClass=%q", got) + } +} + func TestLambdaProviderConfigDefaultsAndEnv(t *testing.T) { clearConfigEnv(t) cfg := baseConfig() @@ -5322,6 +5426,12 @@ vast: user: root workRoot: /workspaces/vast-test releaseAction: stop +fal: + apiUrl: https://api.fal.example.test/v1 + instanceType: gpu_8x_h100_sxm5 + sector: sector_2 + user: root + workRoot: /workspaces/fal-test islo: baseUrl: https://islo.example.test image: docker.io/library/ubuntu:24.04 @@ -5582,6 +5692,9 @@ ssh: if cfg.Vast.APIURL != "https://vast.example.test/api/v0" || cfg.Vast.InstanceType != "on-demand" || cfg.Vast.GPUName != "RTX 4090" || cfg.Vast.GPUCount != 2 || cfg.Vast.Image != "nvidia/cuda:vast-file" || cfg.Vast.TemplateID != "vast-tpl-file" || cfg.Vast.Runtype != "ssh_direct" || cfg.Vast.DiskGB != 60 || cfg.Vast.MaxDphTotal != 3.5 || cfg.Vast.MinReliability != 0.9 || cfg.Vast.Order != "reliability desc" || cfg.Vast.User != "root" || cfg.Vast.WorkRoot != "/workspaces/vast-test" || cfg.Vast.ReleaseAction != "stop" { t.Fatalf("vast config not loaded: %#v", cfg.Vast) } + if cfg.Fal.APIURL != "https://api.fal.example.test/v1" || cfg.Fal.InstanceType != "gpu_8x_h100_sxm5" || cfg.Fal.Sector != "sector_2" || cfg.Fal.User != "root" || cfg.Fal.WorkRoot != "/workspaces/fal-test" || cfg.Fal.APIKey != "" { + t.Fatalf("fal config not loaded safely: %#v", cfg.Fal) + } if cfg.Islo.BaseURL != "https://islo.example.test" || cfg.Islo.Image != "docker.io/library/ubuntu:24.04" || cfg.Islo.Workdir != "crabbox" || cfg.Islo.GatewayProfile != "default" || cfg.Islo.SnapshotName != "snap-ready" || cfg.Islo.VCPUs != 4 || cfg.Islo.MemoryMB != 8192 || cfg.Islo.DiskGB != 40 { t.Fatalf("islo config not loaded: %#v", cfg.Islo) } diff --git a/internal/cli/credential_provenance.go b/internal/cli/credential_provenance.go index 2ca2f283f..1e980613a 100644 --- a/internal/cli/credential_provenance.go +++ b/internal/cli/credential_provenance.go @@ -60,6 +60,8 @@ type credentialDestinationProvenance struct { runpodAPIKey credentialValueSource vastAPIURL credentialValueSource vastAPIKey credentialValueSource + falAPIURL credentialValueSource + falAPIKey credentialValueSource isloBaseURL credentialValueSource isloAPIKey credentialValueSource tenkiEndpoint credentialValueSource @@ -212,6 +214,9 @@ func markCredentialDestinationFlagSources(cfg *Config, fs *flag.FlagSet) { if flagWasSet(fs, "vast-api-url") { provenance.vastAPIURL = credentialSourceFlag } + if flagWasSet(fs, "fal-api-url") { + provenance.falAPIURL = credentialSourceFlag + } if flagWasSet(fs, "islo-base-url") { provenance.isloBaseURL = credentialSourceFlag } @@ -386,6 +391,11 @@ func validateProviderCredentialDestination(cfg Config) error { inheritedCredential(sourcedCredential{cfg.Vast.APIKey, provenance.vastAPIKey}) { return repositoryCredentialDestinationError("vast", "vast.apiUrl", "CRABBOX_VAST_API_URL or --vast-api-url") } + case "fal": + if provenance.falAPIURL == credentialSourceRepository && + inheritedCredential(sourcedCredential{cfg.Fal.APIKey, provenance.falAPIKey}) { + return repositoryCredentialDestinationError("fal", "fal.apiUrl", "CRABBOX_FAL_API_URL or --fal-api-url") + } case "islo": if provenance.isloBaseURL == credentialSourceRepository && inheritedCredential(sourcedCredential{cfg.Islo.APIKey, provenance.isloAPIKey}) { diff --git a/internal/cli/credential_provenance_test.go b/internal/cli/credential_provenance_test.go index b656b8f54..1cf2a8d3e 100644 --- a/internal/cli/credential_provenance_test.go +++ b/internal/cli/credential_provenance_test.go @@ -162,6 +162,18 @@ func TestRepositoryCredentialDestinationsRejectInheritedCredentials(t *testing.T }, want: "vast.apiUrl", }, + { + name: "fal api", + cfg: Config{ + Provider: "fal", + Fal: FalConfig{APIURL: "https://repo.example.test", APIKey: "secret"}, + credentialProvenance: credentialDestinationProvenance{ + falAPIURL: credentialSourceRepository, + falAPIKey: credentialSourceEnvironment, + }, + }, + want: "fal.apiUrl", + }, { name: "islo api", cfg: Config{ @@ -466,6 +478,31 @@ func TestVastCredentialDestinationAllowsExplicitFlagOverride(t *testing.T) { } } +func TestFalCredentialDestinationAllowsExplicitFlagOverride(t *testing.T) { + cfg := Config{ + Provider: "fal", + Fal: FalConfig{APIURL: "https://repo.example.test", APIKey: "secret"}, + credentialProvenance: credentialDestinationProvenance{ + falAPIURL: credentialSourceRepository, + falAPIKey: credentialSourceEnvironment, + }, + } + fs := newFlagSet("test", io.Discard) + values := registerProviderFlags(fs, cfg) + if err := parseFlags(fs, []string{"--fal-api-url", "https://approved.example.test/v1"}); err != nil { + t.Fatal(err) + } + if err := applyProviderFlags(&cfg, fs, values); err != nil { + t.Fatal(err) + } + if err := validateProviderCredentialDestination(cfg); err != nil { + t.Fatalf("explicit fal flag override rejected: %v", err) + } + if cfg.Fal.APIURL != "https://approved.example.test/v1" { + t.Fatalf("fal apiUrl=%q", cfg.Fal.APIURL) + } +} + func TestAzureDynamicSessionsCredentialDestinationAllowsExplicitFlagOverride(t *testing.T) { cfg := Config{ Provider: "azure-dynamic-sessions", diff --git a/internal/cli/durable_directory.go b/internal/cli/durable_directory.go new file mode 100644 index 000000000..ff9e9b7ac --- /dev/null +++ b/internal/cli/durable_directory.go @@ -0,0 +1,127 @@ +package cli + +import ( + "os" + "path/filepath" + "strings" +) + +func ensurePrivateDirectoryDurableWithSync(dir, logicalRoot string, syncDirectory func(string) error) error { + boundary, err := privateDirectoryDurabilityBoundary(dir, logicalRoot) + if err != nil { + return err + } + return ensurePrivateDirectoryDurableWithinWithSync(dir, boundary, syncDirectory) +} + +func privateDirectoryDurabilityBoundary(dir, logicalRoot string) (string, error) { + if filepath.Separator == '\\' && (isWindowsDevicePath(dir) || isWindowsDevicePath(logicalRoot)) { + return "", exit(2, "Windows device paths are not supported for private directory durability: dir=%s root=%s", dir, logicalRoot) + } + dir = filepath.Clean(dir) + logicalRoot = filepath.Clean(logicalRoot) + if !filepath.IsAbs(dir) || !filepath.IsAbs(logicalRoot) { + return "", exit(2, "private directory and logical root must be absolute: dir=%s root=%s", dir, logicalRoot) + } + if !pathWithinRoot(dir, logicalRoot) { + return "", exit(2, "private directory %s is outside logical root %s", dir, logicalRoot) + } + // Prefer the outer temporary root before HOME so a HOME created beneath it + // cannot narrow the boundary after an interrupted first attempt. + for _, candidate := range []string{os.TempDir(), userHomeDirectory()} { + candidate = filepath.Clean(candidate) + if candidate == "." || filepath.Dir(candidate) == candidate || !pathWithinRoot(logicalRoot, candidate) { + continue + } + info, err := os.Stat(candidate) + if err != nil || !info.IsDir() { + return "", exit(2, "trusted private directory boundary %s is unavailable", candidate) + } + return candidate, nil + } + return stableTopLevelDirectoryBoundary(logicalRoot) +} + +func isWindowsDevicePath(path string) bool { + prefix := path + if len(prefix) > 4 { + prefix = prefix[:4] + } + prefix = strings.ReplaceAll(prefix, "/", `\`) + return strings.HasPrefix(prefix, `\\?\`) || + strings.HasPrefix(prefix, `\??\`) || + strings.HasPrefix(prefix, `\\.\`) +} + +func stableTopLevelDirectoryBoundary(path string) (string, error) { + path = filepath.Clean(path) + if !filepath.IsAbs(path) { + return "", exit(2, "private directory logical root %s must be absolute", path) + } + volume := filepath.VolumeName(path) + root := volume + string(filepath.Separator) + relative, err := filepath.Rel(root, path) + if err != nil || relative == "." || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) { + return "", exit(2, "private directory logical root %s has no safe top-level boundary", path) + } + first, _, _ := strings.Cut(relative, string(filepath.Separator)) + boundary := filepath.Join(root, first) + info, err := os.Stat(boundary) + if err != nil { + return "", exit(2, "inspect private directory top-level boundary %s: %v", boundary, err) + } + if !info.IsDir() || filepath.Dir(boundary) == boundary { + return "", exit(2, "private directory top-level boundary %s is not a safe existing directory", boundary) + } + return boundary, nil +} + +func userHomeDirectory() string { + home, _ := os.UserHomeDir() + return home +} + +func ensurePrivateDirectoryDurableWithinWithSync(dir, boundary string, syncDirectory func(string) error) error { + dir = filepath.Clean(dir) + boundary = filepath.Clean(boundary) + info, err := os.Stat(boundary) + if err != nil || !info.IsDir() || filepath.Dir(boundary) == boundary { + return exit(2, "private directory boundary %s is not a safe existing directory", boundary) + } + if !pathWithinRoot(dir, boundary) { + return exit(2, "private directory %s is outside durability boundary %s", dir, boundary) + } + if err := os.MkdirAll(dir, 0o700); err != nil { + return exit(2, "create private directory %s: %v", dir, err) + } + if dir == boundary { + return nil + } + for current := filepath.Dir(dir); ; current = filepath.Dir(current) { + if err := syncDirectory(current); err != nil { + return exit(2, "sync private directory namespace parent %s: %v", current, err) + } + if current == boundary { + return nil + } + if parent := filepath.Dir(current); parent == current { + return exit(2, "sync private directory namespace: boundary %s is not an ancestor of %s", boundary, dir) + } + } +} + +func ensureCrabboxClaimNamespaceDurable() error { + return ensureCrabboxClaimNamespaceDurableWithSync(syncControllerDirectory) +} + +func ensureCrabboxClaimNamespaceDurableWithSync(syncDirectory func(string) error) error { + stateDir, err := crabboxStateDir() + if err != nil { + return err + } + stateRoot, err := crabboxStateRootDir() + if err != nil { + return err + } + return ensurePrivateDirectoryDurableWithSync(filepath.Join(stateDir, "claims"), stateRoot, syncDirectory) +} diff --git a/internal/cli/lease.go b/internal/cli/lease.go index a046a57da..86e043575 100644 --- a/internal/cli/lease.go +++ b/internal/cli/lease.go @@ -60,16 +60,95 @@ func EnsureTestboxKey(leaseID string) (string, string, error) { } func ensureTestboxKeyForConfig(cfg Config, leaseID string) (string, string, error) { - if (cfg.Provider == "aws" || cfg.Provider == "azure") && cfg.TargetOS == targetWindows { - return ensureTestboxKeyWithType(leaseID, "rsa") - } - return ensureTestboxKey(leaseID) + return ensureTestboxKeyWithType(leaseID, testboxKeyTypeForConfig(cfg)) } func EnsureTestboxKeyForConfig(cfg Config, leaseID string) (string, string, error) { return ensureTestboxKeyForConfig(cfg, leaseID) } +func testboxKeyTypeForConfig(cfg Config) string { + if (cfg.Provider == "aws" || cfg.Provider == "azure") && cfg.TargetOS == targetWindows { + return "rsa" + } + return "ed25519" +} + +func ensureTestboxKeyForConfigDurable(cfg Config, leaseID string) (string, string, error) { + return ensureTestboxKeyForConfigDurableWithSync(cfg, leaseID, syncControllerDirectory) +} + +func ensureTestboxKeyForConfigDurableWithSync(cfg Config, leaseID string, syncDirectory func(string) error) (string, string, error) { + privatePath, err := testboxKeyPath(leaseID) + if err != nil { + return "", "", err + } + configDir, err := os.UserConfigDir() + if err != nil { + return "", "", exit(2, "user config directory is unavailable") + } + if err := ensurePrivateDirectoryDurableWithSync(filepath.Dir(privatePath), configDir, syncDirectory); err != nil { + return "", "", exit(2, "create durable testbox key directory: %v", err) + } + privatePath, publicKey, err := ensureTestboxKeyWithType(leaseID, testboxKeyTypeForConfig(cfg)) + if err != nil { + return "", "", err + } + if err := syncStoredTestboxKeyWithSync(leaseID, syncDirectory); err != nil { + return "", "", exit(2, "sync durable testbox key: %v", err) + } + return privatePath, publicKey, nil +} + +func syncStoredTestboxKey(leaseID string) error { + return syncStoredTestboxKeyWithSync(leaseID, syncControllerDirectory) +} + +func syncStoredTestboxKeyWithSync(leaseID string, syncDirectory func(string) error) error { + privatePath, err := testboxKeyPath(leaseID) + if err != nil { + return err + } + for _, path := range []string{privatePath, privatePath + ".pub"} { + file, err := os.OpenFile(path, os.O_RDWR, 0) + if err != nil { + return err + } + if err := file.Sync(); err != nil { + _ = file.Close() + return err + } + if err := file.Close(); err != nil { + return err + } + } + configDir, err := os.UserConfigDir() + if err != nil { + return exit(2, "user config directory is unavailable") + } + return syncTestboxKeyDirectoriesWithSync(filepath.Dir(privatePath), configDir, syncDirectory) +} + +func syncTestboxKeyDirectoriesWithSync(keyDir, configDir string, syncDirectory func(string) error) error { + keyDir = filepath.Clean(keyDir) + configDir = filepath.Clean(configDir) + relative, err := filepath.Rel(configDir, keyDir) + if err != nil || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) { + return exit(2, "testbox key directory %s is outside user config directory %s", keyDir, configDir) + } + for current := keyDir; ; current = filepath.Dir(current) { + if err := syncDirectory(current); err != nil { + return exit(2, "sync testbox key directory %s: %v", current, err) + } + if current == configDir { + return nil + } + if parent := filepath.Dir(current); parent == current { + return exit(2, "sync testbox key directory: boundary %s is not an ancestor of %s", configDir, keyDir) + } + } +} + func ensureTestboxKeyWithType(leaseID, keyType string) (string, string, error) { privatePath, err := testboxKeyPath(leaseID) if err != nil { @@ -160,17 +239,59 @@ func MoveStoredTestboxKey(oldLeaseID, newLeaseID string) error { return moveStoredTestboxKey(oldLeaseID, newLeaseID) } -func removeStoredTestboxKey(leaseID string) { +func removeStoredTestboxKeyWithError(leaseID string) error { keyPath, err := testboxKeyPath(leaseID) - if err == nil { - _ = os.RemoveAll(filepath.Dir(keyPath)) + if err != nil { + return err + } + keyDir := filepath.Dir(keyPath) + if _, err := os.Stat(keyDir); errors.Is(err, os.ErrNotExist) { + return syncNearestExistingDirectory(filepath.Dir(keyDir)) + } else if err != nil { + return err + } + if err := os.RemoveAll(keyDir); err != nil { + return err + } + return syncNearestExistingDirectory(filepath.Dir(keyDir)) +} + +func syncNearestExistingDirectory(path string) error { + return syncNearestExistingDirectoryWithSync(path, syncControllerDirectory) +} + +func syncNearestExistingDirectoryWithSync(path string, syncDirectory func(string) error) error { + for { + info, err := os.Stat(path) + if err == nil { + if !info.IsDir() { + return exit(2, "state path %s is not a directory", path) + } + return syncDirectory(path) + } + if !errors.Is(err, os.ErrNotExist) { + return err + } + parent := filepath.Dir(path) + if parent == path { + return err + } + path = parent } } +func removeStoredTestboxKey(leaseID string) { + _ = removeStoredTestboxKeyWithError(leaseID) +} + func RemoveStoredTestboxKey(leaseID string) { removeStoredTestboxKey(leaseID) } +func RemoveStoredTestboxKeyWithError(leaseID string) error { + return removeStoredTestboxKeyWithError(leaseID) +} + func providerKeyForLease(leaseID string) string { return strings.ReplaceAll("crabbox-"+leaseID, "_", "-") } diff --git a/internal/cli/lease_test.go b/internal/cli/lease_test.go index d34bd5a79..23926665c 100644 --- a/internal/cli/lease_test.go +++ b/internal/cli/lease_test.go @@ -1,8 +1,10 @@ package cli import ( + "errors" "os" "path/filepath" + "slices" "strings" "testing" ) @@ -36,6 +38,118 @@ func TestTestboxKeyPathAllowsSafeCustomIDs(t *testing.T) { } } +func TestSyncAndRemoveStoredTestboxKey(t *testing.T) { + t.Setenv("HOME", t.TempDir()) + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + const leaseID = "cbx_durable_key" + keyPath, _, err := ensureTestboxKey(leaseID) + if err != nil { + t.Fatal(err) + } + if err := syncStoredTestboxKey(leaseID); err != nil { + t.Fatal(err) + } + if err := removeStoredTestboxKeyWithError(leaseID); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(keyPath); !os.IsNotExist(err) { + t.Fatalf("removed key stat error=%v", err) + } +} + +func TestSyncStoredTestboxKeyStopsAtUserConfigBoundary(t *testing.T) { + t.Setenv("HOME", t.TempDir()) + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + configDir, err := os.UserConfigDir() + if err != nil { + t.Fatal(err) + } + const leaseID = "cbx_bounded_key_sync" + keyPath, _, err := ensureTestboxKey(leaseID) + if err != nil { + t.Fatal(err) + } + var synced []string + if err := syncStoredTestboxKeyWithSync(leaseID, func(dir string) error { + synced = append(synced, filepath.Clean(dir)) + return nil + }); err != nil { + t.Fatal(err) + } + want := []string{ + filepath.Dir(keyPath), + filepath.Dir(filepath.Dir(keyPath)), + filepath.Dir(filepath.Dir(filepath.Dir(keyPath))), + filepath.Clean(configDir), + } + if len(synced) != len(want) { + t.Fatalf("synced=%q want=%q", synced, want) + } + for index := range want { + if synced[index] != filepath.Clean(want[index]) { + t.Fatalf("synced=%q want=%q", synced, want) + } + } +} + +func TestEnsureTestboxKeyForConfigDurableRetriesStableBoundary(t *testing.T) { + base := t.TempDir() + home := filepath.Join(base, "fresh", "home") + t.Setenv("HOME", home) + t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config")) + t.Setenv("APPDATA", filepath.Join(home, "AppData", "Roaming")) + configDir, err := os.UserConfigDir() + if err != nil { + t.Fatal(err) + } + const leaseID = "cbx_durable_key_boundary" + keyPath, err := testboxKeyPath(leaseID) + if err != nil { + t.Fatal(err) + } + boundary, err := privateDirectoryDurabilityBoundary(filepath.Dir(keyPath), configDir) + if err != nil { + t.Fatal(err) + } + syncErr := errors.New("stable boundary sync unavailable") + var failedSyncs []string + _, _, err = ensureTestboxKeyForConfigDurableWithSync(Config{}, leaseID, func(dir string) error { + dir = filepath.Clean(dir) + failedSyncs = append(failedSyncs, dir) + if dir == boundary { + return syncErr + } + return nil + }) + if err == nil || !strings.Contains(err.Error(), syncErr.Error()) { + t.Fatalf("durable key err=%v", err) + } + if _, err := os.Stat(keyPath); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("key created before durable namespace: %v", err) + } + if len(failedSyncs) == 0 || failedSyncs[len(failedSyncs)-1] != boundary { + t.Fatalf("failed syncs=%q want boundary %q", failedSyncs, boundary) + } + + var retrySyncs []string + gotPath, publicKey, err := ensureTestboxKeyForConfigDurableWithSync(Config{}, leaseID, func(dir string) error { + retrySyncs = append(retrySyncs, filepath.Clean(dir)) + return nil + }) + if err != nil { + t.Fatal(err) + } + if gotPath != keyPath || publicKey == "" { + t.Fatalf("path=%q publicKeyPresent=%t", gotPath, publicKey != "") + } + if _, err := os.Stat(keyPath + ".pub"); err != nil { + t.Fatalf("public key stat: %v", err) + } + if !slices.Contains(retrySyncs, boundary) || retrySyncs[len(retrySyncs)-1] != filepath.Clean(configDir) { + t.Fatalf("retry syncs=%q want stable boundary %q and final config dir %q", retrySyncs, boundary, configDir) + } +} + func TestUseLeaseKnownHostsScopesAndEnforcesHostVerification(t *testing.T) { t.Setenv("HOME", t.TempDir()) t.Setenv("XDG_CONFIG_HOME", t.TempDir()) diff --git a/internal/cli/profiles_test.go b/internal/cli/profiles_test.go index ba69291e2..25aa00f54 100644 --- a/internal/cli/profiles_test.go +++ b/internal/cli/profiles_test.go @@ -753,6 +753,11 @@ func TestRunStopCommandRedactsProviderURLUserinfo(t *testing.T) { cfg: Config{Provider: "hostinger", Hostinger: HostingerConfig{APIURL: rawURL}}, wantFlag: "--hostinger-url 'https://provider.example.test/path?view=1'", }, + { + name: "fal", + cfg: Config{Provider: "fal", Fal: FalConfig{APIURL: rawURL}}, + wantFlag: "--fal-api-url 'https://provider.example.test/path?view=1'", + }, } { t.Run(test.name, func(t *testing.T) { got := runStopCommand(test.cfg, "cbx_123") diff --git a/internal/cli/provider_backend.go b/internal/cli/provider_backend.go index 0c7fbe465..03145c706 100644 --- a/internal/cli/provider_backend.go +++ b/internal/cli/provider_backend.go @@ -62,6 +62,18 @@ type ProviderCommandRoutingArgs interface { CommandRoutingArgs(cfg Config, leaseID string) []string } +// ProviderClaimScopeProvider binds local claims to provider-owned routing +// identity that core cannot interpret. +type ProviderClaimScopeProvider interface { + ProviderClaimScope(cfg Config) string +} + +// ProviderStopRoutingArgsProvider preserves provider-owned routing when core +// constructs a detached cleanup command. +type ProviderStopRoutingArgsProvider interface { + StopRoutingArgs(cfg Config, leaseID string) []string +} + type DesktopCredentials struct { Username string Password string diff --git a/internal/cli/provider_backend_test.go b/internal/cli/provider_backend_test.go index b0433e627..986b5f831 100644 --- a/internal/cli/provider_backend_test.go +++ b/internal/cli/provider_backend_test.go @@ -179,6 +179,12 @@ func TestLeaseOptionsFromConfigCanonicalizesProviderScope(t *testing.T) { if scope := leaseOptionsFromConfig(cfg).ProviderScope; scope != "endpoint:https://pve.example.test:8006|node:pve1" { t.Fatalf("proxmox provider scope=%q", scope) } + + cfg.Provider = "fal-ai" + cfg.Fal.APIURL = "HTTPS://API.FAL.AI:443/v1/" + if scope := leaseOptionsFromConfig(cfg).ProviderScope; scope != "endpoint:https://api.fal.ai/v1" { + t.Fatalf("fal provider scope=%q", scope) + } } func TestProviderHelpAllIncludesDelegatedProviders(t *testing.T) { diff --git a/internal/cli/provider_categories_generated.go b/internal/cli/provider_categories_generated.go index 38ff2a9bc..ab456f69f 100644 --- a/internal/cli/provider_categories_generated.go +++ b/internal/cli/provider_categories_generated.go @@ -28,6 +28,7 @@ var benchmarkProviderCategories = map[string]string{ "e2b": "delegated-sandbox", "exe-dev": "direct-cloud", "external": "external-provider", + "fal": "gpu-cloud", "fastapi-cloud": "service-control", "firecracker": "self-hosted-virtualization", "freestyle": "delegated-sandbox", diff --git a/internal/cli/provider_exports.go b/internal/cli/provider_exports.go index e38cd8ced..27aa30e64 100644 --- a/internal/cli/provider_exports.go +++ b/internal/cli/provider_exports.go @@ -134,6 +134,12 @@ func ClaimLeaseTargetForConfigScopeIfUnchanged(leaseID, slug string, cfg Config, return claimLeaseTargetForConfigScopeIfUnchanged(leaseID, slug, cfg, providerScope, server, target, idleTimeout, expected, expectedExists) } +// ClaimLeaseTargetForConfigScopeIfUnchangedDurable performs the same guarded +// update and also durably syncs any newly created claim namespace ancestors. +func ClaimLeaseTargetForConfigScopeIfUnchangedDurable(leaseID, slug string, cfg Config, providerScope string, server Server, target SSHTarget, idleTimeout time.Duration, expected LeaseClaim, expectedExists bool) (LeaseClaim, error) { + return claimLeaseTargetForConfigScopeIfUnchangedDurable(leaseID, slug, cfg, providerScope, server, target, idleTimeout, expected, expectedExists) +} + func ClaimLeaseTargetForRepoConfigIfUnchanged(leaseID, slug string, cfg Config, server Server, target SSHTarget, repoRoot string, idleTimeout time.Duration, reclaim bool, expected LeaseClaim, expectedExists bool) (LeaseClaim, error) { return claimLeaseTargetForRepoConfigIfUnchanged(leaseID, slug, cfg, server, target, repoRoot, idleTimeout, reclaim, expected, expectedExists) } @@ -188,6 +194,18 @@ func ProviderClaimScope(provider string, cfg Config) string { return providerClaimScope(canonicalClaimProvider(provider), cfg) } +// NormalizeProviderClaimEndpoint canonicalizes a provider API endpoint for use +// as non-secret local claim identity. +func NormalizeProviderClaimEndpoint(raw string) string { + return normalizedProviderClaimEndpoint(raw) +} + +// RoutingSafeURL removes URL userinfo before a provider routes configuration +// through a generated command line. +func RoutingSafeURL(raw string) string { + return routingSafeURL(raw) +} + func RemoveLeaseClaim(leaseID string) { removeLeaseClaim(leaseID) } @@ -211,6 +229,13 @@ func RemoveLeaseClaimIfUnchangedAfter(leaseID string, expected LeaseClaim, actio func CleanupLeaseClaimIfUnchangedAfter(leaseID string, expected LeaseClaim, expectedExists bool, action func() error) error { return cleanupLeaseClaimIfUnchangedAfter(leaseID, expected, expectedExists, action) } + +// FinalizeAbsentLeaseClaimAfterSync verifies and syncs durable claim absence +// before running action under the claim lock. +func FinalizeAbsentLeaseClaimAfterSync(leaseID string, action func() error) error { + return finalizeAbsentLeaseClaimAfterSync(leaseID, action) +} + func RestoreLeaseClaimIfUnchanged(leaseID string, current, previous LeaseClaim, previousExists bool) error { return restoreLeaseClaimIfUnchanged(leaseID, current, previous, previousExists) } @@ -422,6 +447,14 @@ func CrabboxStateDir() (string, error) { return crabboxStateDir() } +func EnsureCrabboxClaimNamespaceDurable() error { + return ensureCrabboxClaimNamespaceDurable() +} + +func EnsureTestboxKeyForConfigDurable(cfg Config, leaseID string) (string, string, error) { + return ensureTestboxKeyForConfigDurable(cfg, leaseID) +} + func DirectLeaseLabels(cfg Config, leaseID, slug, provider, market string, keep bool, now time.Time) map[string]string { return directLeaseLabels(cfg, leaseID, slug, provider, market, keep, now) } diff --git a/internal/cli/providers_builtin_test.go b/internal/cli/providers_builtin_test.go index 299819640..a636b11aa 100644 --- a/internal/cli/providers_builtin_test.go +++ b/internal/cli/providers_builtin_test.go @@ -34,6 +34,7 @@ func init() { RegisterProvider(testExeDevProvider{}) RegisterProvider(testRunPodProvider{}) RegisterProvider(testVastProvider{}) + RegisterProvider(testFalProvider{}) RegisterProvider(testNvidiaBrevProvider{}) RegisterProvider(testBlacksmithProvider{}) RegisterProvider(testNamespaceProvider{}) @@ -1138,6 +1139,79 @@ func (p testVastProvider) Configure(Config, Runtime) (Backend, error) { return testSSHBackend{spec: p.Spec()}, nil } +type testFalProvider struct{} + +func (testFalProvider) Name() string { return "fal" } +func (testFalProvider) Aliases() []string { return []string{"fal-ai"} } +func (testFalProvider) ProviderClaimScope(cfg Config) string { + endpoint := normalizedProviderClaimEndpoint(cfg.Fal.APIURL) + if endpoint == "" { + return "" + } + return "endpoint:" + endpoint +} +func (testFalProvider) StopRoutingArgs(cfg Config, _ string) []string { + apiURL := strings.TrimSpace(cfg.Fal.APIURL) + if apiURL == "" { + return nil + } + return []string{"--fal-api-url", routingSafeURL(apiURL)} +} +func (testFalProvider) Spec() ProviderSpec { + return ProviderSpec{ + Name: "fal", + Family: "fal", + Kind: ProviderKindSSHLease, + Targets: []TargetSpec{{OS: targetLinux}}, + Features: FeatureSet{FeatureSSH, FeatureCrabboxSync, FeatureCleanup}, + Coordinator: CoordinatorNever, + } +} +func (testFalProvider) RegisterFlags(fs *flag.FlagSet, defaults Config) any { + return testFalFlagValues{ + APIURL: fs.String("fal-api-url", defaults.Fal.APIURL, "fal Compute API URL"), + InstanceType: fs.String("fal-instance-type", defaults.Fal.InstanceType, "fal Compute instance type"), + Sector: fs.String("fal-sector", defaults.Fal.Sector, "fal Compute sector"), + User: fs.String("fal-user", defaults.Fal.User, "SSH user for fal Compute instances"), + WorkRoot: fs.String("fal-work-root", defaults.Fal.WorkRoot, "remote Crabbox work root on fal Compute instances"), + } +} +func (testFalProvider) ApplyFlags(cfg *Config, fs *flag.FlagSet, values any) error { + v, ok := values.(testFalFlagValues) + if !ok { + return nil + } + if flagWasSet(fs, "fal-api-url") { + cfg.Fal.APIURL = *v.APIURL + } + if flagWasSet(fs, "fal-instance-type") { + cfg.Fal.InstanceType = *v.InstanceType + } + if flagWasSet(fs, "fal-sector") { + cfg.Fal.Sector = *v.Sector + } + if flagWasSet(fs, "fal-user") { + cfg.Fal.User = *v.User + cfg.SSHUser = *v.User + } + if flagWasSet(fs, "fal-work-root") { + cfg.Fal.WorkRoot = *v.WorkRoot + cfg.WorkRoot = *v.WorkRoot + } + return nil +} +func (p testFalProvider) Configure(Config, Runtime) (Backend, error) { + return testSSHBackend{spec: p.Spec()}, nil +} + +type testFalFlagValues struct { + APIURL *string + InstanceType *string + Sector *string + User *string + WorkRoot *string +} + type testNvidiaBrevProvider struct{} func (testNvidiaBrevProvider) Name() string { return "nvidia-brev" } diff --git a/internal/cli/providers_test.go b/internal/cli/providers_test.go index 2f8ba587f..cc51a6416 100644 --- a/internal/cli/providers_test.go +++ b/internal/cli/providers_test.go @@ -18,6 +18,7 @@ func TestProviderMatrixIncludesCapabilities(t *testing.T) { var vultr *providerMatrixEntry var firecracker *providerMatrixEntry var vast *providerMatrixEntry + var fal *providerMatrixEntry var nvidiaBrev *providerMatrixEntry var linode *providerMatrixEntry var nebius *providerMatrixEntry @@ -47,6 +48,9 @@ func TestProviderMatrixIncludesCapabilities(t *testing.T) { if entries[i].Provider == "vast" { vast = &entries[i] } + if entries[i].Provider == "fal" { + fal = &entries[i] + } if entries[i].Provider == "nvidia-brev" { nvidiaBrev = &entries[i] } @@ -96,6 +100,9 @@ func TestProviderMatrixIncludesCapabilities(t *testing.T) { if vast == nil { t.Fatal("vast provider not found") } + if fal == nil { + t.Fatal("fal provider not found") + } if nvidiaBrev == nil { t.Fatal("nvidia-brev provider not found") } @@ -193,6 +200,29 @@ func TestProviderMatrixIncludesCapabilities(t *testing.T) { if len(firecracker.Aliases) != 0 { t.Fatalf("firecracker aliases=%v, want none", firecracker.Aliases) } + if fal.Kind != ProviderKindSSHLease || fal.Family != "fal" || fal.Coordinator != string(CoordinatorNever) { + t.Fatalf("fal kind/family/coordinator=%q/%q/%q", fal.Kind, fal.Family, fal.Coordinator) + } + if !containsString(fal.Targets, targetLinux) { + t.Fatalf("fal targets=%v", fal.Targets) + } + for _, feature := range []Feature{FeatureSSH, FeatureCrabboxSync, FeatureCleanup} { + if !containsFeature(fal.Features, feature) { + t.Fatalf("fal features=%v missing %s", fal.Features, feature) + } + } + if !containsString(fal.Runtime, "ssh-host") { + t.Fatalf("fal runtime=%v missing ssh-host", fal.Runtime) + } + if !containsString(fal.Reachability, "ssh-tunnel") { + t.Fatalf("fal reachability=%v missing ssh-tunnel", fal.Reachability) + } + if !containsString(fal.Lifecycle, "cleanup") { + t.Fatalf("fal lifecycle=%v missing cleanup", fal.Lifecycle) + } + if !containsString(fal.Aliases, "fal-ai") { + t.Fatalf("fal aliases=%v", fal.Aliases) + } if linode.Kind != ProviderKindSSHLease || linode.Family != "linode" || linode.Coordinator != string(CoordinatorNever) { t.Fatalf("linode kind/family/coordinator=%q/%q/%q", linode.Kind, linode.Family, linode.Coordinator) } @@ -367,6 +397,17 @@ func TestProvidersCommandJSON(t *testing.T) { if entry.Provider == "blacksmith-testbox" && !containsString(entry.Lifecycle, "run-session") { t.Fatalf("blacksmith json missing run-session lifecycle: %#v", entry) } + if entry.Provider == "fal" && (entry.Kind != ProviderKindSSHLease || + !containsFeature(entry.Features, FeatureSSH) || + !containsFeature(entry.Features, FeatureCrabboxSync) || + !containsFeature(entry.Features, FeatureCleanup) || + !containsString(entry.Aliases, "fal-ai") || + !containsString(entry.Runtime, "ssh-host") || + !containsString(entry.Reachability, "ssh-tunnel") || + !containsString(entry.Lifecycle, "cleanup") || + entry.Family != "fal") { + t.Fatalf("fal json entry incomplete: %#v", entry) + } } } diff --git a/internal/cli/run.go b/internal/cli/run.go index 971641244..81c0157cd 100644 --- a/internal/cli/run.go +++ b/internal/cli/run.go @@ -2173,6 +2173,11 @@ func appendProviderStopRoutingArgs(args []string, cfg Config, id string) []strin } } } + if provider, err := ProviderFor(cfg.Provider); err == nil { + if routed, ok := provider.(ProviderStopRoutingArgsProvider); ok { + args = append(args, routed.StopRoutingArgs(cfg, id)...) + } + } return args } diff --git a/internal/providers/all/all.go b/internal/providers/all/all.go index e84dcba32..f8733ec1f 100644 --- a/internal/providers/all/all.go +++ b/internal/providers/all/all.go @@ -26,6 +26,7 @@ import ( _ "github.com/openclaw/crabbox/internal/providers/e2b" _ "github.com/openclaw/crabbox/internal/providers/exedev" _ "github.com/openclaw/crabbox/internal/providers/external" + _ "github.com/openclaw/crabbox/internal/providers/fal" _ "github.com/openclaw/crabbox/internal/providers/fastapicloud" _ "github.com/openclaw/crabbox/internal/providers/firecracker" _ "github.com/openclaw/crabbox/internal/providers/freestyle" diff --git a/internal/providers/all/all_test.go b/internal/providers/all/all_test.go index c62f58cf7..f7a6bf000 100644 --- a/internal/providers/all/all_test.go +++ b/internal/providers/all/all_test.go @@ -1273,6 +1273,7 @@ func allBuiltInProviderNames() []string { "e2b", "exe-dev", "external", + "fal", "fastapi-cloud", "firecracker", "freestyle", diff --git a/internal/providers/fal/backend.go b/internal/providers/fal/backend.go new file mode 100644 index 000000000..8f4826cc3 --- /dev/null +++ b/internal/providers/fal/backend.go @@ -0,0 +1,2171 @@ +package fal + +import ( + "context" + "crypto/sha256" + "errors" + "fmt" + "os" + "strconv" + "strings" + "time" + + core "github.com/openclaw/crabbox/internal/cli" +) + +const ( + falPollInterval = 3 * time.Second + falPollTimeout = 10 * time.Minute + falCreateReconcileAttempts = 3 + falCreateReconcileRetryWait = time.Second + falCreateRecoveryWindow = 9 * time.Minute + falCreateRequestTimeout = 30 * time.Second + falDeleteConfirmationTimeout = 30 * time.Second + falCredentialBindingLabel = "fal_credential_binding" + falCreateAttemptLabel = "fal_create_attempt" + falCreateRequestLabel = "fal_create_request_binding" + falCreateRejectedRecovery = "create-rejected" + falDeleteAttemptLabel = "fal_delete_attempt" + falDeleteAcceptedLabel = "fal_delete_accepted" + falDeleteConfirmedLabel = "fal_delete_confirmed" + falAcquireLifetimeLabel = "fal_acquire_lifetime" + falAcquireLifetimeVersion = "1" +) + +var ( + errFalProviderAbsenceNotAccountBound = errors.New("fal provider absence is not account-bound") + errFalCredentialBindingMismatch = errors.New("fal credential binding mismatch") + errFalRecoveryClaimRemoved = errors.New("fal recovery claim was removed by concurrent cleanup") + errFalClaimMutationSuperseded = errors.New("fal claim mutation was superseded") +) + +func (b *backend) Acquire(ctx context.Context, req core.AcquireRequest) (core.LeaseTarget, error) { + cfg := b.configForRun() + cfg.Fal.InstanceType = strings.TrimSpace(cfg.Fal.InstanceType) + cfg.ServerType = cfg.Fal.InstanceType + cfg.Fal.Sector = strings.TrimSpace(cfg.Fal.Sector) + if InstanceType(cfg.Fal.InstanceType) != InstanceTypeH100x8 { + cfg.Fal.Sector = "" + } + client, err := b.api() + if err != nil { + return core.LeaseTarget{}, err + } + leaseID := core.NewLeaseID() + unlockAcquire, err := lockFalAcquireLifetime(ctx, leaseID) + if err != nil { + return core.LeaseTarget{}, err + } + defer unlockAcquire() + unlockSlug, err := lockFalSlugAllocation(ctx) + if err != nil { + return core.LeaseTarget{}, err + } + defer unlockSlug() + slug, err := core.AllocateClaimLeaseSlug(leaseID, req.RequestedSlug) + if err != nil { + return core.LeaseTarget{}, err + } + keyPath, publicKey, err := b.ensureFalCreateKey(cfg, leaseID) + if err != nil { + return core.LeaseTarget{}, errors.Join(err, core.RemoveStoredTestboxKeyWithError(leaseID)) + } + cfg.SSHKey = keyPath + fmt.Fprintf(b.rt.Stderr, "provisioning provider=%s lease=%s slug=%s instance=%s sector=%s keep=%v\n", + providerName, leaseID, slug, cfg.Fal.InstanceType, cfg.Fal.Sector, req.Keep) + + createRequest := CreateInstanceRequest{ + InstanceType: InstanceType(cfg.Fal.InstanceType), + SSHKey: publicKey, + Sector: Sector(cfg.Fal.Sector), + } + createStarted := b.now() + intentClaim, claimErr := b.persistInitialFalCreateIntent(leaseID, slug, cfg, req.Repo.Root, req.Keep, createStarted, createRequest) + if claimErr != nil { + cleanupErr := b.cleanupRejectedFalCreateIntent(leaseID, intentClaim) + return core.LeaseTarget{}, errors.Join(fmt.Errorf("persist fal create intent before provider mutation: %w", claimErr), cleanupErr) + } + if unlockSlug != nil { + unlockSlug() + unlockSlug = nil + } + created, initialClaim, ambiguous, createErr := b.replayFalCreateWithClaim(ctx, client, createRequest, intentClaim, cfg, "provisioning", req.Keep) + needsBind := strings.TrimSpace(created.ID) != "" && createErr != nil + if createErr != nil && !ambiguous && !needsBind { + cleanupErr := b.cleanupRejectedFalCreateIntent(leaseID, initialClaim) + return core.LeaseTarget{}, errors.Join(createErr, cleanupErr) + } + if ambiguous { + created, initialClaim, createErr = b.reconcileAmbiguousCreate(ctx, client, createRequest, initialClaim, cfg, req.Keep, createErr) + needsBind = strings.TrimSpace(created.ID) != "" && createErr != nil + if createErr != nil && !needsBind { + if falAmbiguousCreateClaim(initialClaim) { + createErr = fmt.Errorf( + "fal create outcome is ambiguous for lease=%s slug=%s; run crabbox stop --provider fal --id %s immediately to recover and delete any created instance: %w", + leaseID, + slug, + leaseID, + createErr, + ) + } + return core.LeaseTarget{}, createErr + } + } + instanceID := strings.TrimSpace(created.ID) + var bindErr error + if needsBind { + base := initialClaim + if base.LeaseID == "" { + base = intentClaim + } + var bound bool + initialClaim, bound, bindErr = b.adoptOrBindKnownFalInstance(base, cfg, instanceID, "provisioning", req.Keep) + if bindErr == nil && !bound { + bindErr = errFalRecoveryClaimRemoved + } + } + if bindErr != nil { + cause := fmt.Errorf("persist fal provisioning claim after creating instance %s: %w", instanceID, bindErr) + return core.LeaseTarget{}, b.cleanupKnownFalCreateAfterBindFailure(ctx, client, intentClaim, cfg, instanceID, "rollback-cleanup", req.Keep, cause) + } + if req.OnAcquired != nil { + rawTarget := core.LeaseTarget{ + Server: falServer(created, cfg, leaseID, slug, req.Keep, createStarted), + LeaseID: leaseID, + } + if rawSSH, sshErr := falSSHTarget(cfg, created); sshErr == nil { + rawTarget.SSH = rawSSH + } + if err := req.OnAcquired(rawTarget); err != nil { + return core.LeaseTarget{}, b.rollbackClaimedAcquire(instanceID, leaseID, slug, cfg, req.Repo.Root, "rollback-cleanup", err) + } + } + + ready, err := b.waitForInstanceReady(ctx, client, instanceID) + if err != nil { + return core.LeaseTarget{}, b.handleFailedAcquire(instanceID, leaseID, slug, cfg, req.Repo.Root, req.Keep, err) + } + server := falServer(ready, cfg, leaseID, slug, req.Keep, createStarted) + ssh, err := falSSHTarget(cfg, ready) + if err != nil { + return core.LeaseTarget{}, b.handleFailedAcquire(instanceID, leaseID, slug, cfg, req.Repo.Root, req.Keep, err) + } + if err := b.waitForSSH(ctx, &ssh, "fal bootstrap", core.BootstrapWaitTimeout(cfg)); err != nil { + return core.LeaseTarget{}, b.handleFailedAcquire(instanceID, leaseID, slug, cfg, req.Repo.Root, req.Keep, err) + } + claimServer, err := falClaimServer(server, cfg) + if err != nil { + return core.LeaseTarget{}, b.handleFailedAcquire(instanceID, leaseID, slug, cfg, req.Repo.Root, req.Keep, err) + } + unlockReady, err := lockFalLeaseOperation(ctx, leaseID) + if err != nil { + return core.LeaseTarget{}, b.handleFailedAcquire(instanceID, leaseID, slug, cfg, req.Repo.Root, req.Keep, err) + } + defer unlockReady() + var readyClaim core.LeaseClaim + var publishErr error + if req.Repo.Root != "" { + readyClaim, publishErr = core.ClaimLeaseTargetForRepoConfigScopeIfUnchangedDurable(leaseID, slug, cfg, falClaimScope(cfg), claimServer, ssh, req.Repo.Root, cfg.IdleTimeout, req.Reclaim, initialClaim, true) + } else { + readyClaim, publishErr = core.ClaimLeaseTargetForConfigScopeIfUnchangedDurable(leaseID, slug, cfg, falClaimScope(cfg), claimServer, ssh, cfg.IdleTimeout, initialClaim, true) + } + if publishErr != nil { + publishErr = b.reconcileFalReadyClaimWrite(ctx, client, initialClaim, readyClaim, instanceID, publishErr) + } + unlockReady() + if publishErr != nil { + if errors.Is(publishErr, errFalClaimMutationSuperseded) { + return core.LeaseTarget{}, publishErr + } + return core.LeaseTarget{}, b.handleFailedAcquire(instanceID, leaseID, slug, cfg, req.Repo.Root, req.Keep, publishErr) + } + fmt.Fprintf(b.rt.Stderr, "provisioned lease=%s fal=%s state=ready\n", leaseID, instanceID) + core.SetServerLeaseClaimSnapshot(&server, readyClaim, true) + delete(server.Labels, falAcquireLifetimeLabel) + target := core.LeaseTarget{Server: server, SSH: ssh, LeaseID: leaseID} + return target, nil +} + +func (b *backend) reconcileFalReadyClaimWrite(ctx context.Context, client computeAPI, expected, published core.LeaseClaim, instanceID string, writeErr error) error { + if published.LeaseID != "" && core.VerifyLeaseClaimUnchanged(published.LeaseID, published) == nil { + if _, retryErr := replaceFalClaimDurably(published, published); retryErr == nil { + return nil + } else { + return errors.Join(writeErr, retryErr) + } + } + current, exists, readErr := core.ReadLeaseClaimWithPresence(expected.LeaseID) + if readErr != nil { + return errors.Join(writeErr, readErr) + } + if !exists { + keyAbsent, keyErr := falLeaseKeyAbsent(expected.LeaseID) + if keyErr != nil { + return errors.Join(writeErr, keyErr) + } + if !keyAbsent { + return writeErr + } + live, getErr := client.GetInstance(ctx, instanceID) + if isFalNotFound(getErr) { + ids, inventoryErr := falInventoryIDs(ctx, client) + if inventoryErr != nil { + return errors.Join(writeErr, inventoryErr) + } + for _, id := range ids { + if id == instanceID { + return writeErr + } + } + return fmt.Errorf("%w for lease %s: %v", errFalClaimMutationSuperseded, expected.LeaseID, writeErr) + } + if getErr != nil { + return errors.Join(writeErr, getErr) + } + if strings.TrimSpace(live.ID) != instanceID { + return errors.Join(writeErr, exit(2, "fal ready-claim recovery returned changed instance identity for %s", instanceID)) + } + return writeErr + } + if falDeletionInProgress(current) || core.VerifyLeaseClaimUnchanged(expected.LeaseID, expected) != nil { + return fmt.Errorf("%w for lease %s: %v", errFalClaimMutationSuperseded, expected.LeaseID, writeErr) + } + return writeErr +} + +func (b *backend) reconcileAmbiguousCreate(ctx context.Context, client computeAPI, req CreateInstanceRequest, claim core.LeaseClaim, cfg Config, keep bool, cause error) (ComputeInstance, core.LeaseClaim, error) { + var lastErr error + for attempt := 1; attempt <= falCreateReconcileAttempts; attempt++ { + instance, updated, ambiguous, err := b.replayFalCreateWithClaim(ctx, client, req, claim, cfg, "provisioning", keep) + claim = updated + if err == nil && strings.TrimSpace(instance.ID) != "" { + return instance, claim, nil + } + if strings.TrimSpace(instance.ID) != "" { + return instance, claim, errors.Join(cause, fmt.Errorf("persist fal idempotent create result: %w", err)) + } + lastErr = err + if !ambiguous { + return ComputeInstance{}, claim, errors.Join(cause, fmt.Errorf("fal idempotent create retry failed: %w", err)) + } + createStarted := falClaimStartedAt(claim, time.Time{}) + if createStarted.IsZero() || !b.now().Before(createStarted.Add(falCreateRecoveryWindow)) { + return ComputeInstance{}, claim, errors.Join( + fmt.Errorf("fal instance creation remains indeterminate after the provider idempotency replay window expired; no provider id was returned"), + cause, + lastErr, + ) + } + if attempt == falCreateReconcileAttempts { + break + } + timer := time.NewTimer(falCreateReconcileRetryWait) + select { + case <-ctx.Done(): + timer.Stop() + return ComputeInstance{}, claim, errors.Join(cause, ctx.Err()) + case <-timer.C: + } + } + return ComputeInstance{}, claim, errors.Join( + fmt.Errorf("fal instance creation remains indeterminate after idempotent retry; no provider id was returned"), + cause, + lastErr, + ) +} + +func (b *backend) replayFalCreateWithClaim(ctx context.Context, client computeAPI, req CreateInstanceRequest, claim core.LeaseClaim, cfg Config, successReason string, keep bool) (ComputeInstance, core.LeaseClaim, bool, error) { + if claim.LeaseID == "" || claim.CloudID != "" { + return ComputeInstance{}, claim, false, exit(2, "fal create replay requires an identityless recovery claim") + } + if claim.Provider != providerName || claim.ProviderScope != falClaimScope(cfg) { + return ComputeInstance{}, claim, false, exit(2, "fal lease %s create replay ownership changed", claim.LeaseID) + } + if err := verifyFalClaimCredential(claim, cfg); err != nil { + return ComputeInstance{}, claim, false, err + } + if err := verifyFalCreateRequestBinding(claim, req); err != nil { + return ComputeInstance{}, claim, false, err + } + unlock, err := lockFalLeaseOperation(ctx, claim.LeaseID) + if err != nil { + return ComputeInstance{}, claim, false, err + } + defer unlock() + + if err := core.VerifyLeaseClaimUnchanged(claim.LeaseID, claim); err != nil { + return ComputeInstance{}, claim, false, err + } + current, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if err != nil { + return ComputeInstance{}, claim, false, err + } + if !exists { + return ComputeInstance{}, core.LeaseClaim{}, false, errFalRecoveryClaimRemoved + } + if current.Provider != providerName || current.ProviderScope != falClaimScope(cfg) { + return ComputeInstance{}, current, false, exit(2, "fal lease %s create replay ownership changed", current.LeaseID) + } + if err := verifyFalClaimCredential(current, cfg); err != nil { + return ComputeInstance{}, current, false, err + } + if err := verifyFalCreateRequestBinding(current, req); err != nil { + return ComputeInstance{}, current, false, err + } + claim = current + recovery := strings.TrimSpace(claim.Labels["recovery"]) + if recovery != "create-intent" && recovery != "ambiguous-create" && recovery != "ambiguous-create-inflight" { + return ComputeInstance{}, claim, false, exit(4, "fal recovery is still pending for lease=%s; local recovery state retained", claim.LeaseID) + } + createStarted := falClaimStartedAt(claim, time.Time{}) + createCtx, cancel, err := b.createReplayContext(ctx, createStarted) + if err != nil { + return ComputeInstance{}, claim, false, exit(5, "fal create recovery window expired for lease=%s; local recovery claim retained for manual provider reconciliation", claim.LeaseID) + } + defer cancel() + hadPriorAttempt := recovery != "create-intent" || strings.TrimSpace(claim.Labels[falCreateAttemptLabel]) != "" + + inflight, err := b.newFalRecoveryClaim(claim, cfg, "", "ambiguous-create-inflight", keep) + if err != nil { + return ComputeInstance{}, claim, false, err + } + inflight.Labels[falCreateAttemptLabel] = core.NewLeaseID() + if err := core.ReplaceLeaseClaimIfUnchangedDurable(claim.LeaseID, claim, inflight); err != nil { + return ComputeInstance{}, claim, false, fmt.Errorf("claim fal create replay attempt: %w", err) + } + + var created ComputeInstance + var createErr error + created, createErr = client.CreateInstance(createCtx, req, claim.LeaseID) + instanceID := strings.TrimSpace(created.ID) + if createErr == nil && instanceID == "" { + createErr = exit(5, "fal idempotent create returned an empty id") + } + updated := core.LeaseClaim{} + exists = true + var mutationErr error + if instanceID == "" && createErr != nil && !isAmbiguousFalMutationError(createErr) && !hadPriorAttempt { + updated, mutationErr = b.newFalRecoveryClaim(inflight, cfg, "", falCreateRejectedRecovery, keep) + if mutationErr == nil { + updated.Labels[falCreateAttemptLabel] = inflight.Labels[falCreateAttemptLabel] + mutationErr = core.ReplaceLeaseClaimIfUnchangedDurable(claim.LeaseID, inflight, updated) + } + if mutationErr == nil { + mutationErr = core.RemoveLeaseClaimIfUnchangedAfter(claim.LeaseID, updated, func() error { + return b.removeFalLeaseKey(claim.LeaseID) + }) + } + if mutationErr == nil { + exists = false + } + } else { + reason := successReason + if instanceID == "" { + reason = "ambiguous-create" + } + updated, mutationErr = b.newFalRecoveryClaim(inflight, cfg, instanceID, reason, keep) + if mutationErr == nil { + updated.Labels[falCreateAttemptLabel] = inflight.Labels[falCreateAttemptLabel] + mutationErr = core.ReplaceLeaseClaimIfUnchangedDurable(claim.LeaseID, inflight, updated) + } + } + if mutationErr != nil { + if current, currentExists, readErr := core.ReadLeaseClaimWithPresence(claim.LeaseID); readErr == nil && currentExists && + current.Provider == providerName && current.ProviderScope == falClaimScope(cfg) && verifyFalClaimCredential(current, cfg) == nil { + updated = current + exists = true + } + outcomeErr := fmt.Errorf("persist fal create replay outcome: %w", mutationErr) + if createErr != nil { + outcomeErr = errors.Join(createErr, outcomeErr) + } + return created, updated, createErr != nil && isAmbiguousFalMutationError(createErr), outcomeErr + } + if strings.TrimSpace(created.ID) == "" && createErr != nil && !isAmbiguousFalMutationError(createErr) { + return ComputeInstance{}, updated, false, createErr + } + if strings.TrimSpace(created.ID) != "" && createErr != nil { + return created, updated, false, createErr + } + if createErr != nil { + return ComputeInstance{}, updated, true, createErr + } + return created, updated, false, nil +} + +func (b *backend) Resolve(ctx context.Context, req core.ResolveRequest) (core.LeaseTarget, error) { + cfg := b.configForRun() + claim, ok, err := resolveFalClaim(req.ID, falClaimScope(cfg)) + if err != nil { + return core.LeaseTarget{}, err + } + if !ok { + return core.LeaseTarget{}, exit(4, "lease/fal instance not found or not locally claimed: %s", strings.TrimSpace(req.ID)) + } + confirmedDeletionID := falConfirmedDeletionInstanceID(claim) + if confirmedDeletionID != "" { + if req.ReleaseOnly { + return leaseTargetFromClaim(claim, cfg, false) + } + return core.LeaseTarget{}, exit(4, "fal lease %s deletion is confirmed; retry stop to finish local cleanup", claim.LeaseID) + } + client, err := b.api() + if err != nil { + return core.LeaseTarget{}, err + } + rebindMayPersist := !req.NoLocalStateMutations && claim.CloudID != "" && verifyFalClaimCredential(claim, cfg) != nil + needsAcquireSerialization := req.ReleaseOnly && falClaimMayBeAcquiring(claim) || rebindMayPersist + if needsAcquireSerialization && !falDeletionInProgress(claim) { + unlockAcquire, inactive, lockErr := tryLockFalAcquireLifetime(ctx, claim.LeaseID) + if lockErr != nil { + return core.LeaseTarget{}, lockErr + } + if !inactive { + return core.LeaseTarget{}, exit(4, "fal lease %s acquisition is still in progress; retry after acquisition completes", claim.LeaseID) + } + defer unlockAcquire() + if err := core.VerifyLeaseClaimUnchanged(claim.LeaseID, claim); err != nil { + return core.LeaseTarget{}, err + } + } + claim, err = b.ensureFalClaimCredential(ctx, client, claim, cfg, !req.NoLocalStateMutations) + if err != nil { + return core.LeaseTarget{}, err + } + if falDeletionInProgress(claim) { + if req.ReleaseOnly { + return leaseTargetFromClaim(claim, cfg, false) + } + return core.LeaseTarget{}, exit(4, "fal lease %s deletion is in progress; retry stop to finish cleanup", claim.LeaseID) + } + if claim.Labels["recovery"] == "provisioning" && !req.ReleaseOnly { + return core.LeaseTarget{}, exit(4, "fal lease %s is still provisioning; retry after acquisition completes", claim.LeaseID) + } + if claim.CloudID == "" { + if !req.ReleaseOnly { + return core.LeaseTarget{}, exit(4, "fal recovery is still pending for lease=%s; local recovery state retained", claim.LeaseID) + } + if claim.Labels["recovery"] == "create-intent" || claim.Labels["recovery"] == falCreateRejectedRecovery { + return leaseTargetFromClaim(claim, cfg, false) + } + claim, err = b.recoverAmbiguousCreateForRelease(ctx, client, claim, cfg) + if err != nil { + return core.LeaseTarget{}, err + } + } + instance, err := client.GetInstance(ctx, claim.CloudID) + if err != nil { + if req.ReleaseOnly && isFalNotFound(err) { + return leaseTargetFromClaim(claim, cfg, false) + } + return core.LeaseTarget{}, err + } + includeSSH := !req.ReleaseOnly && (!req.StatusOnly || (req.ReadyProbe && strings.TrimSpace(instance.IP) != "")) + target, err := leaseTargetFromClaimedInstance(instance, claim, cfg, includeSSH) + if err != nil { + return core.LeaseTarget{}, err + } + if req.ReleaseOnly { + target.SSH = core.SSHTarget{} + return target, nil + } + if req.Repo.Root != "" && !req.NoLocalStateMutations { + claimServer, err := falClaimServer(target.Server, cfg) + if err != nil { + return core.LeaseTarget{}, err + } + if _, err := core.ClaimLeaseTargetForRepoConfigScopeIfUnchangedDurable(target.LeaseID, claim.Slug, cfg, falClaimScope(cfg), claimServer, target.SSH, req.Repo.Root, cfg.IdleTimeout, req.Reclaim, claim, true); err != nil { + return core.LeaseTarget{}, err + } + } + return target, nil +} + +func (b *backend) List(ctx context.Context, _ core.ListRequest) ([]core.LeaseView, error) { + cfg := b.configForRun() + claims, err := falClaims(falClaimScope(cfg)) + if err != nil { + return nil, err + } + views := make([]core.LeaseView, 0, len(claims)) + needsProvider := false + for _, claim := range claims { + if err := verifyFalClaimCredential(claim, cfg); err != nil { + server, viewErr := falClaimView(claim, cfg, "credential-binding-mismatch") + if viewErr != nil { + return nil, viewErr + } + views = append(views, server) + continue + } + if claim.CloudID == "" { + server, err := falClaimView(claim, cfg, firstNonBlank(claim.Labels["recovery"], "recovery-pending")) + if err != nil { + return nil, err + } + views = append(views, server) + continue + } + needsProvider = true + } + if !needsProvider { + return views, nil + } + client, apiErr := b.api() + for _, claim := range claims { + if claim.CloudID == "" { + continue + } + if verifyFalClaimCredential(claim, cfg) != nil { + continue + } + if apiErr != nil { + server, viewErr := falClaimView(claim, cfg, "provider-verification-unavailable") + if viewErr != nil { + return nil, viewErr + } + views = append(views, server) + continue + } + instance, err := client.GetInstance(ctx, claim.CloudID) + if isFalNotFound(err) { + server, viewErr := falClaimView(claim, cfg, "provider-absence-unverified") + if viewErr != nil { + return nil, viewErr + } + views = append(views, server) + continue + } + if err != nil { + server, viewErr := falClaimView(claim, cfg, "provider-verification-unavailable") + if viewErr != nil { + return nil, viewErr + } + views = append(views, server) + continue + } + target, err := leaseTargetFromClaimedInstance(instance, claim, cfg, false) + if err != nil { + return nil, err + } + views = append(views, target.Server) + } + return views, nil +} + +func falClaimView(claim core.LeaseClaim, cfg Config, status string) (core.LeaseView, error) { + server, err := serverFromClaim(claim, cfg) + if err != nil { + return core.LeaseView{}, err + } + server.Status = status + server.Labels = cloneLabels(server.Labels) + server.Labels["state"] = status + return server, nil +} + +func (b *backend) Touch(_ context.Context, req core.TouchRequest) (core.Server, error) { + server := req.Lease.Server + if req.Lease.LeaseID == "" { + return core.Server{}, exit(2, "provider=%s touch requires a lease id", providerName) + } + claim, ok, err := core.ReadLeaseClaimWithPresence(req.Lease.LeaseID) + if err != nil { + return core.Server{}, err + } + if !ok || claim.Provider != providerName { + return core.Server{}, exit(2, "no local claim for fal lease %s", req.Lease.LeaseID) + } + cfg := b.configForRun() + if claim.ProviderScope != falClaimScope(cfg) { + return core.Server{}, exit(2, "fal lease %s belongs to a different API endpoint; refusing to touch it", req.Lease.LeaseID) + } + if err := verifyFalClaimCredential(claim, cfg); err != nil { + return core.Server{}, err + } + if falDeletionInProgress(claim) { + return core.Server{}, exit(4, "fal lease %s deletion is in progress; refusing to update it", claim.LeaseID) + } + if claim.Labels["recovery"] == "provisioning" { + return core.Server{}, exit(4, "fal lease %s is still provisioning; refusing to update it", claim.LeaseID) + } + if req.IdleTimeout > 0 { + cfg.IdleTimeout = req.IdleTimeout + } + labels := cloneLabels(server.Labels) + if len(labels) == 0 { + labels = cloneLabels(claim.Labels) + } + server.Labels = core.TouchDirectLeaseLabels(labels, cfg, req.State, b.now()) + claimLabels := cloneLabels(server.Labels) + claimLabels[falCredentialBindingLabel] = claim.Labels[falCredentialBindingLabel] + claimLabels[falAcquireLifetimeLabel] = claim.Labels[falAcquireLifetimeLabel] + replacement := claim + replacement.Labels = claimLabels + if err := core.ReplaceLeaseClaimIfUnchangedDurable(req.Lease.LeaseID, claim, replacement); err != nil { + return core.Server{}, err + } + delete(server.Labels, falCredentialBindingLabel) + delete(server.Labels, falAcquireLifetimeLabel) + return server, nil +} + +func (b *backend) ReleaseLease(ctx context.Context, req core.ReleaseLeaseRequest) error { + if err := core.ValidateLeaseTargetProviderIdentity(req.Lease, req.ExpectedProviderIdentity); err != nil { + return err + } + leaseID := strings.TrimSpace(req.Lease.LeaseID) + if leaseID == "" { + leaseID = req.Lease.Server.Labels["lease"] + } + if leaseID == "" { + return exit(2, "provider=%s release requires a lease id", providerName) + } + claim, ok, err := core.ReadLeaseClaimWithPresence(leaseID) + if err != nil { + return err + } + if !ok || claim.Provider != providerName { + return exit(2, "no local claim for fal lease %s; refusing to delete provider resources", leaseID) + } + cfg := b.configForRun() + if claim.ProviderScope != falClaimScope(cfg) { + return exit(2, "fal lease %s belongs to a different API endpoint; refusing to delete provider resources", leaseID) + } + confirmedDeletionID := falConfirmedDeletionInstanceID(claim) + if confirmedDeletionID != "" { + if req.Lease.Server.CloudID != "" && confirmedDeletionID != req.Lease.Server.CloudID { + return exit(2, "refusing to finalize fal instance %s from stale local claim", req.Lease.Server.CloudID) + } + return b.finalizeAcceptedFalDeletion(claim, confirmedDeletionID) + } + if falClaimMayBeAcquiring(claim) && !falDeletionInProgress(claim) { + unlockAcquire, inactive, err := tryLockFalAcquireLifetime(ctx, leaseID) + if err != nil { + return err + } + if !inactive { + return exit(4, "fal lease %s acquisition is still in progress; retry stop after acquisition completes", leaseID) + } + defer unlockAcquire() + if err := core.VerifyLeaseClaimUnchanged(leaseID, claim); err != nil { + return err + } + } + var client computeAPI + if err := verifyFalClaimCredential(claim, cfg); err != nil { + if claim.CloudID == "" { + return err + } + client, err = b.api() + if err != nil { + return err + } + claim, err = b.ensureFalClaimCredential(ctx, client, claim, cfg, true) + if err != nil { + return err + } + } + if claim.CloudID == "" { + if claim.Labels["recovery"] == "create-intent" || claim.Labels["recovery"] == falCreateRejectedRecovery { + if req.Lease.Server.CloudID != "" { + return exit(2, "refusing to cancel fal create intent %s from stale provider identity", leaseID) + } + return core.RemoveLeaseClaimIfUnchangedAfter(leaseID, claim, func() error { + return core.RemoveStoredTestboxKeyWithError(leaseID) + }) + } + return exit(4, "fal recovery is still pending for lease=%s; local recovery state retained", leaseID) + } + if req.Lease.Server.CloudID != "" && claim.CloudID != req.Lease.Server.CloudID { + return exit(2, "refusing to release fal instance %s from stale local claim", req.Lease.Server.CloudID) + } + instanceID := claim.CloudID + if client == nil { + client, err = b.api() + if err != nil { + return err + } + } + return b.deleteClaimedFalInstance(ctx, client, claim, cfg, instanceID) +} + +func (b *backend) Cleanup(ctx context.Context, req core.CleanupRequest) error { + cfg := b.configForRun() + claims, err := falClaims(falClaimScope(cfg)) + if err != nil { + return err + } + if len(claims) == 0 { + return nil + } + var client computeAPI + cleanupClient := func() (computeAPI, error) { + if client != nil { + return client, nil + } + var clientErr error + client, clientErr = b.api() + return client, clientErr + } + var cleanupErr error + for _, claim := range claims { + if err := b.cleanupFalClaim(ctx, req, claim, cfg, cleanupClient); err != nil { + cleanupErr = errors.Join(cleanupErr, fmt.Errorf("clean up fal lease %s: %w", claim.LeaseID, err)) + } + if cause := context.Cause(ctx); cause != nil { + if !errors.Is(cleanupErr, cause) { + cleanupErr = errors.Join(cleanupErr, cause) + } + break + } + } + return cleanupErr +} + +func (b *backend) cleanupFalClaim(ctx context.Context, req core.CleanupRequest, claim core.LeaseClaim, cfg Config, cleanupClient func() (computeAPI, error)) error { + server, err := serverFromClaim(claim, cfg) + if err != nil { + return err + } + recovery := strings.TrimSpace(claim.Labels["recovery"]) + deletionInProgress := falDeletionInProgress(claim) + forcedCleanup := deletionInProgress || recovery == "rollback-cleanup" || recovery == falCreateRejectedRecovery + if strings.EqualFold(strings.TrimSpace(claim.Labels["keep"]), "true") && !forcedCleanup { + fmt.Fprintf(b.rt.Stderr, "skip server id=%s name=%s reason=keep=true\n", server.DisplayID(), server.Name) + return nil + } + if !forcedCleanup && falRecoveryNeedsAcquireLifetime(recovery) && claim.Labels[falAcquireLifetimeLabel] != falAcquireLifetimeVersion { + // Claims written before the lifetime-lock protocol may belong to an + // acquisition still running under an older binary. Retain them rather + // than treating the absence of a new lock file as proof of abandonment. + fmt.Fprintf(b.rt.Stderr, "skip server id=%s name=%s reason=legacy_recovery_pending\n", server.DisplayID(), server.Name) + return nil + } + + // A durable rollback or delete marker supersedes acquisition liveness; the + // per-lease operation lock and claim CAS serialize the forced cleanup. + if !forcedCleanup { + unlockAcquire, inactive, err := tryLockFalAcquireLifetime(ctx, claim.LeaseID) + if err != nil { + return err + } + if !inactive { + fmt.Fprintf(b.rt.Stderr, "skip server id=%s name=%s reason=acquire-in-progress\n", server.DisplayID(), server.Name) + return nil + } + defer unlockAcquire() + } + if err := core.VerifyLeaseClaimUnchanged(claim.LeaseID, claim); err != nil { + return err + } + confirmedDeletionID := falConfirmedDeletionInstanceID(claim) + if confirmedDeletionID != "" { + if req.DryRun { + fmt.Fprintf(b.rt.Stderr, "delete server id=%s name=%s reason=deletion-confirmed\n", server.DisplayID(), server.Name) + return nil + } + if err := b.finalizeAcceptedFalDeletion(claim, confirmedDeletionID); err != nil { + return err + } + fmt.Fprintf(b.rt.Stderr, "delete server id=%s name=%s reason=deletion-confirmed\n", server.DisplayID(), server.Name) + return nil + } + if err := verifyFalClaimCredential(claim, cfg); err != nil { + if claim.CloudID == "" { + fmt.Fprintf(b.rt.Stderr, "skip server id=%s name=%s reason=credential_binding_mismatch\n", firstNonBlank(claim.CloudID, claim.LeaseID), firstNonBlank(claim.Slug, claim.LeaseID)) + return nil + } + if falCredentialBinding(cfg) == "" { + fmt.Fprintf(b.rt.Stderr, "skip server id=%s name=%s reason=credential_binding_mismatch\n", firstNonBlank(claim.CloudID, claim.LeaseID), firstNonBlank(claim.Slug, claim.LeaseID)) + return nil + } + client, clientErr := cleanupClient() + if clientErr != nil { + return clientErr + } + updatedClaim, ensureErr := b.ensureFalClaimCredential(ctx, client, claim, cfg, !req.DryRun) + if ensureErr != nil { + if errors.Is(ensureErr, errFalCredentialBindingMismatch) { + fmt.Fprintf(b.rt.Stderr, "skip server id=%s name=%s reason=credential_binding_mismatch\n", firstNonBlank(claim.CloudID, claim.LeaseID), firstNonBlank(claim.Slug, claim.LeaseID)) + return nil + } + return ensureErr + } + claim = updatedClaim + } + + if claim.CloudID == "" { + switch recovery { + case "create-intent", falCreateRejectedRecovery: + if req.DryRun { + fmt.Fprintf(b.rt.Stderr, "delete server id=%s name=%s reason=%s\n", server.DisplayID(), server.Name, recovery) + return nil + } + if err := b.removeAbandonedFalCreateClaim(ctx, claim, cfg); err != nil { + return err + } + fmt.Fprintf(b.rt.Stderr, "delete server id=%s name=%s reason=%s\n", server.DisplayID(), server.Name, recovery) + return nil + case "ambiguous-create", "ambiguous-create-inflight": + if err := b.validateFalCreateRecoveryWindow(claim); err != nil { + return err + } + if req.DryRun { + if _, _, err := falCreateRecoveryRequest(claim, cfg); err != nil { + return err + } + fmt.Fprintf(b.rt.Stderr, "delete server id=%s name=%s reason=recover_ambiguous_create\n", server.DisplayID(), server.Name) + return nil + } + client, err := cleanupClient() + if err != nil { + return err + } + recovered, err := b.recoverAmbiguousCreateForRelease(ctx, client, claim, cfg) + if err != nil { + return err + } + if strings.TrimSpace(recovered.CloudID) == "" { + return exit(5, "fal cleanup recovered no provider identity for lease=%s; local recovery claim retained for manual provider reconciliation", claim.LeaseID) + } + fmt.Fprintf(b.rt.Stderr, "delete server id=%s name=%s reason=recovered_ambiguous_create\n", recovered.CloudID, server.Name) + return b.deleteClaimedFalInstance(ctx, client, recovered, cfg, recovered.CloudID) + default: + fmt.Fprintf(b.rt.Stderr, "skip server id=%s name=%s reason=recovery_pending\n", server.DisplayID(), server.Name) + return nil + } + } + + remove := recovery == "rollback-cleanup" || recovery == "provisioning" || deletionInProgress + reason := "rollback-cleanup" + if deletionInProgress { + reason = "deletion-in-progress" + } else if recovery == "provisioning" { + reason = "abandoned-provisioning" + } else if !remove { + remove, reason = core.ShouldCleanupServer(server, b.now()) + } + if !remove { + fmt.Fprintf(b.rt.Stderr, "skip server id=%s name=%s reason=%s\n", server.DisplayID(), server.Name, reason) + return nil + } + client, err := cleanupClient() + if err != nil { + return err + } + verifyLive := func() error { + live, getErr := client.GetInstance(ctx, claim.CloudID) + if isFalNotFound(getErr) { + return fmt.Errorf("%w: %s", errFalProviderAbsenceNotAccountBound, claim.CloudID) + } + if getErr != nil { + return getErr + } + if strings.TrimSpace(live.ID) != claim.CloudID { + return exit(2, "refusing cleanup for fal lease %s after provider identity changed", claim.LeaseID) + } + return nil + } + if req.DryRun { + if err := verifyLive(); err != nil { + if errors.Is(err, errFalProviderAbsenceNotAccountBound) { + fmt.Fprintf(b.rt.Stderr, "skip server id=%s name=%s reason=provider_absence_not_account_bound\n", server.DisplayID(), server.Name) + return nil + } + return err + } + fmt.Fprintf(b.rt.Stderr, "delete server id=%s name=%s\n", claim.CloudID, server.Name) + return nil + } + fmt.Fprintf(b.rt.Stderr, "delete server id=%s name=%s\n", claim.CloudID, server.Name) + err = b.deleteClaimedFalInstance(ctx, client, claim, cfg, claim.CloudID) + if errors.Is(err, errFalProviderAbsenceNotAccountBound) { + fmt.Fprintf(b.rt.Stderr, "skip server id=%s name=%s reason=provider_absence_not_account_bound\n", server.DisplayID(), server.Name) + return nil + } + return err +} + +func falRecoveryNeedsAcquireLifetime(recovery string) bool { + switch strings.TrimSpace(recovery) { + case "create-intent", "ambiguous-create", "ambiguous-create-inflight", "provisioning": + return true + default: + return false + } +} + +func falAmbiguousCreateClaim(claim core.LeaseClaim) bool { + recovery := strings.TrimSpace(claim.Labels["recovery"]) + return claim.CloudID == "" && (recovery == "ambiguous-create" || recovery == "ambiguous-create-inflight") +} + +func (b *backend) removeAbandonedFalCreateClaim(ctx context.Context, expected core.LeaseClaim, cfg Config) error { + unlock, err := lockFalLeaseOperation(ctx, expected.LeaseID) + if err != nil { + return err + } + defer unlock() + if err := core.VerifyLeaseClaimUnchanged(expected.LeaseID, expected); err != nil { + return err + } + recovery := strings.TrimSpace(expected.Labels["recovery"]) + if expected.Provider != providerName || expected.ProviderScope != falClaimScope(cfg) || expected.CloudID != "" || + (recovery != "create-intent" && recovery != falCreateRejectedRecovery) { + return exit(2, "fal lease %s abandoned create ownership changed; refusing local cleanup", expected.LeaseID) + } + if strings.EqualFold(strings.TrimSpace(expected.Labels["keep"]), "true") && recovery != falCreateRejectedRecovery { + return exit(2, "fal lease %s changed to keep=true; refusing local cleanup", expected.LeaseID) + } + if err := verifyFalClaimCredential(expected, cfg); err != nil { + return err + } + return core.RemoveLeaseClaimIfUnchangedAfter(expected.LeaseID, expected, func() error { + return b.removeFalLeaseKey(expected.LeaseID) + }) +} + +func (b *backend) validateFalCreateRecoveryWindow(claim core.LeaseClaim) error { + createdAt, err := strconv.ParseInt(strings.TrimSpace(claim.Labels["create_started_at"]), 10, 64) + if err != nil || createdAt <= 0 || !b.now().Before(time.Unix(createdAt, 0).Add(falCreateRecoveryWindow)) { + return exit(5, "fal create recovery window expired for lease=%s; local recovery claim retained for manual provider reconciliation", claim.LeaseID) + } + return nil +} + +func (b *backend) deleteClaimedFalInstance(ctx context.Context, client computeAPI, expected core.LeaseClaim, cfg Config, instanceID string) error { + instanceID = strings.TrimSpace(instanceID) + if expected.LeaseID == "" || instanceID == "" { + return exit(2, "fal claimed delete requires lease and instance identities") + } + unlock, err := lockFalLeaseOperation(ctx, expected.LeaseID) + if err != nil { + return err + } + defer unlock() + if err := core.VerifyLeaseClaimUnchanged(expected.LeaseID, expected); err != nil { + return err + } + if expected.Provider != providerName || expected.ProviderScope != falClaimScope(cfg) { + return exit(2, "fal lease %s delete ownership changed; refusing instance %s", expected.LeaseID, instanceID) + } + if expected.CloudID != "" && expected.CloudID != instanceID { + return exit(2, "fal lease %s delete instance changed from %s to %s", expected.LeaseID, expected.CloudID, instanceID) + } + if expected.CloudID == "" && !falUnboundCleanupClaim(expected) { + return exit(2, "fal lease %s has no durable create-attempt ownership for instance %s", expected.LeaseID, instanceID) + } + if falDeleteConfirmed(expected, instanceID) { + return b.finalizeAcceptedFalDeletion(expected, instanceID) + } + if err := verifyFalClaimCredential(expected, cfg); err != nil { + return err + } + live, err := client.GetInstance(ctx, instanceID) + if isFalNotFound(err) { + if falDeleteAccepted(expected, instanceID) { + if err := b.confirmFalInstanceDeletion(ctx, client, instanceID); err != nil { + return err + } + confirmed, stateErr := persistFalDeleteState(expected, falDeleteConfirmedLabel, instanceID) + if stateErr != nil { + return stateErr + } + return b.finalizeAcceptedFalDeletion(confirmed, instanceID) + } + if falDeleteStateMatches(expected, falDeleteAttemptLabel, instanceID) { + if err := b.confirmFalInstanceDeletion(ctx, client, instanceID); err != nil { + return errors.Join(fmt.Errorf("%w: %s", errFalProviderAbsenceNotAccountBound, instanceID), err) + } + confirmed, stateErr := persistFalDeleteState(expected, falDeleteConfirmedLabel, instanceID) + if stateErr != nil { + return stateErr + } + return b.finalizeAcceptedFalDeletion(confirmed, instanceID) + } + if err := b.confirmFalInstanceDeletion(ctx, client, instanceID); err != nil { + return errors.Join(fmt.Errorf("%w: %s", errFalProviderAbsenceNotAccountBound, instanceID), err) + } + accepted, stateErr := persistFalDeleteState(expected, falDeleteConfirmedLabel, instanceID) + if stateErr != nil { + return stateErr + } + return b.finalizeAcceptedFalDeletion(accepted, instanceID) + } + if err != nil { + return err + } + if strings.TrimSpace(live.ID) != instanceID { + return exit(2, "refusing to delete fal instance %s after provider identity changed", instanceID) + } + if falDeleteAccepted(expected, instanceID) { + return b.confirmAndFinalizeFalDeletion(ctx, client, expected, instanceID) + } + attempted, err := persistFalDeleteState(expected, falDeleteAttemptLabel, instanceID) + if err != nil { + return err + } + deleteErr := client.DeleteInstance(ctx, instanceID) + if deleteErr != nil && !isFalNotFound(deleteErr) { + return deleteErr + } + if isFalNotFound(deleteErr) { + if err := b.confirmFalInstanceDeletion(ctx, client, instanceID); err != nil { + return err + } + accepted, err := persistFalDeleteState(attempted, falDeleteConfirmedLabel, instanceID) + if err != nil { + return err + } + return b.finalizeAcceptedFalDeletion(accepted, instanceID) + } + accepted, err := persistFalDeleteState(attempted, falDeleteAcceptedLabel, instanceID) + if err != nil { + return err + } + return b.confirmAndFinalizeFalDeletion(ctx, client, accepted, instanceID) +} + +func (b *backend) confirmAndFinalizeFalDeletion(ctx context.Context, client computeAPI, accepted core.LeaseClaim, instanceID string) error { + if err := b.confirmFalInstanceDeletion(ctx, client, instanceID); err != nil { + return err + } + confirmed, err := persistFalDeleteState(accepted, falDeleteConfirmedLabel, instanceID) + if err != nil { + return err + } + return b.finalizeAcceptedFalDeletion(confirmed, instanceID) +} + +func (b *backend) confirmFalInstanceDeletion(ctx context.Context, client computeAPI, instanceID string) error { + timeout := falDeleteConfirmationTimeout + if b.pollTimeout > 0 && b.pollTimeout < timeout { + timeout = b.pollTimeout + } + confirmCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), timeout) + defer cancel() + if err := b.waitForFalInstanceAbsence(confirmCtx, client, instanceID); err != nil { + return fmt.Errorf("confirm fal instance %s deletion: %w", instanceID, err) + } + return nil +} + +func persistFalDeleteState(expected core.LeaseClaim, label, instanceID string) (core.LeaseClaim, error) { + updated := expected + if updated.CloudID == "" { + updated.CloudID = strings.TrimSpace(instanceID) + } + updated.Labels = cloneLabels(expected.Labels) + updated.Labels[falDeleteAttemptLabel] = instanceID + if label == falDeleteAcceptedLabel { + updated.Labels[falDeleteAcceptedLabel] = instanceID + } + if label == falDeleteConfirmedLabel { + updated.Labels[falDeleteAttemptLabel] = instanceID + updated.Labels[falDeleteAcceptedLabel] = instanceID + updated.Labels[falDeleteConfirmedLabel] = instanceID + } + return replaceFalClaimDurably(expected, updated) +} + +func replaceFalClaimDurably(expected, updated core.LeaseClaim) (core.LeaseClaim, error) { + if err := core.ReplaceLeaseClaimIfUnchangedDurable(expected.LeaseID, expected, updated); err != nil { + if verifyErr := core.VerifyLeaseClaimUnchanged(expected.LeaseID, updated); verifyErr != nil { + return core.LeaseClaim{}, errors.Join(err, verifyErr) + } + if retryErr := core.ReplaceLeaseClaimIfUnchangedDurable(expected.LeaseID, updated, updated); retryErr != nil { + return core.LeaseClaim{}, errors.Join(err, retryErr) + } + } + return updated, nil +} + +func falDeleteStateMatches(claim core.LeaseClaim, label, instanceID string) bool { + return strings.TrimSpace(instanceID) != "" && strings.TrimSpace(claim.Labels[label]) == strings.TrimSpace(instanceID) +} + +func falDeletionInProgress(claim core.LeaseClaim) bool { + return strings.TrimSpace(claim.Labels[falDeleteAttemptLabel]) != "" || strings.TrimSpace(claim.Labels[falDeleteAcceptedLabel]) != "" +} + +func falDeleteAccepted(claim core.LeaseClaim, instanceID string) bool { + return falDeleteStateMatches(claim, falDeleteAttemptLabel, instanceID) && + falDeleteStateMatches(claim, falDeleteAcceptedLabel, instanceID) +} + +func falDeleteConfirmed(claim core.LeaseClaim, instanceID string) bool { + return falDeleteAccepted(claim, instanceID) && falDeleteStateMatches(claim, falDeleteConfirmedLabel, instanceID) +} + +func falConfirmedDeletionInstanceID(claim core.LeaseClaim) string { + instanceID := strings.TrimSpace(claim.Labels[falDeleteConfirmedLabel]) + if falDeleteConfirmed(claim, instanceID) { + return instanceID + } + return "" +} + +func falClaimMayBeAcquiring(claim core.LeaseClaim) bool { + return claim.Labels[falAcquireLifetimeLabel] == falAcquireLifetimeVersion || falRecoveryNeedsAcquireLifetime(claim.Labels["recovery"]) +} + +func (b *backend) finalizeAcceptedFalDeletion(claim core.LeaseClaim, instanceID string) error { + if !falDeleteConfirmed(claim, instanceID) { + return exit(2, "fal lease %s deletion is not durably confirmed", claim.LeaseID) + } + return core.RemoveLeaseClaimIfUnchangedAfter(claim.LeaseID, claim, func() error { + return b.removeFalLeaseKey(claim.LeaseID) + }) +} + +func (b *backend) removeFalLeaseKey(leaseID string) error { + if b.removeLeaseKey != nil { + return b.removeLeaseKey(leaseID) + } + return core.RemoveStoredTestboxKeyWithError(leaseID) +} + +func falUnboundCleanupClaim(claim core.LeaseClaim) bool { + recovery := strings.TrimSpace(claim.Labels["recovery"]) + return strings.TrimSpace(claim.Labels[falCreateRequestLabel]) != "" && + strings.TrimSpace(claim.Labels[falCreateAttemptLabel]) != "" && + (recovery == "ambiguous-create-inflight" || recovery == "ambiguous-create") +} + +func (b *backend) waitForFalInstanceAbsence(ctx context.Context, client computeAPI, instanceID string) error { + interval := b.pollInterval + if interval <= 0 { + interval = falPollInterval + } + for { + live, err := client.GetInstance(ctx, instanceID) + if isFalNotFound(err) { + ids, inventoryErr := falInventoryIDs(ctx, client) + if inventoryErr != nil { + return inventoryErr + } + found := false + for _, id := range ids { + if id == instanceID { + found = true + break + } + } + if !found { + second, secondErr := client.GetInstance(ctx, instanceID) + if isFalNotFound(secondErr) { + return nil + } + if secondErr != nil { + return secondErr + } + if strings.TrimSpace(second.ID) != instanceID { + return exit(2, "fal instance %s deletion confirmation returned changed identity", instanceID) + } + } + } else { + if err != nil { + return err + } + if strings.TrimSpace(live.ID) != instanceID { + return exit(2, "fal instance %s deletion check returned changed identity", instanceID) + } + } + timer := time.NewTimer(interval) + select { + case <-ctx.Done(): + timer.Stop() + return ctx.Err() + case <-timer.C: + } + } +} + +func (b *backend) api() (computeAPI, error) { + if b.clientFactory == nil { + b.clientFactory = newClient + } + return b.clientFactory(b.configForRun(), b.rt) +} + +func (b *backend) configForRun() Config { + cfg := b.cfg + applyFalDefaults(&cfg) + return cfg +} + +func (b *backend) waitForSSH(ctx context.Context, target *core.SSHTarget, phase string, timeout time.Duration) error { + if b.waitSSH != nil { + return b.waitSSH(ctx, target, phase, timeout) + } + return core.WaitForSSHReady(ctx, target, b.rt.Stderr, phase, timeout) +} + +func (b *backend) ensureFalCreateKey(cfg Config, leaseID string) (string, string, error) { + if b.ensureCreateKey != nil { + return b.ensureCreateKey(cfg, leaseID) + } + return core.EnsureTestboxKeyForConfigDurable(cfg, leaseID) +} + +func (b *backend) waitForInstanceReady(ctx context.Context, client computeAPI, id string) (ComputeInstance, error) { + id = strings.TrimSpace(id) + if id == "" { + return ComputeInstance{}, exit(2, "fal readiness wait requires an instance id") + } + readyCtx, cancel := context.WithTimeout(ctx, b.effectivePollTimeout()) + defer cancel() + for { + item, err := client.GetInstance(readyCtx, id) + if err != nil { + if readyCtx.Err() != nil { + return ComputeInstance{}, b.falReadinessContextError(ctx, readyCtx, id) + } + return ComputeInstance{}, err + } + if strings.TrimSpace(item.ID) != id { + return ComputeInstance{}, exit(2, "fal readiness check for instance %s returned changed identity %s", id, strings.TrimSpace(item.ID)) + } + if !item.Status.Known() { + return ComputeInstance{}, exit(5, "fal instance %s reported unknown status %s", id, item.Status) + } + if item.Status == InstanceStatusReady { + if strings.TrimSpace(item.IP) == "" { + return ComputeInstance{}, exit(5, "fal instance %s is ready without an SSH host", id) + } + return item, nil + } + if item.Status == InstanceStatusStopped { + return ComputeInstance{}, exit(5, "fal instance %s reached terminal status %s", id, item.Status) + } + timer := time.NewTimer(b.effectivePollInterval()) + select { + case <-readyCtx.Done(): + timer.Stop() + return ComputeInstance{}, b.falReadinessContextError(ctx, readyCtx, id) + case <-timer.C: + } + } +} + +func (b *backend) effectivePollTimeout() time.Duration { + if b.pollTimeout > 0 { + return b.pollTimeout + } + return falPollTimeout +} + +func (b *backend) falReadinessContextError(parent, ready context.Context, id string) error { + if parent.Err() != nil { + return context.Cause(parent) + } + return errors.Join(exit(5, "timed out waiting for fal instance %s to become ready", id), ready.Err()) +} + +func (b *backend) effectivePollInterval() time.Duration { + if b.pollInterval > 0 { + return b.pollInterval + } + return falPollInterval +} + +func (b *backend) recoverAmbiguousCreateForRelease(ctx context.Context, client computeAPI, claim core.LeaseClaim, cfg Config) (core.LeaseClaim, error) { + if claim.Labels["recovery"] != "ambiguous-create" && claim.Labels["recovery"] != "ambiguous-create-inflight" { + return core.LeaseClaim{}, exit(4, "fal recovery is still pending for lease=%s; local recovery state retained", claim.LeaseID) + } + if err := b.validateFalCreateRecoveryWindow(claim); err != nil { + return core.LeaseClaim{}, err + } + request, cfg, err := falCreateRecoveryRequest(claim, cfg) + if err != nil { + return core.LeaseClaim{}, err + } + created, updated, ambiguous, replayErr := b.replayFalCreateWithClaim(ctx, client, request, claim, cfg, "rollback-cleanup", false) + instanceID := strings.TrimSpace(created.ID) + if replayErr == nil { + if instanceID == "" || updated.CloudID != instanceID { + return core.LeaseClaim{}, exit(5, "fal recovered instance claim is unavailable for lease=%s", claim.LeaseID) + } + return updated, nil + } + if instanceID == "" { + if ambiguous { + return core.LeaseClaim{}, exit(5, "fal create recovery retry failed for lease=%s; local recovery claim retained: %v", claim.LeaseID, replayErr) + } + return core.LeaseClaim{}, replayErr + } + persistErr := fmt.Errorf("persist recovered fal instance %s claim: %w", instanceID, replayErr) + base := updated + if base.LeaseID == "" { + base = claim + } + owned, exists, ownerErr := b.adoptOrBindKnownFalInstance(base, cfg, instanceID, "rollback-cleanup", false) + if ownerErr == nil && exists { + return owned, nil + } + if ownerErr == nil && !exists { + return core.LeaseClaim{}, b.rollbackAcquireAfterClaimRemoval(instanceID, claim.LeaseID, claim.Slug, cfg, claim.RepoRoot, "rollback-cleanup", persistErr) + } + current, currentExists, readErr := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if readErr == nil && !currentExists { + return core.LeaseClaim{}, b.rollbackAcquireAfterClaimRemoval(instanceID, claim.LeaseID, claim.Slug, cfg, claim.RepoRoot, "rollback-cleanup", persistErr) + } + if readErr == nil && currentExists && current.Provider == providerName && current.ProviderScope == falClaimScope(cfg) && verifyFalClaimCredential(current, cfg) == nil { + cleanupCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + cleanupErr := b.deleteClaimedFalInstance(cleanupCtx, client, current, cfg, instanceID) + if cleanupErr == nil { + return core.LeaseClaim{}, persistErr + } + return core.LeaseClaim{}, errors.Join(persistErr, ownerErr, cleanupErr) + } + return core.LeaseClaim{}, errors.Join(persistErr, ownerErr, readErr, fmt.Errorf("fal instance %s retained until durable recovery ownership can be persisted", instanceID)) +} + +func falCreateRecoveryRequest(claim core.LeaseClaim, cfg Config) (CreateInstanceRequest, Config, error) { + keyPath, err := core.TestboxKeyPath(claim.LeaseID) + if err != nil { + return CreateInstanceRequest{}, cfg, err + } + if _, err := os.Stat(keyPath); err != nil { + return CreateInstanceRequest{}, cfg, exit(5, "fal create recovery key is unavailable for lease=%s; local recovery claim retained", claim.LeaseID) + } + publicKeyData, err := os.ReadFile(keyPath + ".pub") + if err != nil { + return CreateInstanceRequest{}, cfg, exit(5, "fal create recovery public key is unavailable for lease=%s; local recovery claim retained", claim.LeaseID) + } + publicKey := strings.TrimSpace(string(publicKeyData)) + if publicKey == "" { + return CreateInstanceRequest{}, cfg, exit(5, "fal create recovery public key is empty for lease=%s; local recovery claim retained", claim.LeaseID) + } + instanceType := firstNonBlank(claim.Labels["server_type"], cfg.Fal.InstanceType) + sector := claim.Labels["sector"] + if InstanceType(instanceType) != InstanceTypeH100x8 { + sector = "" + } + cfg.Fal.InstanceType = instanceType + cfg.ServerType = instanceType + cfg.Fal.Sector = sector + request := CreateInstanceRequest{ + InstanceType: InstanceType(instanceType), + SSHKey: publicKey, + Sector: Sector(sector), + } + if err := verifyFalCreateRequestBinding(claim, request); err != nil { + return CreateInstanceRequest{}, cfg, err + } + return request, cfg, nil +} + +func (b *backend) persistInitialFalCreateIntent(leaseID, slug string, cfg Config, repoRoot string, keep bool, createStarted time.Time, req CreateInstanceRequest) (core.LeaseClaim, error) { + if b.persistCreateIntent != nil { + return b.persistCreateIntent(leaseID, slug, cfg, repoRoot, keep, createStarted, req) + } + return b.persistRecoveryClaimAtIfUnchanged( + leaseID, + slug, + cfg, + repoRoot, + "", + "create-intent", + keep, + createStarted, + core.LeaseClaim{}, + false, + req, + ) +} + +func preserveFalAcquireLifetimeLabel(labels map[string]string, current core.LeaseClaim) { + if value, ok := current.Labels[falAcquireLifetimeLabel]; ok { + labels[falAcquireLifetimeLabel] = value + } else { + delete(labels, falAcquireLifetimeLabel) + } +} + +func (b *backend) cleanupRejectedFalCreateIntent(leaseID string, intent core.LeaseClaim) error { + if intent.LeaseID == "" { + return core.FinalizeAbsentLeaseClaimAfterSync(leaseID, func() error { + return b.removeFalLeaseKey(leaseID) + }) + } + _, exists, err := core.ReadLeaseClaimWithPresence(intent.LeaseID) + if err != nil { + return err + } + if !exists { + return core.FinalizeAbsentLeaseClaimAfterSync(leaseID, func() error { + return b.removeFalLeaseKey(leaseID) + }) + } + if err := core.RemoveLeaseClaimIfUnchangedAfter(intent.LeaseID, intent, func() error { + return b.removeFalLeaseKey(intent.LeaseID) + }); err != nil { + return fmt.Errorf("clean up rejected fal create intent: %w", err) + } + return nil +} + +func falRecoveryClaimReplacement(current core.LeaseClaim, cfg Config, instanceID, reason string, keep bool) (core.LeaseClaim, error) { + if falDeletionInProgress(current) { + return core.LeaseClaim{}, exit(4, "fal lease %s deletion is in progress; refusing recovery transition", current.LeaseID) + } + createStarted := falClaimStartedAt(current, time.Time{}) + if createStarted.IsZero() { + return core.LeaseClaim{}, exit(2, "fal lease %s recovery claim has no create timestamp", current.LeaseID) + } + labels := falLabels(cfg, current.LeaseID, current.Slug, keep, createStarted) + preserveFalAcquireLifetimeLabel(labels, current) + if current.Labels["recovery"] == "create-intent" { + labels[falAcquireLifetimeLabel] = falAcquireLifetimeVersion + } + binding := falCredentialBinding(cfg) + if binding == "" { + return core.LeaseClaim{}, exit(2, "provider=%s requires fal credentials to persist recovery ownership", providerName) + } + labels[falCredentialBindingLabel] = binding + if requestBinding := strings.TrimSpace(current.Labels[falCreateRequestLabel]); requestBinding != "" { + labels[falCreateRequestLabel] = requestBinding + } + labels["create_started_at"] = strconv.FormatInt(createStarted.UTC().Unix(), 10) + labels["state"] = reason + labels["recovery"] = reason + replacement := current + replacement.Provider = providerName + replacement.ProviderScope = falClaimScope(cfg) + replacement.CloudID = strings.TrimSpace(instanceID) + replacement.Labels = labels + return replacement, nil +} + +func falCreateRequestBinding(req CreateInstanceRequest) string { + value := string(req.InstanceType) + "\x00" + string(req.Sector) + "\x00" + req.SSHKey + sum := sha256.Sum256([]byte(value)) + return fmt.Sprintf("sha256:%x", sum[:]) +} + +func verifyFalCreateRequestBinding(claim core.LeaseClaim, req CreateInstanceRequest) error { + want := strings.TrimSpace(claim.Labels[falCreateRequestLabel]) + if want == "" { + return exit(5, "fal create request binding is unavailable for lease=%s; local recovery claim retained", claim.LeaseID) + } + if got := falCreateRequestBinding(req); got != want { + return exit(5, "fal create request changed for lease=%s; local recovery claim retained", claim.LeaseID) + } + return nil +} + +func (b *backend) newFalRecoveryClaim(current core.LeaseClaim, cfg Config, instanceID, reason string, keep bool) (core.LeaseClaim, error) { + if b.recoveryClaimReplacement != nil { + return b.recoveryClaimReplacement(current, cfg, instanceID, reason, keep) + } + return falRecoveryClaimReplacement(current, cfg, instanceID, reason, keep) +} + +func (b *backend) adoptOrBindKnownFalInstance(intent core.LeaseClaim, cfg Config, instanceID, reason string, keep bool) (core.LeaseClaim, bool, error) { + var lastErr error + for attempt := 0; attempt < 3; attempt++ { + current, exists, err := core.ReadLeaseClaimWithPresence(intent.LeaseID) + if err != nil { + return core.LeaseClaim{}, false, err + } + if !exists { + return core.LeaseClaim{}, false, nil + } + if current.Provider != providerName || current.ProviderScope != falClaimScope(cfg) { + return core.LeaseClaim{}, true, exit(2, "fal lease %s recovery ownership changed", intent.LeaseID) + } + if err := verifyFalClaimCredential(current, cfg); err != nil { + return core.LeaseClaim{}, true, err + } + if current.CloudID != "" { + if current.CloudID != instanceID { + return core.LeaseClaim{}, true, exit(2, "fal lease %s recovery instance changed from %s to %s", intent.LeaseID, instanceID, current.CloudID) + } + } + replacement, replaceErr := b.newFalRecoveryClaim(current, cfg, instanceID, reason, keep) + if replaceErr == nil { + replaceErr = core.ReplaceLeaseClaimIfUnchangedDurable(intent.LeaseID, current, replacement) + } + if replaceErr == nil { + return replacement, true, nil + } + lastErr = replaceErr + } + return core.LeaseClaim{}, true, fmt.Errorf("bind fal instance %s to recovery claim %s: %w", instanceID, intent.LeaseID, lastErr) +} + +func (b *backend) cleanupKnownFalCreateAfterBindFailure(ctx context.Context, client computeAPI, intent core.LeaseClaim, cfg Config, instanceID, reason string, keep bool, cause error) error { + current, exists, readErr := core.ReadLeaseClaimWithPresence(intent.LeaseID) + if readErr != nil { + return errors.Join(cause, readErr) + } + if !exists { + return b.rollbackAcquireAfterClaimRemoval(instanceID, intent.LeaseID, intent.Slug, cfg, intent.RepoRoot, "rollback-cleanup", cause) + } + if current.Provider != providerName || current.ProviderScope != falClaimScope(cfg) || + (current.CloudID != "" && current.CloudID != instanceID) { + return errors.Join(cause, exit(2, "fal lease %s recovery identity changed; refusing cleanup of instance %s", intent.LeaseID, instanceID)) + } + if err := verifyFalClaimCredential(current, cfg); err != nil { + return errors.Join(cause, err) + } + owned, ownedExists, ownerErr := b.adoptOrBindKnownFalInstance(current, cfg, instanceID, reason, keep) + if ownerErr != nil { + cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second) + defer cancel() + cleanupErr := b.deleteClaimedFalInstance(cleanupCtx, client, current, cfg, instanceID) + if cleanupErr == nil { + return errors.Join(cause, ownerErr) + } + return errors.Join(cause, ownerErr, cleanupErr, fmt.Errorf("fal instance %s retained because durable cleanup ownership could not be persisted", instanceID)) + } + if !ownedExists { + return b.rollbackAcquireAfterClaimRemoval(instanceID, intent.LeaseID, intent.Slug, cfg, intent.RepoRoot, "rollback-cleanup", cause) + } + cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second) + defer cancel() + cleanupErr := b.deleteClaimedFalInstance(cleanupCtx, client, owned, cfg, instanceID) + if cleanupErr == nil { + return cause + } + return errors.Join(cause, fmt.Errorf("fal cleanup failed for instance %s: %w", instanceID, cleanupErr), fmt.Errorf("fal instance retained by recovery claim %s", owned.LeaseID)) +} + +func (b *backend) now() time.Time { + if b.rt.Clock != nil { + return b.rt.Clock.Now().UTC() + } + return time.Now().UTC() +} + +func (b *backend) createReplayContext(ctx context.Context, createStarted time.Time) (context.Context, context.CancelFunc, error) { + wallNow := time.Now() + remaining := createStarted.Add(falCreateRecoveryWindow).Sub(b.now()) + if remaining <= 0 { + return nil, nil, context.DeadlineExceeded + } + if remaining > falCreateRequestTimeout { + remaining = falCreateRequestTimeout + } + replayCtx, cancel := context.WithDeadline(ctx, wallNow.Add(remaining)) + return replayCtx, cancel, nil +} + +func (b *backend) rollbackAcquire(instanceID, leaseID, slug string, cfg Config, repoRoot, reason string, cause error) error { + return b.rollbackAcquireWithClaimState(instanceID, leaseID, slug, cfg, repoRoot, reason, cause, false) +} + +func (b *backend) rollbackClaimedAcquire(instanceID, leaseID, slug string, cfg Config, repoRoot, reason string, cause error) error { + return b.rollbackAcquireWithClaimState(instanceID, leaseID, slug, cfg, repoRoot, reason, cause, true) +} + +func (b *backend) persistRollbackRecoveryClaim(leaseID, slug string, cfg Config, repoRoot, instanceID, reason string, keep bool) (core.LeaseClaim, error) { + if b.persistRollbackClaim != nil { + return b.persistRollbackClaim(leaseID, slug, cfg, repoRoot, instanceID, reason, keep) + } + return b.persistRecoveryClaimAtIfUnchanged( + leaseID, + slug, + cfg, + repoRoot, + instanceID, + reason, + keep, + time.Time{}, + core.LeaseClaim{}, + false, + ) +} + +func (b *backend) rollbackAcquireWithClaimState(instanceID, leaseID, slug string, cfg Config, repoRoot, reason string, cause error, expectedClaim bool) error { + claim, claimErr := b.transitionRecoveryClaim(leaseID, slug, cfg, repoRoot, instanceID, reason, false, expectedClaim) + if errors.Is(claimErr, errFalRecoveryClaimRemoved) { + return b.rollbackAcquireAfterClaimRemoval(instanceID, leaseID, slug, cfg, repoRoot, reason, cause) + } + if claim.LeaseID == "" { + return b.rollbackAcquireAfterClaimRemoval(instanceID, leaseID, slug, cfg, repoRoot, reason, errors.Join(cause, claimErr)) + } + if claim.CloudID != instanceID { + owned, exists, ownerErr := b.adoptOrBindKnownFalInstance(claim, cfg, instanceID, reason, false) + if ownerErr != nil || !exists { + return rollbackAcquireError(cause, instanceID, errors.Join(claimErr, ownerErr), fmt.Errorf("fal instance retained because durable cleanup ownership is unavailable")) + } + claim = owned + } + client, err := b.api() + if err != nil { + return rollbackAcquireError(cause, instanceID, claimErr, err) + } + cleanupCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + cleanupErr := b.deleteClaimedFalInstance(cleanupCtx, client, claim, cfg, instanceID) + if cleanupErr != nil { + return rollbackAcquireError(cause, instanceID, claimErr, cleanupErr) + } + if claimErr != nil { + return errors.Join(cause, fmt.Errorf("persist fal recovery claim: %w", claimErr)) + } + return cause +} + +func (b *backend) concurrentFalDeletionCompletedLocked(instanceID, leaseID string) (bool, error) { + _, exists, err := core.ReadLeaseClaimWithPresence(leaseID) + if err != nil || exists { + return false, err + } + keyAbsent, err := falLeaseKeyAbsent(leaseID) + if err != nil || !keyAbsent { + return false, err + } + client, err := b.api() + if err != nil { + return false, err + } + checkCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + live, getErr := client.GetInstance(checkCtx, instanceID) + if getErr == nil { + if strings.TrimSpace(live.ID) != instanceID { + return false, exit(2, "fal deletion check returned changed instance identity for %s", instanceID) + } + return false, nil + } + if !isFalNotFound(getErr) { + return false, getErr + } + ids, inventoryErr := falInventoryIDs(checkCtx, client) + if inventoryErr != nil { + return false, inventoryErr + } + for _, id := range ids { + if id == instanceID { + return false, nil + } + } + return true, nil +} + +func falLeaseKeyAbsent(leaseID string) (bool, error) { + keyPath, err := core.TestboxKeyPath(leaseID) + if err != nil { + return false, err + } + if _, err := os.Stat(keyPath); err == nil { + return false, nil + } else if errors.Is(err, os.ErrNotExist) { + return true, nil + } else { + return false, err + } +} + +func (b *backend) rollbackAcquireAfterClaimRemoval(instanceID, leaseID, slug string, cfg Config, repoRoot, reason string, cause error) error { + unlock, lockErr := lockFalLeaseOperation(context.Background(), leaseID) + if lockErr != nil { + return rollbackAcquireError(cause, instanceID, nil, lockErr) + } + defer unlock() + completed, completionErr := b.concurrentFalDeletionCompletedLocked(instanceID, leaseID) + if completed { + return errors.Join(cause, fmt.Errorf("%w for lease %s", errFalClaimMutationSuperseded, leaseID)) + } + if completionErr != nil { + cause = errors.Join(cause, completionErr) + } + unlockSlug, slugLockErr := lockFalSlugAllocation(context.Background()) + if slugLockErr != nil { + return rollbackAcquireError(cause, instanceID, nil, slugLockErr) + } + defer unlockSlug() + reservedSlug, slugErr := core.AllocateClaimLeaseSlug(leaseID, slug) + if slugErr != nil { + return rollbackAcquireError(cause, instanceID, nil, slugErr) + } + slug = reservedSlug + claim, claimErr := b.persistRollbackRecoveryClaim(leaseID, slug, cfg, repoRoot, instanceID, reason, false) + if claimErr != nil { + current, exists, readErr := core.ReadLeaseClaimWithPresence(leaseID) + if readErr != nil { + unlock() + return rollbackAcquireError(cause, instanceID, claimErr, readErr) + } + if !exists { + unlock() + return rollbackAcquireError(cause, instanceID, claimErr, fmt.Errorf("fal instance retained because durable cleanup ownership is unavailable; reconcile instance %s manually", instanceID)) + } + if claim.LeaseID == "" || core.VerifyLeaseClaimUnchanged(leaseID, claim) != nil { + unlock() + return rollbackAcquireError(cause, instanceID, claimErr, fmt.Errorf("fal instance retained by a concurrent recovery claim; refusing emergency cleanup")) + } + claim = current + if _, syncErr := replaceFalClaimDurably(current, current); syncErr != nil { + unlock() + return rollbackAcquireError(cause, instanceID, errors.Join(claimErr, syncErr), fmt.Errorf("fal instance retained because recovery ownership durability could not be confirmed")) + } + } + unlockSlug() + unlock() + client, err := b.api() + if err != nil { + return rollbackAcquireError(cause, instanceID, claimErr, err) + } + cleanupCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + cleanupErr := b.deleteClaimedFalInstance(cleanupCtx, client, claim, cfg, instanceID) + if cleanupErr == nil { + if claimErr != nil { + return errors.Join(cause, fmt.Errorf("persist fal recovery claim: %w", claimErr)) + } + return cause + } + return rollbackAcquireError(cause, instanceID, claimErr, cleanupErr) +} + +func rollbackAcquireError(cause error, instanceID string, claimErr error, cleanupErr error) error { + errs := []error{cause} + if claimErr != nil { + errs = append(errs, fmt.Errorf("persist fal recovery claim: %w", claimErr)) + } + errs = append(errs, fmt.Errorf("fal cleanup failed for instance %s: %w", instanceID, cleanupErr)) + return errors.Join(errs...) +} + +func (b *backend) handleFailedAcquire(instanceID, leaseID, slug string, cfg Config, repoRoot string, keep bool, cause error) error { + if keep { + if _, claimErr := b.transitionRecoveryClaim(leaseID, slug, cfg, repoRoot, instanceID, "keep-failed-acquire", true, true); claimErr != nil { + return b.rollbackClaimedAcquire(instanceID, leaseID, slug, cfg, repoRoot, "rollback-cleanup", errors.Join( + cause, + fmt.Errorf("persist fal keep recovery claim: %w", claimErr), + fmt.Errorf("deleting fal instance %s because --keep recovery state could not be persisted", instanceID), + )) + } + return cause + } + return b.rollbackClaimedAcquire(instanceID, leaseID, slug, cfg, repoRoot, "rollback-cleanup", cause) +} + +func (b *backend) transitionRecoveryClaim(leaseID, slug string, cfg Config, repoRoot, instanceID, reason string, keep, expectedClaim bool) (core.LeaseClaim, error) { + current, exists, err := core.ReadLeaseClaimWithPresence(leaseID) + if err != nil { + return core.LeaseClaim{}, err + } + if !exists { + if expectedClaim { + return core.LeaseClaim{}, errFalRecoveryClaimRemoved + } + return b.persistRecoveryClaimAtIfUnchanged(leaseID, slug, cfg, repoRoot, instanceID, reason, keep, time.Time{}, core.LeaseClaim{}, false) + } + if current.Provider != providerName || current.ProviderScope != falClaimScope(cfg) || (current.CloudID != "" && current.CloudID != instanceID) { + return core.LeaseClaim{}, exit(2, "fal lease %s recovery identity changed; refusing claim transition", leaseID) + } + if err := verifyFalClaimCredential(current, cfg); err != nil { + return core.LeaseClaim{}, err + } + updated, err := b.persistRecoveryClaimAtIfUnchanged(leaseID, slug, cfg, repoRoot, instanceID, reason, keep, time.Time{}, current, true) + if err != nil { + return current, err + } + return updated, nil +} + +func (b *backend) persistRecoveryClaim(leaseID, slug string, cfg Config, repoRoot, instanceID, reason string, keep bool) error { + return b.persistRecoveryClaimAt(leaseID, slug, cfg, repoRoot, instanceID, reason, keep, time.Time{}) +} + +func (b *backend) persistRecoveryClaimAt(leaseID, slug string, cfg Config, repoRoot, instanceID, reason string, keep bool, createStarted time.Time) error { + _, err := b.persistRecoveryClaimAtIfUnchanged(leaseID, slug, cfg, repoRoot, instanceID, reason, keep, createStarted, core.LeaseClaim{}, false) + return err +} + +func (b *backend) persistRecoveryClaimAtIfUnchanged(leaseID, slug string, cfg Config, repoRoot, instanceID, reason string, keep bool, createStarted time.Time, expected core.LeaseClaim, expectedExists bool, createRequest ...CreateInstanceRequest) (core.LeaseClaim, error) { + if len(createRequest) > 1 { + return core.LeaseClaim{}, exit(2, "fal recovery claim accepts at most one create request binding") + } + if expectedExists && falDeletionInProgress(expected) { + return core.LeaseClaim{}, exit(4, "fal lease %s deletion is in progress; refusing recovery transition", leaseID) + } + createStarted = falClaimStartedAt(expected, createStarted) + if createStarted.IsZero() { + createStarted = b.now() + } + labels := falLabels(cfg, leaseID, slug, keep, createStarted) + if !expectedExists && falRecoveryNeedsAcquireLifetime(reason) { + labels[falAcquireLifetimeLabel] = falAcquireLifetimeVersion + } + if expectedExists { + preserveFalAcquireLifetimeLabel(labels, expected) + if expected.Labels["recovery"] == "create-intent" { + labels[falAcquireLifetimeLabel] = falAcquireLifetimeVersion + } + } + binding := falCredentialBinding(cfg) + if binding == "" { + return core.LeaseClaim{}, exit(2, "provider=%s requires fal credentials to persist recovery ownership", providerName) + } + labels[falCredentialBindingLabel] = binding + if len(createRequest) == 1 { + labels[falCreateRequestLabel] = falCreateRequestBinding(createRequest[0]) + } else if requestBinding := strings.TrimSpace(expected.Labels[falCreateRequestLabel]); requestBinding != "" { + labels[falCreateRequestLabel] = requestBinding + } + labels["create_started_at"] = strconv.FormatInt(createStarted.UTC().Unix(), 10) + labels["state"] = reason + labels["recovery"] = reason + server := core.Server{ + CloudID: strings.TrimSpace(instanceID), + Provider: providerName, + Name: firstNonBlank(slug, leaseID), + Status: reason, + Labels: labels, + } + server.ServerType.Name = cfg.Fal.InstanceType + target := core.SSHTargetFromConfig(cfg, "") + if repoRoot != "" { + return core.ClaimLeaseTargetForRepoConfigScopeIfUnchangedDurable(leaseID, slug, cfg, falClaimScope(cfg), server, target, repoRoot, cfg.IdleTimeout, false, expected, expectedExists) + } + return core.ClaimLeaseTargetForConfigScopeIfUnchangedDurable(leaseID, slug, cfg, falClaimScope(cfg), server, target, cfg.IdleTimeout, expected, expectedExists) +} + +func falClaimStartedAt(claim core.LeaseClaim, fallback time.Time) time.Time { + if raw := strings.TrimSpace(claim.Labels["create_started_at"]); raw != "" { + if unixSeconds, err := strconv.ParseInt(raw, 10, 64); err == nil && unixSeconds > 0 { + return time.Unix(unixSeconds, 0).UTC() + } + } + if raw := strings.TrimSpace(claim.Labels["created_at"]); raw != "" { + if unixSeconds, err := strconv.ParseInt(raw, 10, 64); err == nil && unixSeconds > 0 { + return time.Unix(unixSeconds, 0).UTC() + } + if parsed, err := time.Parse(time.RFC3339, raw); err == nil { + return parsed.UTC() + } + } + return fallback.UTC() +} + +func resolveFalClaim(identifier, providerScope string) (core.LeaseClaim, bool, error) { + identifier = strings.TrimSpace(identifier) + if identifier == "" { + return core.LeaseClaim{}, false, exit(2, "provider=%s requires --id ", providerName) + } + claim, ok, exact, err := core.ResolveLeaseClaimForProviderScopeWithExact(identifier, providerName, providerScope) + if err != nil || ok || exact { + return claim, ok, err + } + return core.ResolveLeaseClaimForProviderCloudIDScope(identifier, providerName, providerScope) +} + +func falClaims(providerScope string) ([]core.LeaseClaim, error) { + claims, err := core.ListLeaseClaims() + if err != nil { + return nil, err + } + out := make([]core.LeaseClaim, 0, len(claims)) + for _, claim := range claims { + if claim.Provider == providerName && claim.ProviderScope == providerScope { + out = append(out, claim) + } + } + return out, nil +} + +func falClaimScope(cfg Config) string { + return core.ProviderClaimScope(providerName, cfg) +} + +func falCredentialBinding(cfg Config) string { + key := strings.TrimSpace(cfg.Fal.APIKey) + if key == "" { + return "" + } + sum := sha256.Sum256([]byte("crabbox/fal/credential-binding/v1\x00" + key)) + return fmt.Sprintf("%x", sum) +} + +func verifyFalClaimCredential(claim core.LeaseClaim, cfg Config) error { + expected := strings.TrimSpace(claim.Labels[falCredentialBindingLabel]) + actual := falCredentialBinding(cfg) + if expected == "" { + return errors.Join(errFalCredentialBindingMismatch, exit(2, "fal lease %s has no credential binding; refusing provider access", claim.LeaseID)) + } + if actual == "" { + return errors.Join(errFalCredentialBindingMismatch, exit(2, "fal lease %s requires the credential that created it; refusing provider access", claim.LeaseID)) + } + if expected != actual { + return errors.Join(errFalCredentialBindingMismatch, exit(2, "fal lease %s belongs to a different credential identity; refusing provider access", claim.LeaseID)) + } + return nil +} + +func (b *backend) ensureFalClaimCredential(ctx context.Context, client computeAPI, claim core.LeaseClaim, cfg Config, persist bool) (core.LeaseClaim, error) { + if err := verifyFalClaimCredential(claim, cfg); err == nil { + return claim, nil + } + actual := falCredentialBinding(cfg) + instanceID := strings.TrimSpace(claim.CloudID) + if actual == "" || instanceID == "" { + return core.LeaseClaim{}, verifyFalClaimCredential(claim, cfg) + } + if falDeleteStateMatches(claim, falDeleteAttemptLabel, instanceID) && !falDeleteConfirmed(claim, instanceID) { + return core.LeaseClaim{}, errors.Join(errFalCredentialBindingMismatch, exit(2, "fal lease %s has an attempted but unconfirmed deletion; credential rotation is unsafe until the original credential confirms deletion", claim.LeaseID)) + } + if claim.Provider != providerName || claim.ProviderScope != falClaimScope(cfg) { + return core.LeaseClaim{}, errors.Join(errFalCredentialBindingMismatch, exit(2, "fal lease %s belongs to a different provider route; refusing credential rebinding", claim.LeaseID)) + } + live, err := client.GetInstance(ctx, instanceID) + if isFalNotFound(err) { + return core.LeaseClaim{}, errors.Join(errFalCredentialBindingMismatch, exit(2, "fal credential rotation cannot prove ownership of instance %s; refusing rebinding", instanceID)) + } + if err != nil { + return core.LeaseClaim{}, fmt.Errorf("verify fal credential rotation against instance %s: %w", instanceID, err) + } + if strings.TrimSpace(live.ID) != instanceID { + return core.LeaseClaim{}, errors.Join(errFalCredentialBindingMismatch, exit(2, "fal credential rotation returned changed instance identity for %s", instanceID)) + } + ids, err := falInventoryIDs(ctx, client) + if err != nil { + return core.LeaseClaim{}, fmt.Errorf("verify fal credential rotation inventory: %w", err) + } + found := false + for _, id := range ids { + if id == instanceID { + found = true + break + } + } + if !found { + return core.LeaseClaim{}, errors.Join(errFalCredentialBindingMismatch, exit(2, "fal credential rotation inventory omitted instance %s; refusing rebinding", instanceID)) + } + updated := claim + updated.Labels = cloneLabels(claim.Labels) + updated.Labels[falCredentialBindingLabel] = actual + if !persist { + return updated, nil + } + updated, err = replaceFalClaimDurably(claim, updated) + if err != nil { + return core.LeaseClaim{}, fmt.Errorf("persist fal credential rotation for lease %s: %w", claim.LeaseID, err) + } + return updated, nil +} + +func falClaimServer(server core.Server, cfg Config) (core.Server, error) { + binding := falCredentialBinding(cfg) + if binding == "" { + return core.Server{}, exit(2, "provider=%s requires fal credentials to persist lease ownership", providerName) + } + server.Labels = cloneLabels(server.Labels) + server.Labels[falCredentialBindingLabel] = binding + return server, nil +} + +func leaseTargetFromClaimedInstance(item ComputeInstance, claim core.LeaseClaim, cfg Config, includeSSH bool) (core.LeaseTarget, error) { + if claim.Provider != providerName { + return core.LeaseTarget{}, exit(2, "lease %s is claimed by provider=%s; refusing fal resolve", claim.LeaseID, claim.Provider) + } + if claim.CloudID != "" && strings.TrimSpace(item.ID) != claim.CloudID { + return core.LeaseTarget{}, exit(2, "refusing to resolve changed fal instance %s", claim.CloudID) + } + server := falServer(item, cfg, claim.LeaseID, claim.Slug, claim.Labels["keep"] == "true", falClaimStartedAt(claim, time.Now().UTC())) + server.Labels = mergeFalClaimLabels(server.Labels, claim.Labels) + server, err := mergeClaimEndpoint(server, claim) + if err != nil { + return core.LeaseTarget{}, err + } + target := core.LeaseTarget{Server: server, LeaseID: claim.LeaseID} + if includeSSH { + ssh, err := falSSHTarget(cfg, item) + if err != nil { + return core.LeaseTarget{}, err + } + applyClaimSSHEndpoint(&ssh, claim) + core.UseStoredTestboxKey(&ssh, claim.LeaseID) + target.SSH = ssh + } + return target, nil +} + +func leaseTargetFromClaim(claim core.LeaseClaim, cfg Config, includeSSH bool) (core.LeaseTarget, error) { + server, err := serverFromClaim(claim, cfg) + if err != nil { + return core.LeaseTarget{}, err + } + target := core.LeaseTarget{Server: server, LeaseID: claim.LeaseID} + if includeSSH && claim.SSHHost != "" { + ssh := core.SSHTargetFromConfig(cfg, claim.SSHHost) + if claim.SSHPort > 0 { + ssh.Port = strconv.Itoa(claim.SSHPort) + } + core.UseStoredTestboxKey(&ssh, claim.LeaseID) + target.SSH = ssh + } + return target, nil +} + +func applyClaimSSHEndpoint(ssh *core.SSHTarget, claim core.LeaseClaim) { + if claim.SSHHost != "" { + ssh.Host = claim.SSHHost + } + if claim.SSHPort > 0 { + ssh.Port = strconv.Itoa(claim.SSHPort) + } + if user := firstNonBlank(claim.Labels["ssh_user"], claim.StaticUser); user != "" { + ssh.User = user + } +} + +func serverFromClaim(claim core.LeaseClaim, cfg Config) (core.Server, error) { + if claim.Provider != providerName { + return core.Server{}, exit(2, "lease %s is claimed by provider=%s; refusing fal cleanup", claim.LeaseID, claim.Provider) + } + labels := cloneLabels(claim.Labels) + if len(labels) == 0 { + labels = falLabels(cfg, claim.LeaseID, claim.Slug, false, time.Now().UTC()) + } + delete(labels, falCredentialBindingLabel) + delete(labels, falDeleteAttemptLabel) + delete(labels, falDeleteAcceptedLabel) + delete(labels, falDeleteConfirmedLabel) + delete(labels, falAcquireLifetimeLabel) + server := core.Server{ + CloudID: claim.CloudID, + Provider: providerName, + Name: firstNonBlank(labels["name"], claim.Slug, claim.LeaseID), + Status: firstNonBlank(labels["state"], "unknown"), + Labels: labels, + } + server.PublicNet.IPv4.IP = claim.SSHHost + server.ServerType.Name = firstNonBlank(labels["server_type"], cfg.Fal.InstanceType, cfg.ServerType) + return server, nil +} + +func mergeClaimEndpoint(server core.Server, claim core.LeaseClaim) (core.Server, error) { + if claim.CloudID != "" && server.CloudID != "" && claim.CloudID != server.CloudID { + return core.Server{}, exit(2, "refusing to list fal instance %s from stale local claim", server.CloudID) + } + if claim.SSHHost != "" { + server.PublicNet.IPv4.IP = claim.SSHHost + } + return server, nil +} + +func falServer(item ComputeInstance, cfg Config, leaseID, slug string, keep bool, createdAt time.Time) core.Server { + labels := falLabels(cfg, leaseID, slug, keep, createdAt) + labels["fal_instance_id"] = strings.TrimSpace(item.ID) + labels["server_type"] = firstNonBlank(string(item.InstanceType), cfg.Fal.InstanceType, cfg.ServerType) + labels["name"] = firstNonBlank(slug, item.ID) + if item.Region != "" { + labels["region"] = item.Region + } + if item.Sector != "" { + labels["sector"] = string(item.Sector) + } + if item.CreatorUserNickname != "" { + labels["creator_user_nickname"] = item.CreatorUserNickname + } + if item.IP != "" { + labels["ssh_host"] = item.IP + } + labels["ssh_port"] = firstNonBlank(cfg.SSHPort, "22") + labels["ssh_user"] = firstNonBlank(cfg.SSHUser, cfg.Fal.User, defaultUser) + status := normalizeFalStatus(item.Status) + labels["state"] = status + server := core.Server{ + CloudID: strings.TrimSpace(item.ID), + Provider: providerName, + Name: firstNonBlank(slug, item.ID), + Status: status, + Labels: labels, + } + server.PublicNet.IPv4.IP = strings.TrimSpace(item.IP) + server.ServerType.Name = firstNonBlank(string(item.InstanceType), cfg.Fal.InstanceType, cfg.ServerType) + return server +} + +func falLabels(cfg Config, leaseID, slug string, keep bool, now time.Time) map[string]string { + labels := core.DirectLeaseLabels(cfg, leaseID, slug, providerName, "", keep, now) + labels["work_root"] = cfg.WorkRoot + labels["server_type"] = firstNonBlank(cfg.Fal.InstanceType, cfg.ServerType) + labels["sector"] = cfg.Fal.Sector + return labels +} + +func falSSHTarget(cfg Config, item ComputeInstance) (core.SSHTarget, error) { + host := strings.TrimSpace(item.IP) + if host == "" { + return core.SSHTarget{}, exit(5, "fal instance %s has no SSH host", item.ID) + } + target := core.SSHTargetFromConfig(cfg, host) + target.TargetOS = core.TargetLinux + target.NetworkKind = core.NetworkPublic + target.ReadyCheck = "command -v git >/dev/null && command -v rsync >/dev/null && command -v tar >/dev/null" + return target, nil +} + +func normalizeFalStatus(status InstanceStatus) string { + value := strings.ToLower(strings.TrimSpace(string(status))) + if value == "" { + return string(InstanceStatusUnknown) + } + return value +} + +func isFalNotFound(err error) bool { + var apiErr *APIError + return errors.As(err, &apiErr) && apiErr.StatusCode == 404 +} + +func isAmbiguousFalMutationError(err error) bool { + var apiErr *APIError + if errors.As(err, &apiErr) { + return apiErr.StatusCode >= 500 || apiErr.StatusCode == 408 || apiErr.StatusCode == 409 + } + return err != nil +} + +func firstNonBlank(values ...string) string { + for _, value := range values { + if strings.TrimSpace(value) != "" { + return strings.TrimSpace(value) + } + } + return "" +} + +func cloneLabels(labels map[string]string) map[string]string { + out := make(map[string]string, len(labels)) + for key, value := range labels { + out[key] = value + } + return out +} + +func mergeLabels(base, overlay map[string]string) map[string]string { + out := cloneLabels(base) + for key, value := range overlay { + if strings.TrimSpace(value) != "" { + out[key] = value + } + } + return out +} + +func mergeFalClaimLabels(live, claim map[string]string) map[string]string { + out := mergeLabels(live, claim) + delete(out, falCredentialBindingLabel) + delete(out, falDeleteAttemptLabel) + delete(out, falDeleteAcceptedLabel) + delete(out, falDeleteConfirmedLabel) + delete(out, falAcquireLifetimeLabel) + for _, key := range []string{"creator_user_nickname", "fal_instance_id", "region", "sector", "server_type", "state"} { + if value := strings.TrimSpace(live[key]); value != "" { + out[key] = value + } else { + delete(out, key) + } + } + return out +} diff --git a/internal/providers/fal/backend_test.go b/internal/providers/fal/backend_test.go new file mode 100644 index 000000000..e8e386df7 --- /dev/null +++ b/internal/providers/fal/backend_test.go @@ -0,0 +1,3624 @@ +package fal + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "sync" + "testing" + "time" + + core "github.com/openclaw/crabbox/internal/cli" +) + +type fakeFalAPI struct { + instances map[string]ComputeInstance + getErr error + createErr error + createErrs []error + deleteErr error + createRequests []CreateInstanceRequest + idempotency []string + deletedIDs []string + listCalls int + createHook func(CreateInstanceRequest, string, ComputeInstance) + beforeCreateHook func(string) + afterDeleteHook func(string) + blockCreateUntilContext bool + blockGetUntilContext bool + retainDeletedInstance bool + removeBeforeDeleteError bool +} + +type mutableFalClock struct { + now time.Time +} + +func (c *mutableFalClock) Now() time.Time { return c.now } + +func (f *fakeFalAPI) ListInstances(context.Context, int, string) (ListInstancesResponse, error) { + f.listCalls++ + items := make([]ComputeInstance, 0, len(f.instances)) + for _, item := range f.instances { + items = append(items, item) + } + return ListInstancesResponse{Instances: items}, nil +} + +func (f *fakeFalAPI) GetInstance(ctx context.Context, id string) (ComputeInstance, error) { + if f.blockGetUntilContext { + <-ctx.Done() + return ComputeInstance{}, ctx.Err() + } + if f.getErr != nil { + return ComputeInstance{}, f.getErr + } + item, ok := f.instances[id] + if !ok { + return ComputeInstance{}, &APIError{StatusCode: 404, Status: "404 Not Found", Message: "not found"} + } + return item, nil +} + +func (f *fakeFalAPI) CreateInstance(ctx context.Context, req CreateInstanceRequest, idempotencyKey string) (ComputeInstance, error) { + f.createRequests = append(f.createRequests, req) + f.idempotency = append(f.idempotency, idempotencyKey) + if f.beforeCreateHook != nil { + f.beforeCreateHook(idempotencyKey) + } + if f.blockCreateUntilContext { + <-ctx.Done() + return ComputeInstance{}, ctx.Err() + } + if len(f.createErrs) > 0 { + err := f.createErrs[0] + f.createErrs = f.createErrs[1:] + if err != nil { + return ComputeInstance{}, err + } + } + if f.createErr != nil { + return ComputeInstance{}, f.createErr + } + if f.instances == nil { + f.instances = map[string]ComputeInstance{} + } + item := ComputeInstance{ + ID: "inst_created", + InstanceType: req.InstanceType, + Sector: req.Sector, + Region: "us-west", + IP: "203.0.113.42", + Status: InstanceStatusReady, + } + f.instances[item.ID] = item + if f.createHook != nil { + f.createHook(req, idempotencyKey, item) + } + return item, nil +} + +func (f *fakeFalAPI) DeleteInstance(_ context.Context, id string) error { + f.deletedIDs = append(f.deletedIDs, id) + if f.afterDeleteHook != nil { + f.afterDeleteHook(id) + } + if f.deleteErr != nil { + if f.removeBeforeDeleteError { + delete(f.instances, id) + } + return f.deleteErr + } + if !f.retainDeletedInstance { + delete(f.instances, id) + } + return nil +} + +func newFalTestBackend(t *testing.T, api *fakeFalAPI) *backend { + t.Helper() + home := t.TempDir() + t.Setenv("HOME", home) + t.Setenv("XDG_CONFIG_HOME", home) + t.Setenv("XDG_STATE_HOME", home) + cfg := core.BaseConfig() + cfg.Provider = providerName + cfg.TargetOS = core.TargetLinux + cfg.Fal.APIKey = "test-key" + cfg.Fal.APIURL = "http://127.0.0.1:8080/v1" + applyFalDefaults(&cfg) + b := &backend{ + spec: Provider{}.Spec(), + cfg: cfg, + rt: core.Runtime{Stdout: io.Discard, Stderr: io.Discard}, + pollInterval: time.Nanosecond, + pollTimeout: time.Second, + } + b.clientFactory = func(Config, Runtime) (computeAPI, error) { return api, nil } + b.waitSSH = func(context.Context, *core.SSHTarget, string, time.Duration) error { return nil } + return b +} + +func persistFalCreateRecoveryClaim(t *testing.T, b *backend, leaseID, slug, reason string, keep bool, started time.Time) (core.LeaseClaim, CreateInstanceRequest) { + t.Helper() + cfg := b.configForRun() + _, publicKey, err := core.EnsureTestboxKeyForConfig(cfg, leaseID) + if err != nil { + t.Fatal(err) + } + sector := Sector(cfg.Fal.Sector) + if InstanceType(cfg.Fal.InstanceType) != InstanceTypeH100x8 { + sector = "" + } + req := CreateInstanceRequest{InstanceType: InstanceType(cfg.Fal.InstanceType), SSHKey: publicKey, Sector: sector} + if started.IsZero() { + started = b.now() + } + claim, err := b.persistRecoveryClaimAtIfUnchanged(leaseID, slug, cfg, "", "", reason, keep, started, core.LeaseClaim{}, false, req) + if err != nil { + t.Fatal(err) + } + return claim, req +} + +func markFalAcquireLifetimeClaimForTest(claim core.LeaseClaim) (core.LeaseClaim, error) { + marked := claim + marked.Labels = cloneLabels(claim.Labels) + marked.Labels[falAcquireLifetimeLabel] = falAcquireLifetimeVersion + if err := core.ReplaceLeaseClaimIfUnchangedDurable(claim.LeaseID, claim, marked); err != nil { + return claim, err + } + return marked, nil +} + +func TestFalAcquireCreatesInstanceWaitsAndClaimsLease(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.cfg.Fal.InstanceType = string(InstanceTypeH100x8) + b.cfg.Fal.Sector = string(Sector2) + b.cfg.Fal.User = "ubuntu" + b.cfg.SSHUser = "ubuntu" + + lease, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "gpu-box"}) + if err != nil { + t.Fatal(err) + } + if lease.LeaseID == "" || lease.Server.CloudID != "inst_created" || lease.SSH.Host != "203.0.113.42" || lease.SSH.User != "ubuntu" { + t.Fatalf("lease=%#v", lease) + } + if len(api.createRequests) != 1 { + t.Fatalf("createRequests=%#v", api.createRequests) + } + req := api.createRequests[0] + if req.InstanceType != InstanceTypeH100x8 || req.Sector != Sector2 || !strings.HasPrefix(req.SSHKey, "ssh-") { + t.Fatalf("create request=%#v", req) + } + if len(api.idempotency) != 1 || api.idempotency[0] != lease.LeaseID { + t.Fatalf("idempotency=%#v lease=%s", api.idempotency, lease.LeaseID) + } + claim, ok, err := core.ResolveLeaseClaimForProvider("gpu-box", providerName) + if err != nil || !ok { + t.Fatalf("claim ok=%v err=%v", ok, err) + } + if claim.CloudID != "inst_created" || claim.SSHHost != "203.0.113.42" || claim.Labels["provider"] != providerName || claim.Labels["sector"] != string(Sector2) { + t.Fatalf("claim=%#v", claim) + } + if claim.ProviderScope != falClaimScope(b.cfg) { + t.Fatalf("claim scope=%q want %q", claim.ProviderScope, falClaimScope(b.cfg)) + } + if claim.Labels[falCredentialBindingLabel] == "" { + t.Fatalf("claim missing credential binding: %#v", claim.Labels) + } + if lease.Server.Labels[falCredentialBindingLabel] != "" { + t.Fatalf("credential binding leaked into lease labels: %#v", lease.Server.Labels) + } +} + +func TestFalAcquirePersistsReplayIntentBeforeProviderMutation(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + observed := false + api.beforeCreateHook = func(leaseID string) { + data, err := os.ReadFile(filepath.Join(os.Getenv("XDG_STATE_HOME"), "crabbox", "claims", leaseID+".json")) + if err != nil { + t.Fatalf("read pre-create claim: %v", err) + } + var claim core.LeaseClaim + if err := json.Unmarshal(data, &claim); err != nil { + t.Fatalf("decode pre-create claim: %v", err) + } + if claim.CloudID != "" || claim.Labels["recovery"] != "ambiguous-create-inflight" || claim.Labels["create_started_at"] == "" || claim.Labels[falAcquireLifetimeLabel] != falAcquireLifetimeVersion || claim.Labels[falCredentialBindingLabel] != falCredentialBinding(b.cfg) { + t.Fatalf("pre-create claim=%#v", claim) + } + observed = true + } + + lease, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "durable-intent"}) + if err != nil { + t.Fatal(err) + } + if !observed || lease.Server.CloudID != "inst_created" { + t.Fatalf("intent observed=%v lease=%#v", observed, lease) + } + snapshot, exists, set := core.ServerLeaseClaimSnapshot(lease.Server) + if !set || !exists || snapshot.LeaseID != lease.LeaseID || snapshot.CloudID != lease.Server.CloudID { + t.Fatalf("ready claim snapshot=%#v exists=%t set=%t", snapshot, exists, set) + } + if snapshot.Labels[falAcquireLifetimeLabel] != falAcquireLifetimeVersion { + t.Fatalf("ready claim lifetime protocol=%q", snapshot.Labels[falAcquireLifetimeLabel]) + } + if lease.Server.Labels[falAcquireLifetimeLabel] != "" { + t.Fatalf("internal lifetime marker leaked to server labels: %#v", lease.Server.Labels) + } +} + +func TestFalAmbiguousCreateReplayRejectsCredentialRotation(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + const leaseID = "cbx_replay_credential" + claim, _ := persistFalCreateRecoveryClaim(t, b, leaseID, "replay-credential", "ambiguous-create", false, b.now()) + keyPath, keyErr := core.TestboxKeyPath(leaseID) + if keyErr != nil { + t.Fatal(keyErr) + } + originalBinding := claim.Labels[falCredentialBindingLabel] + b.cfg.Fal.APIKey = "rotated-test-key" + + _, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, claim, b.configForRun()) + if err == nil || !strings.Contains(err.Error(), "credential identity") { + t.Fatalf("recovery err=%v", err) + } + if len(api.createRequests) != 0 || len(api.deletedIDs) != 0 { + t.Fatalf("credential mismatch mutated provider create=%d delete=%#v", len(api.createRequests), api.deletedIDs) + } + retained, exists, readErr := core.ReadLeaseClaimWithPresence(leaseID) + if readErr != nil || !exists || retained.Labels[falCredentialBindingLabel] != originalBinding { + t.Fatalf("claim=%#v exists=%t err=%v", retained, exists, readErr) + } + if _, statErr := os.Stat(keyPath); statErr != nil { + t.Fatalf("credential mismatch removed recovery key: %v", statErr) + } +} + +func TestFalCreateIntentReplayAtomicallyAddsAcquireLifetimeMarker(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + const leaseID = "cbx_replay_lifetime" + claim, request := persistFalCreateRecoveryClaim(t, b, leaseID, "replay-lifetime", "create-intent", false, b.now()) + unmarked := claim + unmarked.Labels = cloneLabels(claim.Labels) + delete(unmarked.Labels, falAcquireLifetimeLabel) + if err := core.ReplaceLeaseClaimIfUnchangedDurable(leaseID, claim, unmarked); err != nil { + t.Fatal(err) + } + api.beforeCreateHook = func(string) { + current, exists, err := core.ReadLeaseClaimWithPresence(leaseID) + if err != nil || !exists || current.Labels[falAcquireLifetimeLabel] != falAcquireLifetimeVersion { + t.Fatalf("inflight claim=%#v exists=%t err=%v", current, exists, err) + } + } + + created, updated, ambiguous, err := b.replayFalCreateWithClaim(context.Background(), api, request, unmarked, b.configForRun(), "provisioning", false) + if err != nil || ambiguous || created.ID == "" || updated.CloudID != created.ID || updated.Labels[falAcquireLifetimeLabel] != falAcquireLifetimeVersion { + t.Fatalf("created=%#v updated=%#v ambiguous=%t err=%v", created, updated, ambiguous, err) + } + if err := b.deleteClaimedFalInstance(context.Background(), api, updated, b.configForRun(), created.ID); err != nil { + t.Fatal(err) + } +} + +func TestFalAcquireAbortsBeforeMutationWhenIntentPersistenceFails(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + var leaseID string + b.persistCreateIntent = func(id, _ string, _ Config, _ string, _ bool, _ time.Time, _ CreateInstanceRequest) (core.LeaseClaim, error) { + leaseID = id + return core.LeaseClaim{}, errors.New("claim store unavailable") + } + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "intent-write-fail"}) + if err == nil || !strings.Contains(err.Error(), "persist fal create intent before provider mutation") { + t.Fatalf("acquire err=%v", err) + } + if len(api.createRequests) != 0 { + t.Fatalf("provider mutation occurred without durable intent: %#v", api.createRequests) + } + keyPath, keyErr := core.TestboxKeyPath(leaseID) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, statErr := os.Stat(keyPath); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("rejected intent key still exists: %v", statErr) + } +} + +func TestFalAcquireAbortsBeforeIntentWhenKeySyncFails(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + var leaseID string + b.ensureCreateKey = func(_ Config, id string) (string, string, error) { + leaseID = id + return "", "", errors.New("key sync unavailable") + } + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "key-sync-fail"}) + if err == nil || !strings.Contains(err.Error(), "key sync unavailable") { + t.Fatalf("acquire err=%v", err) + } + if len(api.createRequests) != 0 { + t.Fatalf("provider mutation occurred without durable key: %#v", api.createRequests) + } + if _, exists, readErr := core.ReadLeaseClaimWithPresence(leaseID); readErr != nil || exists { + t.Fatalf("claim exists=%v err=%v", exists, readErr) + } + keyPath, keyErr := core.TestboxKeyPath(leaseID) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, statErr := os.Stat(keyPath); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("unsynced key still exists: %v", statErr) + } +} + +func TestFalAcquireCleansKeyWhenFailedIntentResultWasNeverWritten(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + var leaseID string + b.persistCreateIntent = func(id, _ string, _ Config, _ string, _ bool, _ time.Time, _ CreateInstanceRequest) (core.LeaseClaim, error) { + leaseID = id + return core.LeaseClaim{LeaseID: id}, errors.New("temp-file sync unavailable") + } + if _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "unwritten-intent"}); err == nil || !strings.Contains(err.Error(), "temp-file sync unavailable") { + t.Fatalf("acquire err=%v", err) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(leaseID); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + keyPath, err := core.TestboxKeyPath(leaseID) + if err != nil { + t.Fatal(err) + } + if _, statErr := os.Stat(keyPath); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("unwritten intent key still exists: %v", statErr) + } +} + +func TestFalAcquireCleansUpPartiallyPersistedIntentBeforeReturning(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + var leaseID string + b.persistCreateIntent = func(id, slug string, cfg Config, repoRoot string, keep bool, started time.Time, req CreateInstanceRequest) (core.LeaseClaim, error) { + leaseID = id + claim, err := b.persistRecoveryClaimAtIfUnchanged(id, slug, cfg, repoRoot, "", "create-intent", keep, started, core.LeaseClaim{}, false, req) + if err != nil { + return core.LeaseClaim{}, err + } + return claim, errors.New("directory sync unavailable") + } + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "partial-intent"}) + if err == nil || !strings.Contains(err.Error(), "directory sync unavailable") { + t.Fatalf("acquire err=%v", err) + } + if len(api.createRequests) != 0 { + t.Fatalf("provider mutation occurred after rejected intent write: %#v", api.createRequests) + } + if _, exists, readErr := core.ReadLeaseClaimWithPresence(leaseID); readErr != nil || exists { + t.Fatalf("claim exists=%v err=%v", exists, readErr) + } + keyPath, keyErr := core.TestboxKeyPath(leaseID) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, statErr := os.Stat(keyPath); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("partial intent key still exists: %v", statErr) + } +} + +func TestFalAcquireCancellationBeforeProviderMutationCleansIntentAndKey(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + var leaseID string + var unlock func() + b.persistCreateIntent = func(id, slug string, cfg Config, repoRoot string, keep bool, started time.Time, req CreateInstanceRequest) (core.LeaseClaim, error) { + leaseID = id + claim, err := b.persistRecoveryClaimAtIfUnchanged(id, slug, cfg, repoRoot, "", "create-intent", keep, started, core.LeaseClaim{}, false, req) + if err != nil { + return core.LeaseClaim{}, err + } + unlock, err = lockFalLeaseOperation(context.Background(), id) + if err != nil { + return core.LeaseClaim{}, err + } + cancel() + return claim, nil + } + _, err := b.Acquire(ctx, core.AcquireRequest{RequestedSlug: "cancel-before-create"}) + if unlock != nil { + unlock() + } + if !errors.Is(err, context.Canceled) { + t.Fatalf("acquire err=%v", err) + } + if len(api.createRequests) != 0 { + t.Fatalf("provider mutation occurred after canceled lock wait: %#v", api.createRequests) + } + if _, exists, readErr := core.ReadLeaseClaimWithPresence(leaseID); readErr != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, readErr) + } + keyPath, keyErr := core.TestboxKeyPath(leaseID) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, statErr := os.Stat(keyPath); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("canceled create key still exists: %v", statErr) + } +} + +func TestFalAcquireFinalPublishDistinguishesConcurrentDeleteFromClaimLoss(t *testing.T) { + for _, providerAbsent := range []bool{false, true} { + t.Run(fmt.Sprintf("provider_absent_%t", providerAbsent), func(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.waitSSH = func(context.Context, *core.SSHTarget, string, time.Duration) error { + claims, err := core.ListLeaseClaimsWithPrefix("cbx_") + if err != nil || len(claims) != 1 { + return fmt.Errorf("claims=%d err=%w", len(claims), err) + } + if err := core.RemoveLeaseClaimIfUnchanged(claims[0].LeaseID, claims[0]); err != nil { + return err + } + if providerAbsent { + delete(api.instances, "inst_created") + return core.RemoveStoredTestboxKeyWithError(claims[0].LeaseID) + } + return nil + } + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "publish-race"}) + if err == nil { + t.Fatal("expected final publication race") + } + if providerAbsent { + if !errors.Is(err, errFalClaimMutationSuperseded) || len(api.deletedIDs) != 0 { + t.Fatalf("superseded acquire err=%v deleted=%#v", err, api.deletedIDs) + } + } else if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_created" { + t.Fatalf("live claim-loss cleanup err=%v deleted=%#v", err, api.deletedIDs) + } + claims, readErr := core.ListLeaseClaimsWithPrefix("cbx_") + if readErr != nil || len(claims) != 0 { + t.Fatalf("claims=%#v err=%v", claims, readErr) + } + }) + } +} + +func TestFalAcquireFinalPublishRetainsOwnershipWhenGetAbsenceConflictsWithInventory(t *testing.T) { + notFound := &APIError{StatusCode: 404, Status: "404 Not Found", Message: "not found"} + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.waitSSH = func(context.Context, *core.SSHTarget, string, time.Duration) error { + claims, err := core.ListLeaseClaimsWithPrefix("cbx_") + if err != nil || len(claims) != 1 { + return fmt.Errorf("claims=%d err=%w", len(claims), err) + } + if err := core.RemoveLeaseClaimIfUnchanged(claims[0].LeaseID, claims[0]); err != nil { + return err + } + api.getErr = notFound + return nil + } + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "publish-masked-absence"}) + if err == nil || errors.Is(err, errFalClaimMutationSuperseded) || !strings.Contains(err.Error(), errFalProviderAbsenceNotAccountBound.Error()) { + t.Fatalf("acquire err=%v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("provider mutation crossed conflicting absence proof: %#v", api.deletedIDs) + } + claims, readErr := core.ListLeaseClaimsWithPrefix("cbx_") + if readErr != nil || len(claims) != 1 || claims[0].CloudID != "inst_created" || claims[0].Labels["recovery"] != "rollback-cleanup" { + t.Fatalf("claims=%#v err=%v", claims, readErr) + } +} + +func TestFalAcquireFailureRejectsConcurrentStopAndRollsBack(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.waitSSH = func(context.Context, *core.SSHTarget, string, time.Duration) error { + claims, err := core.ListLeaseClaimsWithPrefix("cbx_") + if err != nil || len(claims) != 1 { + return fmt.Errorf("claims=%d err=%w", len(claims), err) + } + claim := claims[0] + stopErr := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: claim.LeaseID, + Server: core.Server{CloudID: claim.CloudID, Provider: providerName, Labels: map[string]string{"lease": claim.LeaseID}}, + }}) + if stopErr == nil || !strings.Contains(stopErr.Error(), "acquisition is still in progress") { + return fmt.Errorf("concurrent stop err=%v", stopErr) + } + if len(api.deletedIDs) != 0 { + return fmt.Errorf("concurrent stop deleted provider instance: %#v", api.deletedIDs) + } + if err := core.VerifyLeaseClaimUnchanged(claim.LeaseID, claim); err != nil { + return fmt.Errorf("concurrent stop changed in-flight claim: %w", err) + } + return errors.New("ssh readiness failed after rejected concurrent stop") + } + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "failure-stop-race"}) + if err == nil || errors.Is(err, errFalClaimMutationSuperseded) || !strings.Contains(err.Error(), "ssh readiness failed after rejected concurrent stop") { + t.Fatalf("acquire err=%v", err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_created" { + t.Fatalf("deletedIDs=%#v", api.deletedIDs) + } + claims, readErr := core.ListLeaseClaimsWithPrefix("cbx_") + if readErr != nil || len(claims) != 0 { + t.Fatalf("claims=%#v err=%v", claims, readErr) + } + if len(api.idempotency) != 1 { + t.Fatalf("idempotency=%#v", api.idempotency) + } + keyPath, keyErr := core.TestboxKeyPath(api.idempotency[0]) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, statErr := os.Stat(keyPath); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("rollback key still exists: %v", statErr) + } +} + +func TestFalAcquireRejectsCredentialRebindingResolveAndRollsBack(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.waitSSH = func(context.Context, *core.SSHTarget, string, time.Duration) error { + claims, err := core.ListLeaseClaimsWithPrefix("cbx_") + if err != nil || len(claims) != 1 { + return fmt.Errorf("claims=%d err=%w", len(claims), err) + } + claim := claims[0] + b.cfg.Fal.APIKey = "rotated-test-key" + _, resolveErr := b.Resolve(context.Background(), core.ResolveRequest{ID: claim.LeaseID}) + if resolveErr == nil || !strings.Contains(resolveErr.Error(), "acquisition is still in progress") { + return fmt.Errorf("credential-rebinding resolve err=%v", resolveErr) + } + if err := core.VerifyLeaseClaimUnchanged(claim.LeaseID, claim); err != nil { + return fmt.Errorf("credential-rebinding resolve changed in-flight claim: %w", err) + } + return errors.New("ssh readiness failed after rejected credential rebinding") + } + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "rebind-resolve-race"}) + if err == nil || errors.Is(err, errFalClaimMutationSuperseded) || !strings.Contains(err.Error(), "ssh readiness failed after rejected credential rebinding") { + t.Fatalf("acquire err=%v", err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_created" { + t.Fatalf("deletedIDs=%#v", api.deletedIDs) + } + claims, readErr := core.ListLeaseClaimsWithPrefix("cbx_") + if readErr != nil || len(claims) != 0 { + t.Fatalf("claims=%#v err=%v", claims, readErr) + } + if len(api.idempotency) != 1 { + t.Fatalf("idempotency=%#v", api.idempotency) + } + keyPath, keyErr := core.TestboxKeyPath(api.idempotency[0]) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, statErr := os.Stat(keyPath); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("rollback key still exists: %v", statErr) + } +} + +func TestFalReadySnapshotPreventsPostDeleteClaimRecreation(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + lease, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "snapshot-delete"}) + if err != nil { + t.Fatal(err) + } + snapshot, exists, set := core.ServerLeaseClaimSnapshot(lease.Server) + if !set || !exists { + t.Fatalf("snapshot=%#v exists=%t set=%t", snapshot, exists, set) + } + if err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: lease}); err != nil { + t.Fatal(err) + } + claimServer, err := falClaimServer(lease.Server, b.configForRun()) + if err != nil { + t.Fatal(err) + } + if _, err := core.ClaimLeaseTargetForRepoConfigScopeIfUnchanged( + lease.LeaseID, snapshot.Slug, b.cfg, falClaimScope(b.configForRun()), claimServer, lease.SSH, "/repo", time.Minute, false, snapshot, true, + ); err == nil { + t.Fatal("deleted ready claim was recreated from a stale acquired target") + } + if _, exists, err := core.ReadLeaseClaimWithPresence(lease.LeaseID); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } +} + +func TestFalAcquireDefaultSingleGPUOmitsSector(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.cfg.Fal.Sector = string(Sector1) + + if _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "single-gpu"}); err != nil { + t.Fatal(err) + } + if len(api.createRequests) != 1 { + t.Fatalf("createRequests=%#v", api.createRequests) + } + req := api.createRequests[0] + if req.InstanceType != InstanceTypeH100x1 || req.Sector != "" { + t.Fatalf("create request=%#v", req) + } +} + +func TestFalAcquireUsesExplicitGenericServerType(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.cfg.ServerType = " gpu_8x_h100_sxm5 " + b.cfg.ServerTypeExplicit = true + b.cfg.Fal.Sector = string(Sector2) + + if _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "generic-type"}); err != nil { + t.Fatal(err) + } + if len(api.createRequests) != 1 { + t.Fatalf("createRequests=%#v", api.createRequests) + } + if got := api.createRequests[0]; got.InstanceType != InstanceTypeH100x8 || got.Sector != Sector2 { + t.Fatalf("create request=%#v", got) + } +} + +func TestFalAcquireReturnsSSHPortUpdatedByReadinessProbe(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.cfg.SSHPort = "2222" + b.waitSSH = func(_ context.Context, target *core.SSHTarget, _ string, _ time.Duration) error { + if target.Port != "2222" { + t.Fatalf("probe received port %q, want configured 2222", target.Port) + } + target.Port = "22" + return nil + } + + lease, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "fallback-port"}) + if err != nil { + t.Fatal(err) + } + if lease.SSH.Port != "22" { + t.Fatalf("returned ssh port=%q, want readiness-updated 22", lease.SSH.Port) + } + claim, ok, claimErr := core.ResolveLeaseClaimForProvider("fallback-port", providerName) + if claimErr != nil || !ok { + t.Fatalf("claim ok=%v err=%v", ok, claimErr) + } + if claim.SSHPort != 22 { + t.Fatalf("persisted ssh port=%d, want readiness-updated 22", claim.SSHPort) + } +} + +func TestFalAcquireReconcilesAmbiguousCreateWithIdempotentRetry(t *testing.T) { + api := &fakeFalAPI{createErrs: []error{io.ErrUnexpectedEOF}} + b := newFalTestBackend(t, api) + + lease, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "retry-create"}) + if err != nil { + t.Fatal(err) + } + if lease.Server.CloudID != "inst_created" { + t.Fatalf("lease=%#v", lease) + } + if len(api.createRequests) != 2 { + t.Fatalf("createRequests=%#v", api.createRequests) + } + if api.idempotency[0] == "" || api.idempotency[0] != api.idempotency[1] || api.idempotency[0] != lease.LeaseID { + t.Fatalf("idempotency=%#v lease=%s", api.idempotency, lease.LeaseID) + } + claim, ok, claimErr := core.ResolveLeaseClaimForProvider("retry-create", providerName) + if claimErr != nil || !ok { + t.Fatalf("claim ok=%v err=%v", ok, claimErr) + } + if claim.CloudID != "inst_created" { + t.Fatalf("claim=%#v", claim) + } +} + +func TestFalAcquireReplaysExactNormalizedCreateRequest(t *testing.T) { + api := &fakeFalAPI{createErrs: []error{io.ErrUnexpectedEOF}} + b := newFalTestBackend(t, api) + b.cfg.Fal.InstanceType = " " + string(InstanceTypeH100x8) + " " + b.cfg.Fal.Sector = " " + string(Sector2) + " " + + if _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "normalized-replay"}); err != nil { + t.Fatal(err) + } + if len(api.createRequests) != 2 || api.createRequests[0] != api.createRequests[1] { + t.Fatalf("create requests=%#v", api.createRequests) + } + want := CreateInstanceRequest{InstanceType: InstanceTypeH100x8, Sector: Sector2, SSHKey: api.createRequests[0].SSHKey} + if api.createRequests[0] != want { + t.Fatalf("create request=%#v want %#v", api.createRequests[0], want) + } +} + +func TestFalAcquireDoesNotReplayExplicitRateLimitRejection(t *testing.T) { + api := &fakeFalAPI{createErr: &APIError{StatusCode: 429, Status: "429 Too Many Requests", Message: "rate limited"}} + b := newFalTestBackend(t, api) + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "rate-limited"}) + if err == nil || !strings.Contains(err.Error(), "rate limited") { + t.Fatalf("err=%v", err) + } + if len(api.createRequests) != 1 { + t.Fatalf("rate-limit rejection replayed create: requests=%d", len(api.createRequests)) + } + if _, ok, claimErr := core.ResolveLeaseClaimForProvider("rate-limited", providerName); claimErr != nil || ok { + t.Fatalf("rate-limit rejection persisted recovery claim: ok=%v err=%v", ok, claimErr) + } + keyPath, keyErr := core.TestboxKeyPath(api.idempotency[0]) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, statErr := os.Stat(keyPath); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("rate-limit rejection key still exists: %v", statErr) + } +} + +func TestFalDefinitiveCreateRejectionCannotBeReplayedAfterCleanupFailure(t *testing.T) { + api := &fakeFalAPI{createErr: &APIError{StatusCode: 403, Status: "403 Forbidden", Message: "forbidden"}} + b := newFalTestBackend(t, api) + cleanupErr := errors.New("key cleanup unavailable") + b.removeLeaseKey = func(string) error { return cleanupErr } + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "terminal-rejection"}) + if err == nil || !strings.Contains(err.Error(), "forbidden") || !strings.Contains(err.Error(), cleanupErr.Error()) { + t.Fatalf("acquire err=%v", err) + } + if len(api.createRequests) != 1 { + t.Fatalf("create requests=%d want one rejected request", len(api.createRequests)) + } + leaseID := api.idempotency[0] + claim, exists, readErr := core.ReadLeaseClaimWithPresence(leaseID) + if readErr != nil || !exists || claim.CloudID != "" || claim.Labels["recovery"] != falCreateRejectedRecovery { + t.Fatalf("claim=%#v exists=%v err=%v", claim, exists, readErr) + } + + b.removeLeaseKey = nil + target, err := b.Resolve(context.Background(), core.ResolveRequest{ID: leaseID, ReleaseOnly: true}) + if err != nil { + t.Fatal(err) + } + if err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: target}); err != nil { + t.Fatal(err) + } + if len(api.createRequests) != 1 { + t.Fatalf("release replayed rejected create: requests=%d", len(api.createRequests)) + } + if _, exists, readErr := core.ReadLeaseClaimWithPresence(leaseID); readErr != nil || exists { + t.Fatalf("terminal rejection claim exists=%v err=%v", exists, readErr) + } +} + +func TestFalOperationLockUsesDurableDirectoryPreparation(t *testing.T) { + t.Setenv("XDG_STATE_HOME", filepath.Join(t.TempDir(), "fresh", "state")) + oldEnsure := ensureFalClaimNamespace + t.Cleanup(func() { ensureFalClaimNamespace = oldEnsure }) + prepared := 0 + ensureFalClaimNamespace = func() error { + prepared++ + stateDir, err := core.CrabboxStateDir() + if err != nil { + return err + } + return os.MkdirAll(filepath.Join(stateDir, "claims"), 0o700) + } + + unlock, err := lockFalSlugAllocation(context.Background()) + if err != nil { + t.Fatal(err) + } + unlock() + stateDir, err := core.CrabboxStateDir() + if err != nil { + t.Fatal(err) + } + want := filepath.Join(stateDir, "claim-locks") + if prepared != 1 { + t.Fatalf("state namespace preparations=%d want 1", prepared) + } + if info, err := os.Stat(want); err != nil || !info.IsDir() { + t.Fatalf("lock directory stat=%v info=%v", err, info) + } +} + +func TestFalAcquireStopsBeforeProviderWhenLockNamespaceIsNotDurable(t *testing.T) { + oldEnsure := ensureFalClaimNamespace + t.Cleanup(func() { ensureFalClaimNamespace = oldEnsure }) + syncErr := errors.New("lock namespace sync unavailable") + ensureFalClaimNamespace = func() error { return syncErr } + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "unsafe-lock-namespace"}) + if err == nil || !strings.Contains(err.Error(), syncErr.Error()) { + t.Fatalf("acquire err=%v", err) + } + if len(api.createRequests) != 0 { + t.Fatalf("provider mutation occurred before durable lock setup: %#v", api.createRequests) + } +} + +func TestFalAcquireRetainsIntentAfterDefinitiveReplayRejection(t *testing.T) { + api := &fakeFalAPI{createErrs: []error{ + io.ErrUnexpectedEOF, + &APIError{StatusCode: 401, Status: "401 Unauthorized", Message: "unauthorized"}, + }} + b := newFalTestBackend(t, api) + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "replay-rejected"}) + if err == nil || !strings.Contains(err.Error(), "unauthorized") { + t.Fatalf("acquire err=%v", err) + } + if len(api.idempotency) != 2 || api.idempotency[0] != api.idempotency[1] { + t.Fatalf("idempotency=%#v", api.idempotency) + } + claim, exists, readErr := core.ReadLeaseClaimWithPresence(api.idempotency[0]) + if readErr != nil || !exists || claim.CloudID != "" || claim.Labels["recovery"] != "ambiguous-create" { + t.Fatalf("claim=%#v exists=%v err=%v", claim, exists, readErr) + } + keyPath, keyErr := core.TestboxKeyPath(api.idempotency[0]) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, statErr := os.Stat(keyPath); statErr != nil { + t.Fatalf("rejected replay key missing: %v", statErr) + } +} + +func TestFalAcquireAnchorsTTLToCreateAttempt(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + started := time.Date(2026, time.July, 9, 12, 0, 0, 0, time.UTC) + clock := &mutableFalClock{now: started} + b.rt.Clock = clock + b.cfg.TTL = 20 * time.Minute + b.cfg.IdleTimeout = time.Hour + b.waitSSH = func(context.Context, *core.SSHTarget, string, time.Duration) error { + clock.now = started.Add(10 * time.Minute) + return nil + } + + lease, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "ttl-anchor"}) + if err != nil { + t.Fatal(err) + } + wantCreated := strconv.FormatInt(started.Unix(), 10) + wantExpires := strconv.FormatInt(started.Add(20*time.Minute).Unix(), 10) + if lease.Server.Labels["created_at"] != wantCreated || lease.Server.Labels["expires_at"] != wantExpires { + t.Fatalf("lease labels=%#v want created=%s expires=%s", lease.Server.Labels, wantCreated, wantExpires) + } + claim, ok, err := core.ResolveLeaseClaimForProvider("ttl-anchor", providerName) + if err != nil || !ok || claim.Labels["created_at"] != wantCreated || claim.Labels["expires_at"] != wantExpires { + t.Fatalf("claim=%#v ok=%v err=%v", claim, ok, err) + } +} + +func TestFalReconcileRefusesReplayAfterIdempotencyWindow(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + started := b.now().Add(-falCreateRecoveryWindow) + req := CreateInstanceRequest{InstanceType: InstanceTypeH100x1, SSHKey: "ssh-ed25519 test"} + claim, err := b.persistRecoveryClaimAtIfUnchanged("cbx_abcdef123456", "expired-replay", b.configForRun(), "", "", "ambiguous-create", false, started, core.LeaseClaim{}, false, req) + if err != nil { + t.Fatal(err) + } + _, _, err = b.reconcileAmbiguousCreate(context.Background(), api, req, claim, b.configForRun(), false, io.ErrUnexpectedEOF) + if err == nil || !strings.Contains(err.Error(), "recovery window expired") { + t.Fatalf("err=%v", err) + } + if len(api.createRequests) != 0 { + t.Fatalf("expired idempotency replay issued create: %#v", api.createRequests) + } +} + +func TestFalReconcileDeadlinesReplayInsideIdempotencyWindow(t *testing.T) { + api := &fakeFalAPI{blockCreateUntilContext: true} + b := newFalTestBackend(t, api) + started := time.Now().Add(-falCreateRecoveryWindow + 2*time.Second) + req := CreateInstanceRequest{InstanceType: InstanceTypeH100x1, SSHKey: "ssh-ed25519 test"} + claim, err := b.persistRecoveryClaimAtIfUnchanged("cbx_abcdef123456", "bounded-replay", b.configForRun(), "", "", "ambiguous-create", false, started, core.LeaseClaim{}, false, req) + if err != nil { + t.Fatal(err) + } + begin := time.Now() + _, _, err = b.reconcileAmbiguousCreate(context.Background(), api, req, claim, b.configForRun(), false, io.ErrUnexpectedEOF) + if err == nil || !strings.Contains(err.Error(), "idempotency replay window expired") { + t.Fatalf("err=%v", err) + } + if elapsed := time.Since(begin); elapsed > 3*time.Second { + t.Fatalf("replay outlived idempotency deadline: %v", elapsed) + } + if len(api.createRequests) != 1 { + t.Fatalf("create requests=%d, want one bounded replay", len(api.createRequests)) + } +} + +func TestFalStopRecoversAmbiguousCreateWithExactIdempotentRequest(t *testing.T) { + api := &fakeFalAPI{createErrs: []error{io.ErrUnexpectedEOF, io.ErrUnexpectedEOF, io.ErrUnexpectedEOF, io.ErrUnexpectedEOF}} + b := newFalTestBackend(t, api) + b.cfg.Fal.InstanceType = string(InstanceTypeH100x8) + b.cfg.Fal.Sector = string(Sector2) + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "unreconciled-create"}) + if err == nil || !strings.Contains(err.Error(), "indeterminate after idempotent retry") || + !strings.Contains(err.Error(), "lease=") || !strings.Contains(err.Error(), "slug=unreconciled-create") || + !strings.Contains(err.Error(), "crabbox stop --provider fal --id") || !strings.Contains(err.Error(), "immediately") { + t.Fatalf("err=%v", err) + } + if len(api.createRequests) != 4 { + t.Fatalf("createRequests=%#v", api.createRequests) + } + claim, ok, claimErr := core.ResolveLeaseClaimForProvider("unreconciled-create", providerName) + if claimErr != nil || !ok || claim.CloudID != "" || claim.Labels["recovery"] != "ambiguous-create" { + t.Fatalf("recovery claim=%#v ok=%v err=%v", claim, ok, claimErr) + } + if claim.Labels["create_started_at"] == "" { + t.Fatalf("recovery claim missing initial mutation time: %#v", claim.Labels) + } + views, err := b.List(context.Background(), core.ListRequest{}) + if err != nil { + t.Fatal(err) + } + if len(views) != 1 || views[0].Status != "ambiguous-create" || views[0].CloudID != "" { + t.Fatalf("recovery views=%#v", views) + } + target, err := b.Resolve(context.Background(), core.ResolveRequest{ID: "unreconciled-create", ReleaseOnly: true}) + if err != nil { + t.Fatal(err) + } + if target.Server.CloudID != "inst_created" { + t.Fatalf("target=%#v", target) + } + if len(api.createRequests) != 5 || api.idempotency[4] != claim.LeaseID { + t.Fatalf("createRequests=%d idempotency=%#v", len(api.createRequests), api.idempotency) + } + if got := api.createRequests[4]; got.InstanceType != InstanceTypeH100x8 || got.Sector != Sector2 || !strings.HasPrefix(got.SSHKey, "ssh-") { + t.Fatalf("recovery request=%#v", got) + } + if err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: target}); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_created" { + t.Fatalf("deletedIDs=%#v", api.deletedIDs) + } + if _, ok, claimErr := core.ResolveLeaseClaimForProvider("unreconciled-create", providerName); claimErr != nil || ok { + t.Fatalf("recovery claim retained ok=%v err=%v", ok, claimErr) + } +} + +func TestFalStopRecoveryRetainsIntentAfterDefinitiveRejection(t *testing.T) { + api := &fakeFalAPI{createErr: &APIError{StatusCode: 403, Status: "403 Forbidden", Message: "forbidden"}} + b := newFalTestBackend(t, api) + const leaseID = "cbx_rejected_recovery" + claim, _ := persistFalCreateRecoveryClaim(t, b, leaseID, "rejected-recovery", "ambiguous-create", false, time.Time{}) + + _, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, claim, b.configForRun()) + if err == nil || !strings.Contains(err.Error(), "forbidden") { + t.Fatalf("recovery err=%v", err) + } + retained, exists, readErr := core.ReadLeaseClaimWithPresence(leaseID) + if readErr != nil || !exists || retained.CloudID != "" || retained.Labels["recovery"] != "ambiguous-create" { + t.Fatalf("claim=%#v exists=%v err=%v", retained, exists, readErr) + } + keyPath, keyErr := core.TestboxKeyPath(leaseID) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, statErr := os.Stat(keyPath); statErr != nil { + t.Fatalf("rejected recovery key missing: %v", statErr) + } +} + +func TestFalStopCancelsCreateIntentWithoutProviderMutation(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + const leaseID = "cbx_cancel_create_intent" + claim, _ := persistFalCreateRecoveryClaim(t, b, leaseID, "cancel-create-intent", "create-intent", false, time.Time{}) + + target, err := b.Resolve(context.Background(), core.ResolveRequest{ID: leaseID, ReleaseOnly: true}) + if err != nil { + t.Fatal(err) + } + if target.LeaseID != leaseID || target.Server.CloudID != "" { + t.Fatalf("target=%#v", target) + } + if err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: target}); err != nil { + t.Fatal(err) + } + if len(api.createRequests) != 0 || len(api.deletedIDs) != 0 { + t.Fatalf("provider mutations create=%d delete=%#v", len(api.createRequests), api.deletedIDs) + } + if _, exists, readErr := core.ReadLeaseClaimWithPresence(claim.LeaseID); readErr != nil || exists { + t.Fatalf("claim exists=%v err=%v", exists, readErr) + } + keyPath, keyErr := core.TestboxKeyPath(leaseID) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, statErr := os.Stat(keyPath); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("cancelled intent key still exists: %v", statErr) + } +} + +func TestFalStopRecoveryRefusesMissingOrReplacedKeyBeforeMutation(t *testing.T) { + for _, tc := range []struct { + name string + mutate func(string) error + message string + }{ + { + name: "missing public key", + mutate: func(path string) error { return os.Remove(path + ".pub") }, + message: "public key is unavailable", + }, + { + name: "replaced public key", + mutate: func(path string) error { + return os.WriteFile(path+".pub", []byte("ssh-ed25519 replaced-key\n"), 0o600) + }, + message: "create request changed", + }, + } { + t.Run(tc.name, func(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + leaseID := "cbx_" + strings.ReplaceAll(tc.name, " ", "_") + claim, _ := persistFalCreateRecoveryClaim(t, b, leaseID, "key-binding", "ambiguous-create", false, time.Time{}) + keyPath, err := core.TestboxKeyPath(leaseID) + if err != nil { + t.Fatal(err) + } + if err := tc.mutate(keyPath); err != nil { + t.Fatal(err) + } + + if _, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, claim, b.configForRun()); err == nil || !strings.Contains(err.Error(), tc.message) { + t.Fatalf("recovery err=%v", err) + } + if len(api.createRequests) != 0 { + t.Fatalf("provider mutation occurred with changed key: %#v", api.createRequests) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(leaseID); err != nil || !exists { + t.Fatalf("claim exists=%v err=%v", exists, err) + } + }) + } +} + +func TestFalStopRecoveryRefusesWhitespaceChangedRequestBeforeMutation(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.cfg.Fal.InstanceType = string(InstanceTypeH100x8) + b.cfg.Fal.Sector = string(Sector2) + claim, _ := persistFalCreateRecoveryClaim(t, b, "cbx_changed_request", "changed-request", "ambiguous-create", false, time.Time{}) + labels := cloneLabels(claim.Labels) + labels["sector"] = " " + string(Sector2) + " " + claim, err := core.UpdateLeaseClaimLabelsIfUnchanged(claim.LeaseID, claim, labels) + if err != nil { + t.Fatal(err) + } + + if _, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, claim, b.configForRun()); err == nil || !strings.Contains(err.Error(), "create request changed") { + t.Fatalf("recovery err=%v", err) + } + if len(api.createRequests) != 0 { + t.Fatalf("provider mutation occurred with changed request: %#v", api.createRequests) + } +} + +func TestFalAmbiguousCreateRecoveryResumesInterruptedInflightAttempt(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + claim, _ := persistFalCreateRecoveryClaim(t, b, "cbx_recovery123", "recovery-race", "ambiguous-create", false, time.Time{}) + inflight, err := falRecoveryClaimReplacement(claim, b.configForRun(), "", "ambiguous-create-inflight", false) + if err != nil { + t.Fatal(err) + } + inflight.Labels[falCreateAttemptLabel] = "interrupted-attempt" + if err := core.ReplaceLeaseClaimIfUnchangedDurable(claim.LeaseID, claim, inflight); err != nil { + t.Fatal(err) + } + + updated, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, inflight, b.configForRun()) + if err != nil { + t.Fatal(err) + } + if updated.CloudID != "inst_created" || updated.Labels["recovery"] != "rollback-cleanup" { + t.Fatalf("updated claim=%#v", updated) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("recovery deleted the resumed instance: %#v", api.deletedIDs) + } +} + +func TestFalConcurrentAmbiguousCreateRecoveryIssuesOneProviderRequest(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + claim, _ := persistFalCreateRecoveryClaim(t, b, "cbx_concurrent_recovery", "concurrent-recovery", "ambiguous-create", false, time.Time{}) + started := make(chan struct{}) + release := make(chan struct{}) + var once sync.Once + api.beforeCreateHook = func(string) { + once.Do(func() { + close(started) + <-release + }) + } + type result struct { + claim core.LeaseClaim + err error + } + firstResult := make(chan result, 1) + secondResult := make(chan result, 1) + go func() { + updated, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, claim, b.configForRun()) + firstResult <- result{claim: updated, err: err} + }() + <-started + go func() { + updated, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, claim, b.configForRun()) + secondResult <- result{claim: updated, err: err} + }() + close(release) + + first := <-firstResult + second := <-secondResult + if first.err != nil || first.claim.CloudID != "inst_created" { + t.Fatalf("first recovery claim=%#v err=%v", first.claim, first.err) + } + if second.err == nil || !strings.Contains(second.err.Error(), "claim changed") { + t.Fatalf("second recovery claim=%#v err=%v", second.claim, second.err) + } + if len(api.createRequests) != 1 { + t.Fatalf("provider create requests=%d want 1", len(api.createRequests)) + } +} + +func TestFalCreateWaiterHonorsCancellationWithoutHoldingClaimLock(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + claim, _ := persistFalCreateRecoveryClaim(t, b, "cbx_cancel_waiter", "cancel-waiter", "ambiguous-create", false, time.Time{}) + started := make(chan struct{}) + release := make(chan struct{}) + var once sync.Once + api.beforeCreateHook = func(string) { + once.Do(func() { + close(started) + <-release + }) + } + + firstResult := make(chan error, 1) + go func() { + _, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, claim, b.configForRun()) + firstResult <- err + }() + <-started + + inflight, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if err != nil || !exists { + t.Fatalf("inflight claim=%#v exists=%t err=%v", inflight, exists, err) + } + if inflight.Labels["recovery"] != "ambiguous-create-inflight" || inflight.Labels[falCreateAttemptLabel] == "" { + t.Fatalf("provider POST started without durable inflight claim: %#v", inflight) + } + verifyDone := make(chan error, 1) + go func() { verifyDone <- core.VerifyLeaseClaimUnchanged(claim.LeaseID, inflight) }() + select { + case err := <-verifyDone: + if err != nil { + t.Fatal(err) + } + case <-time.After(time.Second): + t.Fatal("provider POST held the generic claim lock") + } + + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) + defer cancel() + if _, err := b.recoverAmbiguousCreateForRelease(ctx, api, claim, b.configForRun()); !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("second recovery err=%v want context deadline", err) + } + if len(api.createRequests) != 1 { + t.Fatalf("provider create requests=%d want 1", len(api.createRequests)) + } + close(release) + if err := <-firstResult; err != nil { + t.Fatal(err) + } +} + +func TestFalOperationLocksSerializeAcrossProcesses(t *testing.T) { + if helper := os.Getenv("CRABBOX_FAL_LOCK_HELPER"); helper != "" { + ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond) + defer cancel() + var err error + switch helper { + case "lease": + _, err = lockFalLeaseOperation(ctx, "cbx_cross_process") + case "acquire": + _, err = lockFalAcquireLifetime(ctx, "cbx_cross_process") + case "slug": + _, err = lockFalSlugAllocation(ctx) + default: + t.Fatalf("unknown lock helper %q", helper) + } + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("lock err=%v want context deadline", err) + } + return + } + t.Setenv("XDG_STATE_HOME", t.TempDir()) + for _, test := range []struct { + name string + lock func(context.Context) (func(), error) + }{ + {name: "lease", lock: func(ctx context.Context) (func(), error) { + return lockFalLeaseOperation(ctx, "cbx_cross_process") + }}, + {name: "acquire", lock: func(ctx context.Context) (func(), error) { + return lockFalAcquireLifetime(ctx, "cbx_cross_process") + }}, + {name: "slug", lock: lockFalSlugAllocation}, + } { + t.Run(test.name, func(t *testing.T) { + unlock, err := test.lock(context.Background()) + if err != nil { + t.Fatal(err) + } + cmd := exec.Command(os.Args[0], "-test.run=^TestFalOperationLocksSerializeAcrossProcesses$") + cmd.Env = append(os.Environ(), "CRABBOX_FAL_LOCK_HELPER="+test.name) + output, runErr := cmd.CombinedOutput() + unlock() + if runErr != nil { + t.Fatalf("helper err=%v output=%s", runErr, output) + } + }) + } +} + +func TestFalRequestedSlugReservationSerializesClaimPublication(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + unlock, err := lockFalSlugAllocation(context.Background()) + if err != nil { + t.Fatal(err) + } + type result struct { + slug string + err error + } + results := make(chan result, 2) + for _, leaseID := range []string{"cbx_slug_race_one", "cbx_slug_race_two"} { + leaseID := leaseID + go func() { + unlockSlug, err := lockFalSlugAllocation(context.Background()) + if err != nil { + results <- result{err: err} + return + } + defer unlockSlug() + slug, err := core.AllocateClaimLeaseSlug(leaseID, "shared-name") + if err == nil { + _, err = b.persistRecoveryClaimAtIfUnchanged( + leaseID, slug, b.configForRun(), "", "", "create-intent", false, b.now(), core.LeaseClaim{}, false, + CreateInstanceRequest{InstanceType: InstanceTypeH100x1, SSHKey: "ssh-ed25519 synthetic"}, + ) + } + results <- result{slug: slug, err: err} + }() + } + time.Sleep(20 * time.Millisecond) + unlock() + first := <-results + second := <-results + if first.err != nil || second.err != nil { + t.Fatalf("results=%#v %#v", first, second) + } + if first.slug == second.slug { + t.Fatalf("concurrent requested slugs collided: %q", first.slug) + } + if first.slug != "shared-name" && second.slug != "shared-name" { + t.Fatalf("requested slug was not preserved: %q %q", first.slug, second.slug) + } +} + +func TestFalSlugReservationSerializesCollisionSuffixAgainstDirectRequest(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + if _, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_slug_seed", "shared-name", b.configForRun(), "", "", "create-intent", false, b.now(), core.LeaseClaim{}, false, + CreateInstanceRequest{InstanceType: InstanceTypeH100x1, SSHKey: "ssh-ed25519 seed"}, + ); err != nil { + t.Fatal(err) + } + const firstLeaseID = "cbx_slug_suffix_one" + const secondLeaseID = "cbx_slug_suffix_two" + collisionSlug, err := core.AllocateClaimLeaseSlug(firstLeaseID, "shared-name") + if err != nil { + t.Fatal(err) + } + unlock, err := lockFalSlugAllocation(context.Background()) + if err != nil { + t.Fatal(err) + } + type result struct { + slug string + err error + } + results := make(chan result, 2) + for _, item := range []struct { + leaseID string + requested string + }{ + {leaseID: firstLeaseID, requested: "shared-name"}, + {leaseID: secondLeaseID, requested: collisionSlug}, + } { + item := item + go func() { + unlockSlug, err := lockFalSlugAllocation(context.Background()) + if err != nil { + results <- result{err: err} + return + } + defer unlockSlug() + slug, err := core.AllocateClaimLeaseSlug(item.leaseID, item.requested) + if err == nil { + _, err = b.persistRecoveryClaimAtIfUnchanged( + item.leaseID, slug, b.configForRun(), "", "", "create-intent", false, b.now(), core.LeaseClaim{}, false, + CreateInstanceRequest{InstanceType: InstanceTypeH100x1, SSHKey: "ssh-ed25519 synthetic"}, + ) + } + results <- result{slug: slug, err: err} + }() + } + time.Sleep(20 * time.Millisecond) + unlock() + first := <-results + second := <-results + if first.err != nil || second.err != nil { + t.Fatalf("results=%#v %#v", first, second) + } + if first.slug == second.slug { + t.Fatalf("collision suffix was published twice: %q", first.slug) + } +} + +func TestFalStopRecoveryDeadlinesReplayInsideIdempotencyWindow(t *testing.T) { + api := &fakeFalAPI{blockCreateUntilContext: true} + b := newFalTestBackend(t, api) + created := time.Unix(time.Now().Unix(), 0).UTC() + clock := &mutableFalClock{now: created.Add(falCreateRecoveryWindow - 75*time.Millisecond)} + b.rt.Clock = clock + const leaseID = "cbx_recovery789" + claim, _ := persistFalCreateRecoveryClaim(t, b, leaseID, "recovery-deadline", "ambiguous-create", false, created) + + begin := time.Now() + _, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, claim, b.configForRun()) + if err == nil || !strings.Contains(err.Error(), "recovery retry failed") || !strings.Contains(err.Error(), context.DeadlineExceeded.Error()) { + t.Fatalf("recovery err=%v", err) + } + if elapsed := time.Since(begin); elapsed > time.Second { + t.Fatalf("stop recovery outlived idempotency deadline: %v", elapsed) + } + if _, exists, readErr := core.ReadLeaseClaimWithPresence(leaseID); readErr != nil || !exists { + t.Fatalf("claim exists=%v err=%v", exists, readErr) + } +} + +func TestFalAmbiguousCreateRecoveryCleansUpAfterClaimWriteFailure(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + const leaseID = "cbx_recovery456" + claim, _ := persistFalCreateRecoveryClaim(t, b, leaseID, "recovery-write-fail", "ambiguous-create", false, time.Time{}) + b.recoveryClaimReplacement = func(current core.LeaseClaim, cfg Config, instanceID, reason string, keep bool) (core.LeaseClaim, error) { + if strings.TrimSpace(instanceID) != "" { + return core.LeaseClaim{}, errors.New("disk full") + } + return falRecoveryClaimReplacement(current, cfg, instanceID, reason, keep) + } + + _, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, claim, b.configForRun()) + if err == nil || !strings.Contains(err.Error(), "disk full") || !strings.Contains(err.Error(), "inst_created") { + t.Fatalf("recovery err=%v", err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_created" { + t.Fatalf("known recovered instance was not cleaned up: %#v", api.deletedIDs) + } + if _, exists, readErr := core.ReadLeaseClaimWithPresence(leaseID); readErr != nil || exists { + t.Fatalf("claim exists=%v err=%v", exists, readErr) + } +} + +func TestFalAcquireBindFailureRetainsImmediateCleanupClaim(t *testing.T) { + api := &fakeFalAPI{deleteErr: errors.New("delete unavailable")} + b := newFalTestBackend(t, api) + writes := 0 + b.recoveryClaimReplacement = func(current core.LeaseClaim, cfg Config, instanceID, reason string, keep bool) (core.LeaseClaim, error) { + if strings.TrimSpace(instanceID) != "" { + writes++ + if writes <= 4 { + return core.LeaseClaim{}, errors.New("claim write unavailable") + } + } + return falRecoveryClaimReplacement(current, cfg, instanceID, reason, keep) + } + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "bind-failure-cleanup"}) + if err == nil || !strings.Contains(err.Error(), "claim write unavailable") || !strings.Contains(err.Error(), "delete unavailable") { + t.Fatalf("acquire err=%v", err) + } + claim, exists, readErr := core.ReadLeaseClaimWithPresence(api.idempotency[0]) + if readErr != nil || !exists || claim.CloudID != "inst_created" || claim.Labels["recovery"] != "rollback-cleanup" { + t.Fatalf("claim=%#v exists=%t err=%v", claim, exists, readErr) + } +} + +func TestFalKnownInstanceAdoptionRequiresDurableRewrite(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + claim, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_known_rewrite", + "known-rewrite", + b.configForRun(), + "", + "inst_created", + "provisioning", + false, + b.now(), + core.LeaseClaim{}, + false, + ) + if err != nil { + t.Fatal(err) + } + b.recoveryClaimReplacement = func(core.LeaseClaim, Config, string, string, bool) (core.LeaseClaim, error) { + return core.LeaseClaim{}, errors.New("directory sync unavailable") + } + + _, exists, err := b.adoptOrBindKnownFalInstance(claim, b.configForRun(), "inst_created", "provisioning", false) + if err == nil || !exists || !strings.Contains(err.Error(), "directory sync unavailable") { + t.Fatalf("adopt exists=%v err=%v", exists, err) + } +} + +func TestFalStopRetainsAmbiguousClaimAfterIdempotencyWindow(t *testing.T) { + api := &fakeFalAPI{createErrs: []error{io.ErrUnexpectedEOF, io.ErrUnexpectedEOF, io.ErrUnexpectedEOF, io.ErrUnexpectedEOF}} + b := newFalTestBackend(t, api) + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "expired-recovery"}) + if err == nil { + t.Fatal("expected ambiguous create failure") + } + claim, ok, err := core.ResolveLeaseClaimForProvider("expired-recovery", providerName) + if err != nil || !ok { + t.Fatalf("claim ok=%v err=%v", ok, err) + } + labels := cloneLabels(claim.Labels) + labels["create_started_at"] = strconv.FormatInt(time.Now().Add(-falCreateRecoveryWindow-time.Minute).Unix(), 10) + claim, err = core.UpdateLeaseClaimLabelsIfUnchanged(claim.LeaseID, claim, labels) + if err != nil { + t.Fatal(err) + } + if _, err := b.Resolve(context.Background(), core.ResolveRequest{ID: claim.LeaseID, ReleaseOnly: true}); err == nil || !strings.Contains(err.Error(), "recovery window expired") { + t.Fatalf("resolve err=%v", err) + } + if len(api.createRequests) != 4 { + t.Fatalf("expired recovery replayed create: requests=%d", len(api.createRequests)) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || !exists { + t.Fatalf("claim exists=%v err=%v", exists, err) + } +} + +func TestFalStopRefusesAmbiguousCreateReplayWithDifferentCredential(t *testing.T) { + api := &fakeFalAPI{createErrs: []error{io.ErrUnexpectedEOF, io.ErrUnexpectedEOF, io.ErrUnexpectedEOF, io.ErrUnexpectedEOF}} + b := newFalTestBackend(t, api) + + _, err := b.Acquire(context.Background(), core.AcquireRequest{RequestedSlug: "credential-bound-recovery"}) + if err == nil { + t.Fatal("expected ambiguous create failure") + } + b.cfg.Fal.APIKey = "different-test-key" + _, err = b.Resolve(context.Background(), core.ResolveRequest{ID: "credential-bound-recovery", ReleaseOnly: true}) + if err == nil || !strings.Contains(err.Error(), "different credential identity") { + t.Fatalf("resolve err=%v", err) + } + if len(api.createRequests) != 4 { + t.Fatalf("credential mismatch replayed create: requests=%d", len(api.createRequests)) + } +} + +func TestFalAcquireRollsBackOnCallbackFailure(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + _, err := b.Acquire(context.Background(), core.AcquireRequest{ + RequestedSlug: "rollback", + OnAcquired: func(core.LeaseTarget) error { + return errors.New("controller rejected identity") + }, + }) + if err == nil || !strings.Contains(err.Error(), "controller rejected identity") { + t.Fatalf("err=%v", err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_created" { + t.Fatalf("deletedIDs=%#v", api.deletedIDs) + } + if _, ok, claimErr := core.ResolveLeaseClaimForProvider("rollback", providerName); claimErr != nil || ok { + t.Fatalf("rollback claim ok=%v err=%v", ok, claimErr) + } +} + +func TestFalAcquireKeepFailurePersistsRecoveryClaim(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.waitSSH = func(context.Context, *core.SSHTarget, string, time.Duration) error { + return errors.New("ssh not ready") + } + _, err := b.Acquire(context.Background(), core.AcquireRequest{ + RequestedSlug: "keep-failed", + Keep: true, + }) + if err == nil || !strings.Contains(err.Error(), "ssh not ready") { + t.Fatalf("err=%v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("keep failure deleted instance: %#v", api.deletedIDs) + } + claim, ok, claimErr := core.ResolveLeaseClaimForProvider("keep-failed", providerName) + if claimErr != nil || !ok { + t.Fatalf("recovery claim ok=%v err=%v", ok, claimErr) + } + if claim.CloudID != "inst_created" || claim.Labels["recovery"] != "keep-failed-acquire" || claim.Labels["keep"] != "true" { + t.Fatalf("claim=%#v", claim) + } +} + +func TestFalKeepFailureRetainsKnownInstanceWhenClaimAndOperationStateAreUnavailable(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_created": readyFalInstance("inst_created", "203.0.113.42")}, + } + b := newFalTestBackend(t, api) + stateFile := filepath.Join(t.TempDir(), "state-file") + if err := os.WriteFile(stateFile, []byte("not a directory"), 0o600); err != nil { + t.Fatal(err) + } + t.Setenv("XDG_STATE_HOME", stateFile) + err := b.handleFailedAcquire("inst_created", "cbx_abcdef123456", "keep-failed", b.configForRun(), "", true, errors.New("ssh not ready")) + if err == nil || !strings.Contains(err.Error(), "ssh not ready") || !strings.Contains(err.Error(), "persist fal keep recovery claim") || !strings.Contains(err.Error(), "create claim lock directory") { + t.Fatalf("err=%v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("provider mutation issued without durable state: %#v", api.deletedIDs) + } + if _, ok := api.instances["inst_created"]; !ok { + t.Fatal("known instance removed without durable cleanup serialization") + } +} + +func TestFalAcquireOnAcquiredFailureRollsBackEvenWithKeep(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + _, err := b.Acquire(context.Background(), core.AcquireRequest{ + RequestedSlug: "keep-callback-fail", + Keep: true, + OnAcquired: func(core.LeaseTarget) error { + return errors.New("controller rejected identity") + }, + }) + if err == nil || !strings.Contains(err.Error(), "controller rejected identity") { + t.Fatalf("err=%v", err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_created" { + t.Fatalf("callback failure did not roll back: %#v", api.deletedIDs) + } + if _, ok, claimErr := core.ResolveLeaseClaimForProvider("keep-callback-fail", providerName); claimErr != nil || ok { + t.Fatalf("rollback claim ok=%v err=%v", ok, claimErr) + } +} + +func TestFalAcquireAcknowledgesProviderIdentityBeforeSSHWait(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + b.waitSSH = func(context.Context, *core.SSHTarget, string, time.Duration) error { + return errors.New("ssh not ready") + } + var observed core.LeaseTarget + _, err := b.Acquire(context.Background(), core.AcquireRequest{ + RequestedSlug: "ack-first", + Keep: true, + OnAcquired: func(target core.LeaseTarget) error { + observed = target + claim, ok, claimErr := core.ReadLeaseClaimWithPresence(target.LeaseID) + if claimErr != nil || !ok || claim.CloudID != target.Server.CloudID || claim.Labels["recovery"] != "provisioning" { + t.Fatalf("durable pre-readiness claim=%#v ok=%v err=%v", claim, ok, claimErr) + } + return nil + }, + }) + if err == nil || !strings.Contains(err.Error(), "ssh not ready") { + t.Fatalf("err=%v", err) + } + if observed.Server.CloudID != "inst_created" || observed.SSH.Host != "203.0.113.42" || observed.LeaseID == "" { + t.Fatalf("OnAcquired did not receive provider identity before SSH wait: %#v", observed) + } + claim, ok, claimErr := core.ResolveLeaseClaimForProvider("ack-first", providerName) + if claimErr != nil || !ok || claim.CloudID != "inst_created" { + t.Fatalf("recovery claim=%#v ok=%v err=%v", claim, ok, claimErr) + } +} + +func TestFalAcquireAcknowledgesProviderIdentityBeforeReadinessPolling(t *testing.T) { + api := &fakeFalAPI{getErr: errors.New("readiness API unavailable")} + b := newFalTestBackend(t, api) + var observed core.LeaseTarget + _, err := b.Acquire(context.Background(), core.AcquireRequest{ + RequestedSlug: "ack-before-readiness", + Keep: true, + OnAcquired: func(target core.LeaseTarget) error { + observed = target + return nil + }, + }) + if err == nil || !strings.Contains(err.Error(), "readiness API unavailable") { + t.Fatalf("err=%v", err) + } + if observed.Server.CloudID != "inst_created" || observed.LeaseID == "" || observed.Server.Labels["slug"] != "ack-before-readiness" { + t.Fatalf("OnAcquired did not receive provider identity before readiness polling: %#v", observed) + } + claim, ok, claimErr := core.ResolveLeaseClaimForProvider("ack-before-readiness", providerName) + if claimErr != nil || !ok || claim.CloudID != "inst_created" { + t.Fatalf("recovery claim=%#v ok=%v err=%v", claim, ok, claimErr) + } +} + +func TestFalProvisioningClaimIsCleanupOnly(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_provisioning": readyFalInstance("inst_provisioning", "203.0.113.42"), + }} + b := newFalTestBackend(t, api) + claim, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_abcdef123456", + "provisioning", + b.configForRun(), + "", + "inst_provisioning", + "provisioning", + false, + b.now(), + core.LeaseClaim{}, + false, + ) + if err != nil { + t.Fatal(err) + } + if _, err := b.Resolve(context.Background(), core.ResolveRequest{ID: claim.LeaseID}); err == nil || !strings.Contains(err.Error(), "still provisioning") { + t.Fatalf("resolve err=%v", err) + } + if _, err := b.Touch(context.Background(), core.TouchRequest{Lease: core.LeaseTarget{LeaseID: claim.LeaseID}}); err == nil || !strings.Contains(err.Error(), "still provisioning") { + t.Fatalf("touch err=%v", err) + } + target, err := b.Resolve(context.Background(), core.ResolveRequest{ID: claim.LeaseID, ReleaseOnly: true}) + if err != nil || target.Server.CloudID != "inst_provisioning" { + t.Fatalf("release-only target=%#v err=%v", target, err) + } +} + +func TestFalAmbiguousCreateReportsRecoveryClaimFailure(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + stateFile := filepath.Join(t.TempDir(), "state-file") + if err := os.WriteFile(stateFile, []byte("not a directory"), 0o600); err != nil { + t.Fatal(err) + } + t.Setenv("XDG_STATE_HOME", stateFile) + err := b.persistRecoveryClaim("cbx_abcdef123456", "ambiguous", b.configForRun(), "", "", "ambiguous-create", false) + if err == nil { + t.Fatal("expected recovery claim persistence error") + } + if !strings.Contains(err.Error(), "not a directory") { + t.Fatalf("err=%v", err) + } +} + +func TestFalRollbackReportsRecoveryClaimFailureWhenCleanupFails(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_created": readyFalInstance("inst_created", "203.0.113.42")}, + deleteErr: errors.New("delete unavailable"), + } + b := newFalTestBackend(t, api) + stateFile := filepath.Join(t.TempDir(), "state-file") + if err := os.WriteFile(stateFile, []byte("not a directory"), 0o600); err != nil { + t.Fatal(err) + } + t.Setenv("XDG_STATE_HOME", stateFile) + err := b.rollbackAcquire("inst_created", "cbx_abcdef123456", "rollback", b.configForRun(), "", "rollback-cleanup", errors.New("bootstrap failed")) + if err == nil { + t.Fatal("expected rollback error") + } + message := err.Error() + if !strings.Contains(message, "bootstrap failed") || !strings.Contains(message, "create fal claim namespace") { + t.Fatalf("err=%v", err) + } +} + +func TestFalCleanupImmediatelyRetriesRollbackClaim(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_created": readyFalInstance("inst_created", "203.0.113.42")}, + deleteErr: errors.New("delete unavailable"), + } + b := newFalTestBackend(t, api) + err := b.rollbackAcquire("inst_created", "cbx_abcdef123456", "rollback", b.configForRun(), "", "rollback-cleanup", errors.New("bootstrap failed")) + if err == nil || !strings.Contains(err.Error(), "delete unavailable") { + t.Fatalf("rollback err=%v", err) + } + claim, ok, err := core.ResolveLeaseClaimForProvider("rollback", providerName) + if err != nil || !ok || claim.Labels["recovery"] != "rollback-cleanup" { + t.Fatalf("claim=%#v ok=%v err=%v", claim, ok, err) + } + + api.deleteErr = nil + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 2 || api.deletedIDs[1] != "inst_created" { + t.Fatalf("deletedIDs=%#v", api.deletedIDs) + } + if _, ok, err := core.ResolveLeaseClaimForProvider("rollback", providerName); err != nil || ok { + t.Fatalf("rollback claim retained ok=%v err=%v", ok, err) + } +} + +func TestFalRollbackFinalizesRecreatedClaimAfterProviderConfirmedAbsence(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + claim, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_cleanup1234", + "cleanup-race", + b.configForRun(), + "", + "inst_created", + "provisioning", + false, + b.now(), + core.LeaseClaim{}, + false, + ) + if err != nil { + t.Fatal(err) + } + if _, _, err := core.EnsureTestboxKey(claim.LeaseID); err != nil { + t.Fatal(err) + } + seededKeyPath, err := core.TestboxKeyPath(claim.LeaseID) + if err != nil { + t.Fatal(err) + } + if _, err := os.Stat(seededKeyPath); err != nil { + t.Fatalf("seed rollback key: %v", err) + } + if err := core.RemoveLeaseClaimIfUnchanged(claim.LeaseID, claim); err != nil { + t.Fatal(err) + } + + cause := errors.New("ready transition lost race") + rollbackErr := b.rollbackClaimedAcquire("inst_created", claim.LeaseID, claim.Slug, b.configForRun(), "", "rollback-cleanup", cause) + if !errors.Is(rollbackErr, cause) || strings.Contains(rollbackErr.Error(), errFalProviderAbsenceNotAccountBound.Error()) { + t.Fatalf("rollback err=%v", rollbackErr) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("rollback issued a second delete after concurrent cleanup: %#v", api.deletedIDs) + } + _, exists, readErr := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if readErr != nil || exists { + t.Fatalf("claim exists=%v err=%v", exists, readErr) + } + keyPath, keyErr := core.TestboxKeyPath(claim.LeaseID) + if keyErr != nil { + t.Fatal(keyErr) + } + if _, err := os.Stat(keyPath); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("key residue after provider-confirmed absence: %v", err) + } +} + +func TestFalRollbackReconcilesClaimVisibleAfterDurabilityError(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_created": readyFalInstance("inst_created", "203.0.113.42"), + }} + b := newFalTestBackend(t, api) + b.persistRollbackClaim = func(leaseID, slug string, cfg Config, repoRoot, instanceID, reason string, keep bool) (core.LeaseClaim, error) { + claim, err := b.persistRecoveryClaimAtIfUnchanged( + leaseID, slug, cfg, repoRoot, instanceID, reason, keep, time.Time{}, core.LeaseClaim{}, false, + ) + if err != nil { + return core.LeaseClaim{}, err + } + return claim, errors.New("ancestor sync unavailable after rename") + } + cause := errors.New("ready transition lost claim") + err := b.rollbackAcquireAfterClaimRemoval("inst_created", "cbx_partial_durable", "partial-durable", b.configForRun(), "", "rollback-cleanup", cause) + if !errors.Is(err, cause) || !strings.Contains(err.Error(), "ancestor sync unavailable after rename") { + t.Fatalf("rollback err=%v", err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_created" { + t.Fatalf("deletedIDs=%#v", api.deletedIDs) + } + if _, exists, readErr := core.ReadLeaseClaimWithPresence("cbx_partial_durable"); readErr != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, readErr) + } +} + +func TestFalRollbackRetainsInstanceWhenNoDurableClaimCanBeWritten(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_created": readyFalInstance("inst_created", "203.0.113.42"), + }} + b := newFalTestBackend(t, api) + b.persistRollbackClaim = func(string, string, Config, string, string, string, bool) (core.LeaseClaim, error) { + return core.LeaseClaim{}, errors.New("claim write unavailable") + } + err := b.rollbackAcquireAfterClaimRemoval("inst_created", "cbx_no_durable_claim", "no-durable-claim", b.configForRun(), "", "rollback-cleanup", errors.New("ready transition lost claim")) + if err == nil || !strings.Contains(err.Error(), "claim write unavailable") || !strings.Contains(err.Error(), "reconcile instance inst_created manually") { + t.Fatalf("rollback err=%v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("provider mutation occurred without durable ownership: %#v", api.deletedIDs) + } + if _, ok := api.instances["inst_created"]; !ok { + t.Fatal("instance removed without durable cleanup ownership") + } +} + +func TestFalRollbackRefusesEmergencyDeleteAfterClaimConflict(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_created": readyFalInstance("inst_created", "203.0.113.42"), + }} + b := newFalTestBackend(t, api) + b.persistRollbackClaim = func(leaseID, slug string, cfg Config, repoRoot, _, reason string, keep bool) (core.LeaseClaim, error) { + if _, err := b.persistRecoveryClaimAtIfUnchanged( + leaseID, slug, cfg, repoRoot, "inst_other", reason, keep, time.Time{}, core.LeaseClaim{}, false, + ); err != nil { + return core.LeaseClaim{}, err + } + return core.LeaseClaim{}, errors.New("claim changed") + } + err := b.rollbackAcquireAfterClaimRemoval("inst_created", "cbx_claim_conflict", "claim-conflict", b.configForRun(), "", "rollback-cleanup", errors.New("ready transition lost claim")) + if err == nil || !strings.Contains(err.Error(), "concurrent recovery claim") { + t.Fatalf("rollback err=%v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("emergency delete crossed claim conflict: %#v", api.deletedIDs) + } + claim, exists, readErr := core.ReadLeaseClaimWithPresence("cbx_claim_conflict") + if readErr != nil || !exists || claim.CloudID != "inst_other" { + t.Fatalf("claim=%#v exists=%t err=%v", claim, exists, readErr) + } +} + +func TestFalRollbackReallocatesSlugBeforeRepublishingClaim(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_created": readyFalInstance("inst_created", "203.0.113.42")}, + deleteErr: errors.New("delete unavailable"), + } + b := newFalTestBackend(t, api) + if _, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_slug_new_owner", "reused-slug", b.configForRun(), "", "inst_other", "ready", false, b.now(), core.LeaseClaim{}, false, + ); err != nil { + t.Fatal(err) + } + err := b.rollbackAcquireAfterClaimRemoval("inst_created", "cbx_slug_old_owner", "reused-slug", b.configForRun(), "", "rollback-cleanup", errors.New("ready transition lost claim")) + if err == nil || !strings.Contains(err.Error(), "delete unavailable") { + t.Fatalf("rollback err=%v", err) + } + claim, exists, readErr := core.ReadLeaseClaimWithPresence("cbx_slug_old_owner") + if readErr != nil || !exists || claim.Slug == "reused-slug" || !strings.HasPrefix(claim.Slug, "reused-slug-") { + t.Fatalf("claim=%#v exists=%t err=%v", claim, exists, readErr) + } +} + +func TestFalRollbackDeletesLiveInstanceAfterClaimOnlyDisappearance(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_created": readyFalInstance("inst_created", "203.0.113.42"), + }} + b := newFalTestBackend(t, api) + claim, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_cleanup5678", + "claim-only-loss", + b.configForRun(), + "", + "inst_created", + "provisioning", + false, + b.now(), + core.LeaseClaim{}, + false, + ) + if err != nil { + t.Fatal(err) + } + if err := core.RemoveLeaseClaimIfUnchanged(claim.LeaseID, claim); err != nil { + t.Fatal(err) + } + + cause := errors.New("ready transition lost claim") + if err := b.rollbackClaimedAcquire("inst_created", claim.LeaseID, claim.Slug, b.configForRun(), "", "rollback-cleanup", cause); !errors.Is(err, cause) { + t.Fatalf("rollback err=%v", err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_created" { + t.Fatalf("orphaned live instance after claim loss: %#v", api.deletedIDs) + } + if _, exists := api.instances["inst_created"]; exists { + t.Fatal("instance remained live after claim-only disappearance") + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || exists { + t.Fatalf("claim exists=%v err=%v", exists, err) + } +} + +func TestFalRollbackReclaimsOwnershipWhenClaimLossCleanupFails(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{ + "inst_created": readyFalInstance("inst_created", "203.0.113.42"), + }, + deleteErr: errors.New("delete unavailable"), + } + b := newFalTestBackend(t, api) + claim, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_cleanup9012", + "claim-loss-recovery", + b.configForRun(), + "", + "inst_created", + "provisioning", + false, + b.now(), + core.LeaseClaim{}, + false, + ) + if err != nil { + t.Fatal(err) + } + if err := core.RemoveLeaseClaimIfUnchanged(claim.LeaseID, claim); err != nil { + t.Fatal(err) + } + + err = b.rollbackClaimedAcquire("inst_created", claim.LeaseID, claim.Slug, b.configForRun(), "", "rollback-cleanup", errors.New("ready transition lost claim")) + if err == nil || !strings.Contains(err.Error(), "delete unavailable") { + t.Fatalf("rollback err=%v", err) + } + recovered, exists, readErr := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if readErr != nil || !exists || recovered.CloudID != "inst_created" || recovered.Labels["recovery"] != "rollback-cleanup" { + t.Fatalf("recovered claim=%#v exists=%v err=%v", recovered, exists, readErr) + } +} + +func TestFalRollbackRetainsOwnershipWhenDeleteNotFoundDoesNotConfirmAbsence(t *testing.T) { + notFound := &APIError{StatusCode: 404, Status: "404 Not Found", Message: "not found"} + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{ + "inst_created": readyFalInstance("inst_created", "203.0.113.42"), + }, + deleteErr: notFound, + } + b := newFalTestBackend(t, api) + b.pollInterval = time.Millisecond + b.pollTimeout = 20 * time.Millisecond + claim, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_cleanup7890", + "live-delete-not-found", + b.configForRun(), + "", + "inst_created", + "provisioning", + false, + b.now(), + core.LeaseClaim{}, + false, + ) + if err != nil { + t.Fatal(err) + } + if err := core.RemoveLeaseClaimIfUnchanged(claim.LeaseID, claim); err != nil { + t.Fatal(err) + } + + err = b.rollbackClaimedAcquire("inst_created", claim.LeaseID, claim.Slug, b.configForRun(), "", "rollback-cleanup", errors.New("ready transition lost claim")) + if err == nil || !strings.Contains(err.Error(), "confirm fal instance inst_created deletion") || !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("rollback err=%v", err) + } + recovered, exists, readErr := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if readErr != nil || !exists || recovered.CloudID != "inst_created" { + t.Fatalf("recovered claim=%#v exists=%v err=%v", recovered, exists, readErr) + } +} + +func TestFalRollbackRetainsClaimWhenNotFoundConflictsWithAccountInventory(t *testing.T) { + notFound := &APIError{StatusCode: 404, Status: "404 Not Found", Message: "not found"} + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{ + "inst_created": readyFalInstance("inst_created", "203.0.113.42"), + }, + getErr: notFound, + deleteErr: notFound, + } + b := newFalTestBackend(t, api) + claim, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_cleanup3456", + "masked-absence", + b.configForRun(), + "", + "inst_created", + "provisioning", + false, + b.now(), + core.LeaseClaim{}, + false, + ) + if err != nil { + t.Fatal(err) + } + if err := core.RemoveLeaseClaimIfUnchanged(claim.LeaseID, claim); err != nil { + t.Fatal(err) + } + + err = b.rollbackClaimedAcquire("inst_created", claim.LeaseID, claim.Slug, b.configForRun(), "", "rollback-cleanup", errors.New("ready transition lost claim")) + if err == nil || !strings.Contains(err.Error(), errFalProviderAbsenceNotAccountBound.Error()) { + t.Fatalf("rollback err=%v", err) + } + recovered, exists, readErr := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if readErr != nil || !exists || recovered.CloudID != "inst_created" { + t.Fatalf("recovered claim=%#v exists=%v err=%v", recovered, exists, readErr) + } + if _, exists := api.instances["inst_created"]; !exists { + t.Fatal("masked-absence rollback deleted provider state without confirmation") + } +} + +func TestFalRollbackRetainsClaimWhenDeleteAbsenceIsUnverified(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_created": readyFalInstance("inst_created", "203.0.113.42")}, + deleteErr: &APIError{StatusCode: 404, Status: "404 Not Found", Message: "not found"}, + } + b := newFalTestBackend(t, api) + err := b.rollbackAcquire("inst_created", "cbx_abcdef123456", "rollback", b.configForRun(), "", "rollback-cleanup", errors.New("bootstrap failed")) + if err == nil || !strings.Contains(err.Error(), "fal cleanup failed") { + t.Fatalf("rollback err=%v", err) + } + claim, ok, err := core.ResolveLeaseClaimForProvider("rollback", providerName) + if err != nil || !ok || claim.CloudID != "inst_created" || claim.Labels["recovery"] != "rollback-cleanup" { + t.Fatalf("claim=%#v ok=%v err=%v", claim, ok, err) + } +} + +func TestFalResolveListAndReleaseRequireLocalClaim(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_owned": readyFalInstance("inst_owned", "203.0.113.10"), + "inst_foreign": readyFalInstance("inst_foreign", "203.0.113.11"), + }} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_abcdef123456", "owned", "inst_owned", "203.0.113.10", false) + claim, exists, err := core.ReadLeaseClaimWithPresence("cbx_abcdef123456") + if err != nil || !exists { + t.Fatalf("ready claim exists=%t err=%v", exists, err) + } + if _, err := markFalAcquireLifetimeClaimForTest(claim); err != nil { + t.Fatal(err) + } + + lease, err := b.Resolve(context.Background(), core.ResolveRequest{ID: "owned", Repo: core.Repo{Root: t.TempDir()}}) + if err != nil { + t.Fatal(err) + } + if lease.LeaseID != "cbx_abcdef123456" || lease.Server.CloudID != "inst_owned" || lease.SSH.Host != "203.0.113.10" { + t.Fatalf("lease=%#v", lease) + } + claim, exists, err = core.ReadLeaseClaimWithPresence(lease.LeaseID) + if err != nil || !exists || claim.Labels[falAcquireLifetimeLabel] != falAcquireLifetimeVersion { + t.Fatalf("resolved claim lifetime marker=%q exists=%t err=%v", claim.Labels[falAcquireLifetimeLabel], exists, err) + } + if lease.Server.Labels[falAcquireLifetimeLabel] != "" { + t.Fatalf("internal lifetime marker leaked after resolve: %#v", lease.Server.Labels) + } + lease, err = b.Resolve(context.Background(), core.ResolveRequest{ID: "inst_owned"}) + if err != nil || lease.LeaseID != "cbx_abcdef123456" { + t.Fatalf("resolve by cloud id lease=%#v err=%v", lease, err) + } + if _, err := b.Resolve(context.Background(), core.ResolveRequest{ID: "inst_foreign"}); err == nil || !strings.Contains(err.Error(), "not locally claimed") { + t.Fatalf("foreign resolve err=%v", err) + } + views, err := b.List(context.Background(), core.ListRequest{}) + if err != nil { + t.Fatal(err) + } + if len(views) != 1 || views[0].CloudID != "inst_owned" { + t.Fatalf("views=%#v", views) + } + if err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: lease}); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_owned" { + t.Fatalf("deletedIDs=%#v", api.deletedIDs) + } + if _, ok, err := core.ResolveLeaseClaimForProvider("owned", providerName); err != nil || ok { + t.Fatalf("claim should be removed ok=%v err=%v", ok, err) + } +} + +func TestFalLifecycleRejectsClaimsFromAnotherAPIEndpoint(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_owned": readyFalInstance("inst_owned", "203.0.113.10"), + }} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_abcdef123456", "owned", "inst_owned", "203.0.113.10", true) + + other := *b + other.cfg.Fal.APIURL = "https://other.example.test/v1" + if _, err := other.Resolve(context.Background(), core.ResolveRequest{ID: "owned"}); err == nil || !strings.Contains(err.Error(), "not locally claimed") { + t.Fatalf("cross-endpoint resolve err=%v", err) + } + views, err := other.List(context.Background(), core.ListRequest{}) + if err != nil || len(views) != 0 { + t.Fatalf("cross-endpoint views=%#v err=%v", views, err) + } + err = other.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_abcdef123456", + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": "cbx_abcdef123456"}}, + }}) + if err == nil || !strings.Contains(err.Error(), "different API endpoint") { + t.Fatalf("cross-endpoint release err=%v", err) + } + if err := other.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("cross-endpoint lifecycle deleted instances: %#v", api.deletedIDs) + } +} + +func TestFalResolveUsesPersistedSSHUserAndPort(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_owned": readyFalInstance("inst_owned", "203.0.113.10"), + }} + b := newFalTestBackend(t, api) + claimCfg := b.cfg + claimCfg.Fal.User = "ubuntu" + claimCfg.SSHUser = "ubuntu" + claimCfg.SSHPort = "2222" + claimFalLease(t, claimCfg, "cbx_abcdef123456", "owned", "inst_owned", "203.0.113.10", false) + + b.cfg.Fal.User = defaultUser + b.cfg.SSHUser = defaultUser + b.cfg.SSHPort = "22" + lease, err := b.Resolve(context.Background(), core.ResolveRequest{ID: "owned"}) + if err != nil { + t.Fatal(err) + } + if lease.SSH.User != "ubuntu" || lease.SSH.Port != "2222" { + t.Fatalf("ssh target=%#v, want persisted ubuntu:2222", lease.SSH) + } +} + +func TestFalStatusOnlyResolveDoesNotRequireSSHHost(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_pending": { + ID: "inst_pending", + InstanceType: InstanceTypeH100x1, + Sector: Sector1, + Region: "us-west", + Status: InstanceStatusProvisioning, + }, + }} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_pending123", "pending", "inst_pending", "", false) + lease, err := b.Resolve(context.Background(), core.ResolveRequest{ID: "pending", StatusOnly: true}) + if err != nil { + t.Fatal(err) + } + if lease.Server.Status != string(InstanceStatusProvisioning) || lease.SSH.Host != "" { + t.Fatalf("lease=%#v", lease) + } + if lease.Server.Labels["state"] != string(InstanceStatusProvisioning) { + t.Fatalf("state label=%q, want live provisioning state", lease.Server.Labels["state"]) + } + if _, err := b.Resolve(context.Background(), core.ResolveRequest{ID: "pending"}); err == nil || !strings.Contains(err.Error(), "no SSH host") { + t.Fatalf("non-status resolve err=%v", err) + } +} + +func TestFalStatusReadyProbeIncludesSSHWhenHostIsAvailable(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_ready": readyFalInstance("inst_ready", "203.0.113.50"), + }} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_ready12345", "ready", "inst_ready", "203.0.113.50", false) + lease, err := b.Resolve(context.Background(), core.ResolveRequest{ID: "ready", StatusOnly: true, ReadyProbe: true}) + if err != nil { + t.Fatal(err) + } + if lease.SSH.Host != "203.0.113.50" { + t.Fatalf("ready probe ssh target=%#v", lease.SSH) + } +} + +func TestFalReleaseRetainsClaimOnAmbiguousProviderRead(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_owned": readyFalInstance("inst_owned", "203.0.113.10")}, + getErr: errors.New("temporary inventory failure"), + } + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_abcdef123456", "owned", "inst_owned", "203.0.113.10", false) + + err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_abcdef123456", + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": "cbx_abcdef123456"}}, + }}) + if err == nil || !strings.Contains(err.Error(), "temporary inventory failure") { + t.Fatalf("err=%v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("delete issued despite ambiguous read: %#v", api.deletedIDs) + } + if claim, ok, err := core.ResolveLeaseClaimForProvider("owned", providerName); err != nil || !ok || claim.CloudID != "inst_owned" { + t.Fatalf("claim not retained: claim=%#v ok=%v err=%v", claim, ok, err) + } +} + +func TestFalReleaseRetainsClaimUntilDeletionIsConfirmed(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_owned": readyFalInstance("inst_owned", "203.0.113.10")}, + retainDeletedInstance: true, + } + b := newFalTestBackend(t, api) + b.pollInterval = time.Millisecond + b.pollTimeout = 20 * time.Millisecond + claimFalLease(t, b.cfg, "cbx_delete_pending", "delete-pending", "inst_owned", "203.0.113.10", false) + original, originalExists, originalErr := core.ReadLeaseClaimWithPresence("cbx_delete_pending") + if originalErr != nil || !originalExists { + t.Fatalf("original claim exists=%t err=%v", originalExists, originalErr) + } + + release := core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_delete_pending", + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": "cbx_delete_pending"}}, + }} + err := b.ReleaseLease(context.Background(), release) + if err == nil || !strings.Contains(err.Error(), "confirm fal instance inst_owned deletion") || !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("err=%v", err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_owned" { + t.Fatalf("deletedIDs=%#v", api.deletedIDs) + } + claim, ok, readErr := core.ReadLeaseClaimWithPresence("cbx_delete_pending") + if readErr != nil || !ok || claim.CloudID != "inst_owned" || !falDeleteStateMatches(claim, falDeleteAcceptedLabel, "inst_owned") || falDeleteConfirmed(claim, "inst_owned") { + t.Fatalf("claim=%#v exists=%t err=%v", claim, ok, readErr) + } + stale := original + stale.Labels = cloneLabels(original.Labels) + stale.Labels["state"] = "ready" + if err := core.ReplaceLeaseClaimIfUnchangedDurable(original.LeaseID, original, stale); err == nil { + t.Fatal("stale pre-delete writer replaced the accepted deletion marker") + } + if _, err := b.Touch(context.Background(), core.TouchRequest{Lease: core.LeaseTarget{LeaseID: claim.LeaseID}}); err == nil || !strings.Contains(err.Error(), "deletion is in progress") { + t.Fatalf("touch err=%v", err) + } + if _, err := b.Resolve(context.Background(), core.ResolveRequest{ID: claim.LeaseID}); err == nil || !strings.Contains(err.Error(), "deletion is in progress") { + t.Fatalf("resolve err=%v", err) + } + if _, err := b.Resolve(context.Background(), core.ResolveRequest{ID: claim.LeaseID, ReleaseOnly: true}); err != nil { + t.Fatalf("release-only resolve err=%v", err) + } + if err := b.ReleaseLease(context.Background(), release); err == nil || !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("accepted pending release err=%v", err) + } + if len(api.deletedIDs) != 1 { + t.Fatalf("accepted deletion was reissued while still visible: %#v", api.deletedIDs) + } + delete(api.instances, "inst_owned") + if err := b.ReleaseLease(context.Background(), release); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 1 { + t.Fatalf("accepted deletion was reissued: %#v", api.deletedIDs) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } +} + +func TestFalReleaseFinalizesConfirmedDeletionAfterCredentialRotation(t *testing.T) { + api := &fakeFalAPI{getErr: errors.New("provider access must not be required")} + b := newFalTestBackend(t, api) + const leaseID = "cbx_delete_confirmed" + claimFalLease(t, b.cfg, leaseID, "delete-confirmed", "inst_owned", "203.0.113.10", false) + keyPath, _, err := core.EnsureTestboxKeyForConfig(b.cfg, leaseID) + if err != nil { + t.Fatal(err) + } + claim, exists, err := core.ReadLeaseClaimWithPresence(leaseID) + if err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + confirmed, err := persistFalDeleteState(claim, falDeleteConfirmedLabel, "inst_owned") + if err != nil { + t.Fatal(err) + } + unbound := confirmed + unbound.CloudID = "" + if err := core.ReplaceLeaseClaimIfUnchangedDurable(leaseID, confirmed, unbound); err != nil { + t.Fatal(err) + } + b.cfg.Fal.APIKey = "rotated-test-key" + + if err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: leaseID, + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": leaseID}}, + }}); err != nil { + t.Fatal(err) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(leaseID); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + if _, err := os.Stat(keyPath); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("confirmed deletion key residue: %v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("confirmed deletion contacted delete API: %#v", api.deletedIDs) + } +} + +func TestFalReleaseRetainsUnconfirmedDeletionAfterCredentialRotation(t *testing.T) { + for _, state := range []struct { + name string + label string + }{ + {name: "attempted", label: falDeleteAttemptLabel}, + {name: "accepted", label: falDeleteAcceptedLabel}, + } { + t.Run(state.name, func(t *testing.T) { + for _, visible := range []bool{true, false} { + name := "absent" + instances := map[string]ComputeInstance{} + if visible { + name = "visible" + instances["inst_owned"] = readyFalInstance("inst_owned", "203.0.113.10") + } + t.Run(name, func(t *testing.T) { + api := &fakeFalAPI{instances: instances} + b := newFalTestBackend(t, api) + const leaseID = "cbx_delete_accepted" + claimFalLease(t, b.cfg, leaseID, "delete-accepted", "inst_owned", "203.0.113.10", false) + keyPath, _, err := core.EnsureTestboxKeyForConfig(b.cfg, leaseID) + if err != nil { + t.Fatal(err) + } + claim, exists, err := core.ReadLeaseClaimWithPresence(leaseID) + if err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + accepted, err := persistFalDeleteState(claim, state.label, "inst_owned") + if err != nil { + t.Fatal(err) + } + originalBinding := accepted.Labels[falCredentialBindingLabel] + b.cfg.Fal.APIKey = "rotated-test-key" + + releaseErr := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: leaseID, + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": leaseID}}, + }}) + if releaseErr == nil || !strings.Contains(releaseErr.Error(), "credential rotation") { + t.Fatalf("release err=%v", releaseErr) + } + retained, exists, err := core.ReadLeaseClaimWithPresence(leaseID) + if err != nil || !exists || !falDeleteStateMatches(retained, state.label, "inst_owned") || falDeleteConfirmed(retained, "inst_owned") { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + if got := retained.Labels[falCredentialBindingLabel]; got != originalBinding { + t.Fatalf("accepted deletion binding changed from %q to %q", originalBinding, got) + } + if _, err := os.Stat(keyPath); err != nil { + t.Fatalf("accepted deletion key missing: %v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("accepted absent deletion was reissued: %#v", api.deletedIDs) + } + }) + } + }) + } +} + +func TestFalReleaseSerializesReadyClaimWithAcquireLifetime(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_owned": readyFalInstance("inst_owned", "203.0.113.10"), + }} + b := newFalTestBackend(t, api) + const leaseID = "cbx_ready_acquiring" + claimFalLease(t, b.cfg, leaseID, "ready-acquiring", "inst_owned", "203.0.113.10", false) + claim, exists, err := core.ReadLeaseClaimWithPresence(leaseID) + if err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + marked := claim + marked.Labels = cloneLabels(claim.Labels) + marked.Labels[falAcquireLifetimeLabel] = falAcquireLifetimeVersion + if err := core.ReplaceLeaseClaimIfUnchangedDurable(leaseID, claim, marked); err != nil { + t.Fatal(err) + } + unlock, err := lockFalAcquireLifetime(context.Background(), leaseID) + if err != nil { + t.Fatal(err) + } + defer unlock() + + if _, err := b.Resolve(context.Background(), core.ResolveRequest{ID: leaseID, ReleaseOnly: true}); err == nil || !strings.Contains(err.Error(), "acquisition is still in progress") { + t.Fatalf("resolve err=%v", err) + } + if err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: leaseID, + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": leaseID}}, + }}); err == nil || !strings.Contains(err.Error(), "acquisition is still in progress") { + t.Fatalf("release err=%v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("active acquire instance deleted: %#v", api.deletedIDs) + } + if err := core.VerifyLeaseClaimUnchanged(leaseID, marked); err != nil { + t.Fatalf("active acquire claim changed: %v", err) + } +} + +func TestFalCleanupResumesAcceptedDeletionImmediately(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + claimFalLease(t, b.cfg, "cbx_cleanup_accepted", "cleanup-accepted", "inst_owned", "203.0.113.10", false) + claim, exists, err := core.ReadLeaseClaimWithPresence("cbx_cleanup_accepted") + if err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + claim, err = persistFalDeleteState(claim, falDeleteAcceptedLabel, "inst_owned") + if err != nil { + t.Fatal(err) + } + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } +} + +func TestFalReleaseRetainsAcceptedClaimWhenKeyRemovalFails(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_owned": readyFalInstance("inst_owned", "203.0.113.10"), + }} + b := newFalTestBackend(t, api) + b.removeLeaseKey = func(string) error { return errors.New("key removal unavailable") } + claimFalLease(t, b.cfg, "cbx_delete_key_fail", "delete-key-fail", "inst_owned", "203.0.113.10", false) + release := core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_delete_key_fail", + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": "cbx_delete_key_fail"}}, + }} + if err := b.ReleaseLease(context.Background(), release); err == nil || !strings.Contains(err.Error(), "key removal unavailable") { + t.Fatalf("release err=%v", err) + } + claim, exists, err := core.ReadLeaseClaimWithPresence("cbx_delete_key_fail") + if err != nil || !exists || !falDeleteStateMatches(claim, falDeleteAcceptedLabel, "inst_owned") { + t.Fatalf("claim=%#v exists=%t err=%v", claim, exists, err) + } + b.removeLeaseKey = nil + if err := b.ReleaseLease(context.Background(), release); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 1 { + t.Fatalf("accepted deletion was reissued: %#v", api.deletedIDs) + } + if _, exists, err := core.ReadLeaseClaimWithPresence("cbx_delete_key_fail"); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } +} + +func TestFalReleaseAcceptsDeleteNotFoundAfterIdentityProof(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_owned": readyFalInstance("inst_owned", "203.0.113.10")}, + deleteErr: &APIError{StatusCode: 404, Status: "404 Not Found", Message: "not found"}, + removeBeforeDeleteError: true, + } + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_delete_race", "delete-race", "inst_owned", "203.0.113.10", false) + err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_delete_race", + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": "cbx_delete_race"}}, + }}) + if err != nil { + t.Fatal(err) + } + if _, exists, readErr := core.ReadLeaseClaimWithPresence("cbx_delete_race"); readErr != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, readErr) + } +} + +func TestFalReleaseRetriesDeleteNotFoundWhenInstanceRemainsLive(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_owned": readyFalInstance("inst_owned", "203.0.113.10")}, + deleteErr: &APIError{StatusCode: 404, Status: "404 Not Found", Message: "not found"}, + } + b := newFalTestBackend(t, api) + b.pollInterval = time.Millisecond + b.pollTimeout = 20 * time.Millisecond + claimFalLease(t, b.cfg, "cbx_delete_404_live", "delete-404-live", "inst_owned", "203.0.113.10", false) + release := core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_delete_404_live", + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": "cbx_delete_404_live"}}, + }} + if err := b.ReleaseLease(context.Background(), release); err == nil || !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("first release err=%v", err) + } + claim, exists, err := core.ReadLeaseClaimWithPresence("cbx_delete_404_live") + if err != nil || !exists || !falDeleteStateMatches(claim, falDeleteAttemptLabel, "inst_owned") || falDeleteAccepted(claim, "inst_owned") { + t.Fatalf("claim=%#v exists=%t err=%v", claim, exists, err) + } + api.deleteErr = nil + if err := b.ReleaseLease(context.Background(), release); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 2 { + t.Fatalf("delete requests=%#v", api.deletedIDs) + } + if _, exists, err := core.ReadLeaseClaimWithPresence("cbx_delete_404_live"); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } +} + +func TestFalReleaseDoesNotAcceptDeleteNotFoundWhenInventoryStillContainsInstance(t *testing.T) { + notFound := &APIError{StatusCode: 404, Status: "404 Not Found", Message: "not found"} + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_owned": readyFalInstance("inst_owned", "203.0.113.10")}, + deleteErr: notFound, + } + api.afterDeleteHook = func(string) { api.getErr = notFound } + b := newFalTestBackend(t, api) + b.pollInterval = time.Millisecond + b.pollTimeout = 20 * time.Millisecond + claimFalLease(t, b.cfg, "cbx_delete_404_masked", "delete-404-masked", "inst_owned", "203.0.113.10", false) + err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_delete_404_masked", + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": "cbx_delete_404_masked"}}, + }}) + if err == nil || !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("release err=%v", err) + } + claim, exists, readErr := core.ReadLeaseClaimWithPresence("cbx_delete_404_masked") + if readErr != nil || !exists || !falDeleteStateMatches(claim, falDeleteAttemptLabel, "inst_owned") || falDeleteAccepted(claim, "inst_owned") { + t.Fatalf("claim=%#v exists=%t err=%v", claim, exists, readErr) + } +} + +func TestFalReleaseDoesNotFinalizeDeleteAttemptWhenInventoryConflicts(t *testing.T) { + notFound := &APIError{StatusCode: 404, Status: "404 Not Found", Message: "not found"} + b := newFalTestBackend(t, &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_owned": readyFalInstance("inst_owned", "203.0.113.10")}, + getErr: notFound, + }) + claimFalLease(t, b.cfg, "cbx_delete_attempt", "delete-attempt", "inst_owned", "203.0.113.10", false) + claim, exists, err := core.ReadLeaseClaimWithPresence("cbx_delete_attempt") + if err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + claim, err = persistFalDeleteState(claim, falDeleteAttemptLabel, "inst_owned") + if err != nil { + t.Fatal(err) + } + err = b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: claim.LeaseID, + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": claim.LeaseID}}, + }}) + if err == nil || !strings.Contains(err.Error(), errFalProviderAbsenceNotAccountBound.Error()) { + t.Fatalf("release err=%v", err) + } + current, exists, readErr := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if readErr != nil || !exists || !falDeleteStateMatches(current, falDeleteAttemptLabel, "inst_owned") || falDeleteStateMatches(current, falDeleteAcceptedLabel, "inst_owned") { + t.Fatalf("claim=%#v exists=%t err=%v", current, exists, readErr) + } +} + +func TestFalReleaseReconcilesAmbiguousDeleteWithCompleteAbsenceProof(t *testing.T) { + api := &fakeFalAPI{ + instances: map[string]ComputeInstance{"inst_owned": readyFalInstance("inst_owned", "203.0.113.10")}, + deleteErr: context.Canceled, + removeBeforeDeleteError: true, + } + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_delete_ambiguous", "delete-ambiguous", "inst_owned", "203.0.113.10", false) + release := core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_delete_ambiguous", + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": "cbx_delete_ambiguous"}}, + }} + if err := b.ReleaseLease(context.Background(), release); !errors.Is(err, context.Canceled) { + t.Fatalf("first release err=%v", err) + } + claim, exists, err := core.ReadLeaseClaimWithPresence("cbx_delete_ambiguous") + if err != nil || !exists || !falDeleteStateMatches(claim, falDeleteAttemptLabel, "inst_owned") || falDeleteAccepted(claim, "inst_owned") { + t.Fatalf("claim=%#v exists=%t err=%v", claim, exists, err) + } + if err := b.ReleaseLease(context.Background(), release); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 1 { + t.Fatalf("ambiguous deletion was reissued: %#v", api.deletedIDs) + } + if _, exists, err := core.ReadLeaseClaimWithPresence("cbx_delete_ambiguous"); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } +} + +func TestFalReleaseRejectsClaimChangedBeforeDeletion(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_owned": readyFalInstance("inst_owned", "203.0.113.10"), + }} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_abcdef123456", "owned", "inst_owned", "203.0.113.10", false) + b.clientFactory = func(Config, Runtime) (computeAPI, error) { + claim, ok, err := core.ReadLeaseClaimWithPresence("cbx_abcdef123456") + if err != nil || !ok { + return nil, fmt.Errorf("read claim before release: ok=%v err=%w", ok, err) + } + labels := cloneLabels(claim.Labels) + labels["state"] = "renewed" + if _, err := core.UpdateLeaseClaimLabelsIfUnchanged(claim.LeaseID, claim, labels); err != nil { + return nil, err + } + return api, nil + } + err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_abcdef123456", + Server: core.Server{CloudID: "inst_owned", Provider: providerName, Labels: map[string]string{"lease": "cbx_abcdef123456"}}, + }}) + if err == nil { + t.Fatal("expected changed-claim release rejection") + } + if len(api.deletedIDs) != 0 { + t.Fatalf("changed claim was deleted: %#v", api.deletedIDs) + } + if _, ok, err := core.ReadLeaseClaimWithPresence("cbx_abcdef123456"); err != nil || !ok { + t.Fatalf("changed claim retained=%v err=%v", ok, err) + } +} + +func TestFalReleaseRefusesRecoveryClaimWithoutCloudID(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_foreign": readyFalInstance("inst_foreign", "203.0.113.40"), + }} + b := newFalTestBackend(t, api) + if err := b.persistRecoveryClaim("cbx_recovery123", "recovery", b.configForRun(), "", "", "ambiguous-create", true); err != nil { + t.Fatal(err) + } + err := b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_recovery123", + Server: core.Server{CloudID: "inst_foreign", Provider: providerName, Labels: map[string]string{"lease": "cbx_recovery123"}}, + }}) + if err == nil || !strings.Contains(err.Error(), "recovery is still pending") { + t.Fatalf("err=%v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("deleted unclaimed instance through recovery claim: %#v", api.deletedIDs) + } +} + +func TestFalCleanupDeletesOnlyExpiredClaimedInstances(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_expired": readyFalInstance("inst_expired", "203.0.113.20"), + "inst_foreign": readyFalInstance("inst_foreign", "203.0.113.21"), + }} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_expired1234", "expired", "inst_expired", "203.0.113.20", true) + legacy, exists, err := core.ReadLeaseClaimWithPresence("cbx_expired1234") + if err != nil || !exists { + t.Fatalf("legacy ready claim exists=%t err=%v", exists, err) + } + marked := legacy + legacy.Labels = cloneLabels(legacy.Labels) + delete(legacy.Labels, falAcquireLifetimeLabel) + if err := core.ReplaceLeaseClaimIfUnchangedDurable(legacy.LeaseID, marked, legacy); err != nil { + t.Fatal(err) + } + + if err := b.Cleanup(context.Background(), core.CleanupRequest{DryRun: true}); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("dry-run deleted: %#v", api.deletedIDs) + } + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_expired" { + t.Fatalf("deletedIDs=%#v", api.deletedIDs) + } + if _, ok := api.instances["inst_foreign"]; !ok { + t.Fatal("cleanup deleted unclaimed foreign instance") + } +} + +func TestFalCleanupRetainsUnmarkedLegacyRecoveryClaims(t *testing.T) { + for _, tc := range []struct { + recovery string + instanceID string + }{ + {recovery: "create-intent"}, + {recovery: "ambiguous-create"}, + {recovery: "ambiguous-create-inflight"}, + {recovery: "provisioning", instanceID: "inst_legacy_provisioning"}, + } { + for _, marker := range []string{"", " 1 ", "2"} { + name := tc.recovery + "/marker-" + strings.ReplaceAll(marker, " ", "_") + t.Run(name, func(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{}} + if tc.instanceID != "" { + api.instances[tc.instanceID] = readyFalInstance(tc.instanceID, "203.0.113.42") + } + b := newFalTestBackend(t, api) + var stderr strings.Builder + b.rt.Stderr = &stderr + leaseID := "cbx_legacy_" + strings.ReplaceAll(tc.recovery, "-", "_") + var claim core.LeaseClaim + if tc.instanceID == "" { + claim, _ = persistFalCreateRecoveryClaim(t, b, leaseID, "legacy-recovery", tc.recovery, false, b.now()) + } else { + var err error + claim, err = b.persistRecoveryClaimAtIfUnchanged( + leaseID, "legacy-recovery", b.configForRun(), "", tc.instanceID, tc.recovery, false, b.now(), core.LeaseClaim{}, false, + ) + if err != nil { + t.Fatal(err) + } + } + marked := claim + claim.Labels = cloneLabels(claim.Labels) + if marker == "" { + delete(claim.Labels, falAcquireLifetimeLabel) + } else { + claim.Labels[falAcquireLifetimeLabel] = marker + } + if err := core.ReplaceLeaseClaimIfUnchangedDurable(claim.LeaseID, marked, claim); err != nil { + t.Fatal(err) + } + clientCalls := 0 + b.clientFactory = func(Config, Runtime) (computeAPI, error) { + clientCalls++ + return api, nil + } + + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if clientCalls != 0 || len(api.createRequests) != 0 || len(api.deletedIDs) != 0 { + t.Fatalf("legacy cleanup mutated provider clientCalls=%d create=%d delete=%#v", clientCalls, len(api.createRequests), api.deletedIDs) + } + if !strings.Contains(stderr.String(), "reason=legacy_recovery_pending") { + t.Fatalf("stderr=%q", stderr.String()) + } + current, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if err != nil || !exists || current.Labels[falAcquireLifetimeLabel] != marker { + t.Fatalf("legacy claim exists=%t err=%v labels=%#v", exists, err, current.Labels) + } + }) + } + } +} + +func TestFalLegacyAmbiguousRecoveryPreservesUntrustedLifetimeMarker(t *testing.T) { + for _, marker := range []string{"", " 1 ", "2"} { + t.Run("marker-"+strings.ReplaceAll(marker, " ", "_"), func(t *testing.T) { + api := &fakeFalAPI{createErr: io.ErrUnexpectedEOF} + b := newFalTestBackend(t, api) + claim, _ := persistFalCreateRecoveryClaim(t, b, "cbx_legacy_replay", "legacy-replay", "ambiguous-create", false, b.now()) + marked := claim + claim.Labels = cloneLabels(claim.Labels) + if marker == "" { + delete(claim.Labels, falAcquireLifetimeLabel) + } else { + claim.Labels[falAcquireLifetimeLabel] = marker + } + if err := core.ReplaceLeaseClaimIfUnchangedDurable(claim.LeaseID, marked, claim); err != nil { + t.Fatal(err) + } + + if _, err := b.recoverAmbiguousCreateForRelease(context.Background(), api, claim, b.configForRun()); err == nil { + t.Fatal("expected ambiguous recovery failure") + } + current, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if err != nil || !exists { + t.Fatalf("legacy claim exists=%t err=%v", exists, err) + } + value, present := current.Labels[falAcquireLifetimeLabel] + if value != marker || present != (marker != "") { + t.Fatalf("legacy lifetime marker value=%q present=%t want value=%q present=%t", value, present, marker, marker != "") + } + }) + } +} + +func TestFalCleanupRemovesAbandonedCreateIntentWithoutProviderMutation(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + const leaseID = "cbx_cleanup_intent" + claim, _ := persistFalCreateRecoveryClaim(t, b, leaseID, "cleanup-intent", "create-intent", false, b.now()) + keyPath, err := core.TestboxKeyPath(leaseID) + if err != nil { + t.Fatal(err) + } + clientCalls := 0 + b.clientFactory = func(Config, Runtime) (computeAPI, error) { + clientCalls++ + return nil, errors.New("provider must not be called") + } + + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if clientCalls != 0 { + t.Fatalf("provider client calls=%d", clientCalls) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + if _, err := os.Stat(keyPath); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("key stat err=%v, want not exist", err) + } +} + +func TestFalCleanupRecoversAmbiguousCreateAndDeletesExactInstance(t *testing.T) { + for _, recovery := range []string{"ambiguous-create", "ambiguous-create-inflight"} { + t.Run(recovery, func(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + leaseID := "cbx_cleanup_" + strings.ReplaceAll(recovery, "-", "_") + claim, request := persistFalCreateRecoveryClaim(t, b, leaseID, "cleanup-ambiguous", recovery, false, b.now()) + + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if len(api.createRequests) != 1 || api.createRequests[0] != request { + t.Fatalf("create requests=%#v want %#v", api.createRequests, request) + } + if len(api.idempotency) != 1 || api.idempotency[0] != claim.LeaseID { + t.Fatalf("idempotency=%#v want %s", api.idempotency, claim.LeaseID) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_created" { + t.Fatalf("deleted ids=%#v", api.deletedIDs) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + }) + } +} + +func TestFalCleanupDryRunReportsAmbiguousRecoveryWithoutMutation(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + var stderr strings.Builder + b.rt.Stderr = &stderr + claim, _ := persistFalCreateRecoveryClaim(t, b, "cbx_cleanup_dryrun", "cleanup-dryrun", "ambiguous-create", false, b.now()) + + if err := b.Cleanup(context.Background(), core.CleanupRequest{DryRun: true}); err != nil { + t.Fatal(err) + } + if len(api.createRequests) != 0 || len(api.deletedIDs) != 0 || api.listCalls != 0 { + t.Fatalf("dry-run provider mutations create=%d delete=%d list=%d", len(api.createRequests), len(api.deletedIDs), api.listCalls) + } + if !strings.Contains(stderr.String(), "reason=recover_ambiguous_create") { + t.Fatalf("stderr=%q", stderr.String()) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } +} + +func TestFalCleanupSkipsKeepRecoveryClaims(t *testing.T) { + for _, tc := range []struct { + name string + recovery string + instance string + startedAt time.Time + }{ + {name: "intent", recovery: "create-intent"}, + {name: "ambiguous", recovery: "ambiguous-create"}, + {name: "known-id", recovery: "provisioning", instance: "inst_keep", startedAt: time.Now().Add(-24 * time.Hour)}, + } { + t.Run(tc.name, func(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + const leaseID = "cbx_cleanup_keep" + var claim core.LeaseClaim + if tc.instance == "" { + claim, _ = persistFalCreateRecoveryClaim(t, b, leaseID, "cleanup-keep", tc.recovery, true, tc.startedAt) + } else { + var err error + claim, err = b.persistRecoveryClaimAtIfUnchanged(leaseID, "cleanup-keep", b.configForRun(), "", tc.instance, tc.recovery, true, tc.startedAt, core.LeaseClaim{}, false) + if err != nil { + t.Fatal(err) + } + } + clientCalls := 0 + b.clientFactory = func(Config, Runtime) (computeAPI, error) { + clientCalls++ + return nil, errors.New("provider must not be called") + } + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if clientCalls != 0 { + t.Fatalf("provider client calls=%d", clientCalls) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + }) + } +} + +func TestFalCleanupSkipsLiveIdentitylessAcquireClaims(t *testing.T) { + for _, recovery := range []string{"create-intent", "ambiguous-create", "ambiguous-create-inflight"} { + t.Run(recovery, func(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + var stderr strings.Builder + b.rt.Stderr = &stderr + claim, _ := persistFalCreateRecoveryClaim(t, b, "cbx_cleanup_fresh_"+strings.ReplaceAll(recovery, "-", "_"), "cleanup-fresh", recovery, false, b.now()) + unlockAcquire, err := lockFalAcquireLifetime(context.Background(), claim.LeaseID) + if err != nil { + t.Fatal(err) + } + defer unlockAcquire() + clientCalls := 0 + b.clientFactory = func(Config, Runtime) (computeAPI, error) { + clientCalls++ + return nil, errors.New("provider must not be called") + } + + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if clientCalls != 0 || !strings.Contains(stderr.String(), "reason=acquire-in-progress") { + t.Fatalf("clientCalls=%d stderr=%q", clientCalls, stderr.String()) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + }) + } +} + +func TestFalCleanupOverridesKeepForForcedRecoveryStates(t *testing.T) { + t.Run("terminal rejection", func(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + claim, _ := persistFalCreateRecoveryClaim(t, b, "cbx_cleanup_rejected_keep", "cleanup-rejected-keep", falCreateRejectedRecovery, true, b.now()) + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + }) + + t.Run("rollback cleanup", func(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_rollback_keep": readyFalInstance("inst_rollback_keep", "203.0.113.43"), + }} + b := newFalTestBackend(t, api) + claim, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_cleanup_rollback_keep", "cleanup-rollback-keep", b.configForRun(), "", "inst_rollback_keep", "rollback-cleanup", true, b.now(), core.LeaseClaim{}, false, + ) + if err != nil { + t.Fatal(err) + } + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_rollback_keep" { + t.Fatalf("deleted ids=%#v", api.deletedIDs) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + }) +} + +func TestFalCleanupFinishesDeletionAlreadyStartedForKeptLease(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_keep_delete": readyFalInstance("inst_keep_delete", "203.0.113.42"), + }} + b := newFalTestBackend(t, api) + const leaseID = "cbx_cleanup_keep_delete" + claimFalLease(t, b.cfg, leaseID, "cleanup-keep-delete", "inst_keep_delete", "203.0.113.42", true) + claim, exists, err := core.ReadLeaseClaimWithPresence(leaseID) + if err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + if _, err := persistFalDeleteState(claim, falDeleteAttemptLabel, "inst_keep_delete"); err != nil { + t.Fatal(err) + } + unblockAcquire, err := lockFalAcquireLifetime(context.Background(), leaseID) + if err != nil { + t.Fatal(err) + } + defer unblockAcquire() + + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_keep_delete" { + t.Fatalf("deleted ids=%#v", api.deletedIDs) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(leaseID); err != nil || exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } +} + +func TestFalCleanupExpiredAmbiguousRecoveryRequiresManualReconciliation(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + claim, _ := persistFalCreateRecoveryClaim(t, b, "cbx_cleanup_expired", "cleanup-expired", "ambiguous-create", false, b.now().Add(-falCreateRecoveryWindow)) + + err := b.Cleanup(context.Background(), core.CleanupRequest{}) + if err == nil || !strings.Contains(err.Error(), "recovery window expired") || !strings.Contains(err.Error(), "manual provider reconciliation") { + t.Fatalf("cleanup err=%v", err) + } + if len(api.createRequests) != 0 || len(api.deletedIDs) != 0 { + t.Fatalf("expired recovery mutated provider create=%d delete=%d", len(api.createRequests), len(api.deletedIDs)) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID); err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } +} + +func TestFalCleanupExpiredRecoveryDoesNotStarveLaterForcedCleanup(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_rollback": readyFalInstance("inst_rollback", "203.0.113.44"), + }} + b := newFalTestBackend(t, api) + expired, _ := persistFalCreateRecoveryClaim(t, b, "cbx_a_expired", "expired", "ambiguous-create", false, b.now().Add(-falCreateRecoveryWindow)) + rollback, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_z_rollback", "rollback", b.configForRun(), "", "inst_rollback", "rollback-cleanup", false, b.now(), core.LeaseClaim{}, false, + ) + if err != nil { + t.Fatal(err) + } + + err = b.Cleanup(context.Background(), core.CleanupRequest{}) + if err == nil || !strings.Contains(err.Error(), expired.LeaseID) || !strings.Contains(err.Error(), "recovery window expired") { + t.Fatalf("cleanup err=%v", err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != rollback.CloudID { + t.Fatalf("deleted ids=%#v want %s", api.deletedIDs, rollback.CloudID) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(expired.LeaseID); err != nil || !exists { + t.Fatalf("expired claim exists=%t err=%v", exists, err) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(rollback.LeaseID); err != nil || exists { + t.Fatalf("rollback claim exists=%t err=%v", exists, err) + } +} + +func TestFalCleanupCanceledAfterKeepClaimReportsCancellation(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_rollback": readyFalInstance("inst_rollback", "203.0.113.45"), + }} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_a_keep", "keep", "inst_keep", "203.0.113.46", false) + keep, exists, err := core.ReadLeaseClaimWithPresence("cbx_a_keep") + if err != nil || !exists { + t.Fatalf("keep claim exists=%t err=%v", exists, err) + } + kept := keep + kept.Labels = cloneLabels(keep.Labels) + kept.Labels["keep"] = "true" + if err := core.ReplaceLeaseClaimIfUnchangedDurable(keep.LeaseID, keep, kept); err != nil { + t.Fatal(err) + } + rollback, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_z_rollback_cancel", "rollback-cancel", b.configForRun(), "", "inst_rollback", "rollback-cleanup", false, b.now(), core.LeaseClaim{}, false, + ) + if err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if err := b.Cleanup(ctx, core.CleanupRequest{}); !errors.Is(err, context.Canceled) { + t.Fatalf("cleanup err=%v want context canceled", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("canceled cleanup deleted=%#v", api.deletedIDs) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(rollback.LeaseID); err != nil || !exists { + t.Fatalf("rollback claim exists=%t err=%v", exists, err) + } +} + +func TestFalCleanupSkipsLiveProvisioningAcquireAndCleansAfterExit(t *testing.T) { + started := time.Date(2026, time.July, 9, 12, 0, 0, 0, time.UTC) + clock := &mutableFalClock{now: started.Add(3 * time.Hour)} + api := &fakeFalAPI{instances: map[string]ComputeInstance{"inst_provisioning": readyFalInstance("inst_provisioning", "203.0.113.42")}} + b := newFalTestBackend(t, api) + b.rt.Clock = clock + claim, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_cleanup_provisioning", "cleanup-provisioning", b.configForRun(), "", "inst_provisioning", "provisioning", false, started, core.LeaseClaim{}, false, + ) + if err != nil { + t.Fatal(err) + } + claim, err = markFalAcquireLifetimeClaimForTest(claim) + if err != nil { + t.Fatal(err) + } + unlockAcquire, err := lockFalAcquireLifetime(context.Background(), claim.LeaseID) + if err != nil { + t.Fatal(err) + } + clientCalls := 0 + b.clientFactory = func(Config, Runtime) (computeAPI, error) { + clientCalls++ + return api, nil + } + + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if clientCalls != 0 || len(api.deletedIDs) != 0 { + t.Fatalf("active acquire clientCalls=%d deleted=%#v", clientCalls, api.deletedIDs) + } + unlockAcquire() + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if clientCalls != 1 || len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_provisioning" { + t.Fatalf("stale acquire clientCalls=%d deleted=%#v", clientCalls, api.deletedIDs) + } +} + +func TestFalAcquireHoldsCleanupLivenessThroughOnAcquired(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + var cleanupErr error + lease, err := b.Acquire(context.Background(), core.AcquireRequest{ + RequestedSlug: "live-on-acquired", + OnAcquired: func(core.LeaseTarget) error { + cleanupErr = b.Cleanup(context.Background(), core.CleanupRequest{}) + return cleanupErr + }, + }) + if err != nil { + t.Fatal(err) + } + if cleanupErr != nil || len(api.deletedIDs) != 0 { + t.Fatalf("cleanup err=%v deleted=%#v", cleanupErr, api.deletedIDs) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(lease.LeaseID); err != nil || !exists { + t.Fatalf("ready claim exists=%t err=%v", exists, err) + } +} + +func TestFalCleanupProvisioningCASRejectsConcurrentReadyPublication(t *testing.T) { + started := time.Date(2026, time.July, 9, 12, 0, 0, 0, time.UTC) + clock := &mutableFalClock{now: started} + api := &fakeFalAPI{instances: map[string]ComputeInstance{"inst_ready": readyFalInstance("inst_ready", "203.0.113.42")}} + b := newFalTestBackend(t, api) + b.rt.Clock = clock + b.pollTimeout = time.Minute + claim, err := b.persistRecoveryClaimAtIfUnchanged( + "cbx_cleanup_ready_race", "cleanup-ready-race", b.configForRun(), "", "inst_ready", "provisioning", false, started, core.LeaseClaim{}, false, + ) + if err != nil { + t.Fatal(err) + } + claim, err = markFalAcquireLifetimeClaimForTest(claim) + if err != nil { + t.Fatal(err) + } + unlockReady, err := lockFalLeaseOperation(context.Background(), claim.LeaseID) + if err != nil { + t.Fatal(err) + } + clientRequested := make(chan struct{}) + b.clientFactory = func(Config, Runtime) (computeAPI, error) { + close(clientRequested) + return api, nil + } + + cleanupDone := make(chan error, 1) + go func() { + cleanupDone <- b.Cleanup(context.Background(), core.CleanupRequest{}) + }() + select { + case <-clientRequested: + case <-time.After(time.Second): + unlockReady() + t.Fatal("cleanup did not reach provider client") + } + current, exists, err := core.ReadLeaseClaimWithPresence(claim.LeaseID) + if err != nil || !exists { + unlockReady() + t.Fatalf("read provisioning claim: exists=%t err=%v", exists, err) + } + ready := current + ready.Labels = cloneLabels(current.Labels) + delete(ready.Labels, "recovery") + ready.Labels["state"] = "ready" + if err := core.ReplaceLeaseClaimIfUnchangedDurable(current.LeaseID, current, ready); err != nil { + unlockReady() + t.Fatal(err) + } + unlockReady() + if err := <-cleanupDone; err == nil { + t.Fatal("expected stale provisioning snapshot rejection") + } + if len(api.deletedIDs) != 0 { + t.Fatalf("concurrently published ready instance deleted: %#v", api.deletedIDs) + } + current, exists, err = core.ReadLeaseClaimWithPresence(claim.LeaseID) + if err != nil || !exists || current.Labels["state"] != "ready" { + t.Fatalf("current claim=%#v exists=%t err=%v", current, exists, err) + } +} + +func TestFalWaitForInstanceReadyBoundsBlockedProviderRead(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + b.pollTimeout = 25 * time.Millisecond + api := &fakeFalAPI{blockGetUntilContext: true} + started := time.Now() + _, err := b.waitForInstanceReady(context.Background(), api, "inst_blocked") + if err == nil || !errors.Is(err, context.DeadlineExceeded) || !strings.Contains(err.Error(), "timed out waiting") { + t.Fatalf("wait err=%v", err) + } + if elapsed := time.Since(started); elapsed > time.Second { + t.Fatalf("blocked provider read outlived poll timeout: %v", elapsed) + } +} + +func TestFalWaitForInstanceReadyRejectsChangedIdentity(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_requested": readyFalInstance("inst_other", "203.0.113.42"), + }} + b := newFalTestBackend(t, api) + if _, err := b.waitForInstanceReady(context.Background(), api, "inst_requested"); err == nil || !strings.Contains(err.Error(), "changed identity inst_other") { + t.Fatalf("wait err=%v", err) + } +} + +func TestFalCleanupSkipsOtherCredentialClaimsWithoutBlockingMatches(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_first": readyFalInstance("inst_first", "203.0.113.20"), + "inst_second": readyFalInstance("inst_second", "203.0.113.21"), + }} + b := newFalTestBackend(t, api) + firstCfg := b.cfg + secondCfg := b.cfg + secondCfg.Fal.APIKey = "second-test-key" + claimFalLease(t, firstCfg, "cbx_first123456", "first", "inst_first", "203.0.113.20", true) + claimFalLease(t, secondCfg, "cbx_second12345", "second", "inst_second", "203.0.113.21", true) + delete(api.instances, "inst_second") + + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 1 || api.deletedIDs[0] != "inst_first" { + t.Fatalf("first credential deletedIDs=%#v", api.deletedIDs) + } + if _, ok, err := core.ResolveLeaseClaimForProvider("second", providerName); err != nil || !ok { + t.Fatalf("other credential claim retained=%v err=%v", ok, err) + } + + b.cfg.Fal.APIKey = secondCfg.Fal.APIKey + api.instances["inst_second"] = readyFalInstance("inst_second", "203.0.113.21") + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if len(api.deletedIDs) != 2 || api.deletedIDs[1] != "inst_second" { + t.Fatalf("second credential deletedIDs=%#v", api.deletedIDs) + } +} + +func TestFalCleanupWithoutCredentialStillFinalizesConfirmedDeletion(t *testing.T) { + b := newFalTestBackend(t, &fakeFalAPI{}) + const retainedLeaseID = "cbx_a_missing_key" + const confirmedLeaseID = "cbx_z_confirmed" + claimFalLease(t, b.cfg, retainedLeaseID, "missing-key", "inst_missing_key", "203.0.113.20", true) + claimFalLease(t, b.cfg, confirmedLeaseID, "confirmed", "inst_confirmed", "203.0.113.21", false) + keyPath, _, err := core.EnsureTestboxKeyForConfig(b.cfg, confirmedLeaseID) + if err != nil { + t.Fatal(err) + } + claim, exists, err := core.ReadLeaseClaimWithPresence(confirmedLeaseID) + if err != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, err) + } + if _, err := persistFalDeleteState(claim, falDeleteConfirmedLabel, "inst_confirmed"); err != nil { + t.Fatal(err) + } + b.cfg.Fal.APIKey = "" + b.clientFactory = func(Config, Runtime) (computeAPI, error) { + return nil, errors.New("provider client must not be constructed") + } + + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(retainedLeaseID); err != nil || !exists { + t.Fatalf("unverified claim exists=%t err=%v", exists, err) + } + if _, exists, err := core.ReadLeaseClaimWithPresence(confirmedLeaseID); err != nil || exists { + t.Fatalf("confirmed claim exists=%t err=%v", exists, err) + } + if _, err := os.Stat(keyPath); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("confirmed deletion key residue: %v", err) + } +} + +func TestFalCleanupRejectsClaimChangedBeforeDeletion(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_expired": readyFalInstance("inst_expired", "203.0.113.20"), + }} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_expired1234", "expired", "inst_expired", "203.0.113.20", true) + b.clientFactory = func(Config, Runtime) (computeAPI, error) { + claim, ok, err := core.ReadLeaseClaimWithPresence("cbx_expired1234") + if err != nil || !ok { + return nil, fmt.Errorf("read claim before cleanup: ok=%v err=%w", ok, err) + } + labels := cloneLabels(claim.Labels) + labels["state"] = "renewed" + if _, err := core.UpdateLeaseClaimLabelsIfUnchanged(claim.LeaseID, claim, labels); err != nil { + return nil, err + } + return api, nil + } + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err == nil { + t.Fatal("expected changed-claim cleanup rejection") + } + if len(api.deletedIDs) != 0 { + t.Fatalf("changed claim was deleted: %#v", api.deletedIDs) + } + if _, ok, err := core.ReadLeaseClaimWithPresence("cbx_expired1234"); err != nil || !ok { + t.Fatalf("changed claim retained=%v err=%v", ok, err) + } +} + +func TestFalCleanupRetainsClaimWhenAbsenceIsNotAccountBound(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{}} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_absent12345", "absent", "inst_absent", "203.0.113.31", false) + + if err := b.Cleanup(context.Background(), core.CleanupRequest{}); err != nil { + t.Fatal(err) + } + if claim, ok, err := core.ResolveLeaseClaimForProvider("absent", providerName); err != nil || !ok || claim.CloudID != "inst_absent" { + t.Fatalf("provider-absent claim=%#v ok=%v err=%v", claim, ok, err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("provider-absent cleanup issued delete: %#v", api.deletedIDs) + } + views, err := b.List(context.Background(), core.ListRequest{}) + if err != nil { + t.Fatal(err) + } + if len(views) != 1 || views[0].Status != "provider-absence-unverified" || views[0].CloudID != "inst_absent" { + t.Fatalf("provider-absent views=%#v", views) + } +} + +func TestFalListShowsClaimsWhenProviderVerificationIsUnavailable(t *testing.T) { + api := &fakeFalAPI{getErr: errors.New("control plane unavailable")} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_unverified12", "unverified", "inst_unverified", "203.0.113.32", false) + + views, err := b.List(context.Background(), core.ListRequest{}) + if err != nil { + t.Fatal(err) + } + if len(views) != 1 || views[0].Status != "provider-verification-unavailable" || views[0].CloudID != "inst_unverified" { + t.Fatalf("views=%#v", views) + } + + b.cfg.Fal.APIKey = "" + views, err = b.List(context.Background(), core.ListRequest{}) + if err != nil { + t.Fatal(err) + } + if len(views) != 1 || views[0].Status != "credential-binding-mismatch" { + t.Fatalf("missing-credential views=%#v", views) + } + if views[0].Labels[falCredentialBindingLabel] != "" { + t.Fatalf("credential binding leaked into list labels: %#v", views[0].Labels) + } +} + +func TestFalReleaseFinalizesProviderConfirmedAbsence(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{}} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_absent12345", "absent", "inst_absent", "203.0.113.31", false) + keyPath, _, err := core.EnsureTestboxKeyForConfig(b.cfg, "cbx_absent12345") + if err != nil { + t.Fatal(err) + } + + err = b.ReleaseLease(context.Background(), core.ReleaseLeaseRequest{Lease: core.LeaseTarget{ + LeaseID: "cbx_absent12345", + Server: core.Server{CloudID: "inst_absent", Provider: providerName, Labels: map[string]string{"lease": "cbx_absent12345"}}, + }}) + if err != nil { + t.Fatalf("release err=%v", err) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("delete issued after unverified absence: %#v", api.deletedIDs) + } + if _, ok, err := core.ResolveLeaseClaimForProvider("absent", providerName); err != nil || ok { + t.Fatalf("claim retained=%v err=%v", ok, err) + } + if _, err := os.Stat(keyPath); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("key residue after provider-confirmed absence: %v", err) + } +} + +func TestFalResolveRebindsRotatedCredentialAfterProviderProof(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_rotated": readyFalInstance("inst_rotated", "203.0.113.33"), + }} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_rotated1234", "rotated", "inst_rotated", "203.0.113.33", false) + b.cfg.Fal.APIKey = "rotated-test-key" + + lease, err := b.Resolve(context.Background(), core.ResolveRequest{ID: "rotated", ReleaseOnly: true}) + if err != nil { + t.Fatal(err) + } + claim, ok, err := core.ReadLeaseClaimWithPresence(lease.LeaseID) + if err != nil || !ok { + t.Fatalf("claim exists=%t err=%v", ok, err) + } + if got, want := claim.Labels[falCredentialBindingLabel], falCredentialBinding(b.cfg); got != want { + t.Fatalf("credential binding=%q want %q", got, want) + } +} + +func TestFalCleanupDryRunDoesNotPersistCredentialRebinding(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{ + "inst_rotated_dry_run": readyFalInstance("inst_rotated_dry_run", "203.0.113.35"), + }} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_rotateddry1", "rotated-dry-run", "inst_rotated_dry_run", "203.0.113.35", true) + original, ok, err := core.ResolveLeaseClaimForProvider("rotated-dry-run", providerName) + if err != nil || !ok { + t.Fatalf("original claim retained=%v err=%v", ok, err) + } + originalBinding := original.Labels[falCredentialBindingLabel] + b.cfg.Fal.APIKey = "rotated-test-key" + + if err := b.Cleanup(context.Background(), core.CleanupRequest{DryRun: true}); err != nil { + t.Fatal(err) + } + retained, ok, err := core.ResolveLeaseClaimForProvider("rotated-dry-run", providerName) + if err != nil || !ok { + t.Fatalf("claim retained=%v err=%v", ok, err) + } + if got := retained.Labels[falCredentialBindingLabel]; got != originalBinding { + t.Fatalf("dry-run changed credential binding from %q to %q", originalBinding, got) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("dry-run deleted=%#v", api.deletedIDs) + } +} + +func TestFalCleanupPropagatesCredentialRebindingVerificationFailure(t *testing.T) { + api := &fakeFalAPI{getErr: errors.New("control plane unavailable")} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_rebindfailure", "rebind-failure", "inst_rebind_failure", "203.0.113.36", true) + b.cfg.Fal.APIKey = "rotated-test-key" + + err := b.Cleanup(context.Background(), core.CleanupRequest{}) + if err == nil || !strings.Contains(err.Error(), "control plane unavailable") { + t.Fatalf("cleanup err=%v", err) + } + if _, exists, readErr := core.ReadLeaseClaimWithPresence("cbx_rebindfailure"); readErr != nil || !exists { + t.Fatalf("claim exists=%t err=%v", exists, readErr) + } + if len(api.deletedIDs) != 0 { + t.Fatalf("verification failure deleted=%#v", api.deletedIDs) + } +} + +func TestFalResolveRejectsRotatedCredentialWithoutProviderProof(t *testing.T) { + api := &fakeFalAPI{instances: map[string]ComputeInstance{}} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_otherworkspace", "other-workspace", "inst_other", "203.0.113.34", false) + original, ok, err := core.ResolveLeaseClaimForProvider("other-workspace", providerName) + if err != nil || !ok { + t.Fatalf("original claim retained=%v err=%v", ok, err) + } + originalBinding := original.Labels[falCredentialBindingLabel] + b.cfg.Fal.APIKey = "other-workspace-key" + + if _, err := b.Resolve(context.Background(), core.ResolveRequest{ID: "other-workspace", ReleaseOnly: true}); err == nil || !strings.Contains(err.Error(), "credential rotation") { + t.Fatalf("resolve err=%v", err) + } + retained, ok, err := core.ResolveLeaseClaimForProvider("other-workspace", providerName) + if err != nil || !ok { + t.Fatalf("claim retained=%v err=%v", ok, err) + } + if got := retained.Labels[falCredentialBindingLabel]; got != originalBinding { + t.Fatalf("failed credential proof changed binding from %q to %q", originalBinding, got) + } +} + +func TestFalTouchPersistsLocalClaimLabels(t *testing.T) { + api := &fakeFalAPI{} + b := newFalTestBackend(t, api) + claimFalLease(t, b.cfg, "cbx_touch123456", "touch", "inst_touch", "203.0.113.30", false) + claim, ok, err := core.ResolveLeaseClaimForProvider("touch", providerName) + if err != nil || !ok { + t.Fatalf("claim ok=%v err=%v", ok, err) + } + server := core.Server{CloudID: "inst_touch", Provider: providerName, Labels: claim.Labels} + touched, err := b.Touch(context.Background(), core.TouchRequest{ + Lease: core.LeaseTarget{LeaseID: claim.LeaseID, Server: server}, + State: "running", + IdleTimeout: time.Hour, + }) + if err != nil { + t.Fatal(err) + } + if touched.Labels["state"] != "running" { + t.Fatalf("touched=%#v", touched.Labels) + } + updated, ok, err := core.ResolveLeaseClaimForProvider("touch", providerName) + if err != nil || !ok || updated.Labels["state"] != "running" { + t.Fatalf("updated=%#v ok=%v err=%v", updated, ok, err) + } +} + +func readyFalInstance(id, ip string) ComputeInstance { + return ComputeInstance{ + ID: id, + InstanceType: InstanceTypeH100x1, + Sector: Sector1, + Region: "us-west", + IP: ip, + Status: InstanceStatusReady, + } +} + +func claimFalLease(t *testing.T, cfg Config, leaseID, slug, cloudID, host string, expired bool) { + t.Helper() + labels := falLabels(cfg, leaseID, slug, false, time.Now().UTC()) + labels[falCredentialBindingLabel] = falCredentialBinding(cfg) + labels["ssh_user"] = cfg.SSHUser + labels["ssh_port"] = cfg.SSHPort + if expired { + labels["expires_at"] = core.LeaseLabelTime(time.Now().Add(-time.Hour)) + } + server := core.Server{CloudID: cloudID, Provider: providerName, Name: slug, Status: "ready", Labels: labels} + server.PublicNet.IPv4.IP = host + server.ServerType.Name = cfg.Fal.InstanceType + target := core.SSHTargetFromConfig(cfg, host) + if err := core.ClaimLeaseTargetForConfig(leaseID, slug, cfg, server, target, cfg.IdleTimeout); err != nil { + t.Fatal(err) + } +} diff --git a/internal/providers/fal/client.go b/internal/providers/fal/client.go new file mode 100644 index 000000000..e287b348e --- /dev/null +++ b/internal/providers/fal/client.go @@ -0,0 +1,322 @@ +package fal + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "net/http" + "net/url" + "strings" + "time" + + "github.com/openclaw/crabbox/internal/providers/shared" +) + +type computeAPI interface { + ListInstances(ctx context.Context, limit int, cursor string) (ListInstancesResponse, error) + GetInstance(ctx context.Context, id string) (ComputeInstance, error) + CreateInstance(ctx context.Context, input CreateInstanceRequest, idempotencyKey string) (ComputeInstance, error) + DeleteInstance(ctx context.Context, id string) error +} + +type client struct { + apiKey string + apiURL string + httpClient *http.Client + requestTimeout time.Duration +} + +type falTransportError struct { + cause error + message string +} + +func (e *falTransportError) Error() string { return e.message } +func (e *falTransportError) Unwrap() error { return e.cause } + +func safeFalTransportCause(err error) error { + if errors.Is(err, context.Canceled) { + return context.Canceled + } + if errors.Is(err, context.DeadlineExceeded) { + return context.DeadlineExceeded + } + return nil +} + +const ( + maxResponseBytes = 16 << 20 + defaultRequestTimeout = 2 * time.Minute +) + +var errFalCrossOriginRedirect = errors.New("fal refused cross-origin redirect") +var errFalUnsafeMutationRedirect = errors.New("fal refused method-rewriting mutation redirect") + +// APIError mirrors fal's documented standard error envelope. Request IDs are +// taken from the error body when present; the OpenAPI schema does not document a +// request-id response header for Compute as of 2026-06-25. +type APIError struct { + StatusCode int + Status string + Type string + Message string + DocsURL string + RequestID string +} + +func (e *APIError) Error() string { + if e == nil { + return "" + } + message := strings.TrimSpace(e.Message) + if message == "" { + message = e.Status + } + if e.Type != "" { + return fmt.Sprintf("fal API %s: %s", e.Type, message) + } + return fmt.Sprintf("fal API %s", message) +} + +func newClient(cfg Config, rt Runtime) (computeAPI, error) { + apiKey := strings.TrimSpace(cfg.Fal.APIKey) + if apiKey == "" { + return nil, exit(2, "provider=%s requires fal credentials in environment", providerName) + } + apiURL, err := validateFalAPIURL(cfg.Fal.APIURL) + if err != nil { + return nil, err + } + httpClient := rt.HTTP + if httpClient == nil { + httpClient = http.DefaultClient + } + return &client{ + apiKey: apiKey, + apiURL: apiURL, + httpClient: secureHTTPClient(httpClient, apiURL), + requestTimeout: defaultRequestTimeout, + }, nil +} + +func validateFalAPIURL(raw string) (string, error) { + apiURL := strings.TrimRight(strings.TrimSpace(blank(raw, defaultAPIURL)), "/") + parsed, err := url.Parse(apiURL) + if err != nil || parsed.Scheme == "" || parsed.Host == "" || parsed.Hostname() == "" || parsed.Opaque != "" { + return "", exit(2, "%s api url is invalid", providerName) + } + if parsed.RawPath != "" { + return "", exit(2, "%s api url contains an unsupported escaped path", providerName) + } + if parsed.User != nil || parsed.RawQuery != "" || parsed.ForceQuery || parsed.Fragment != "" { + return "", exit(2, "%s api url contains unsupported sensitive components", providerName) + } + if parsed.Scheme != "https" && !isLoopbackHTTPURL(parsed) { + return "", exit(2, "%s api url must use https unless it targets localhost", providerName) + } + return apiURL, nil +} + +func secureHTTPClient(source *http.Client, apiURL string) *http.Client { + client := *source + trusted, _ := url.Parse(apiURL) + originalCheckRedirect := source.CheckRedirect + client.CheckRedirect = func(req *http.Request, via []*http.Request) error { + if !sameOrigin(trusted, req.URL) { + return errFalCrossOriginRedirect + } + if len(via) > 0 && isMutationMethod(via[0].Method) { + statusCode := 0 + if req.Response != nil { + statusCode = req.Response.StatusCode + } + if (statusCode != http.StatusTemporaryRedirect && statusCode != http.StatusPermanentRedirect) || req.Method != via[0].Method { + return errFalUnsafeMutationRedirect + } + } + if originalCheckRedirect != nil { + return originalCheckRedirect(req, via) + } + if len(via) >= 10 { + return errors.New("stopped after 10 redirects") + } + return nil + } + return &client +} + +func isMutationMethod(method string) bool { + return method == http.MethodPost || method == http.MethodDelete +} + +func sameOrigin(a, b *url.URL) bool { + return a != nil && b != nil && + strings.EqualFold(a.Scheme, b.Scheme) && + strings.EqualFold(a.Hostname(), b.Hostname()) && + effectivePort(a) == effectivePort(b) +} + +func effectivePort(value *url.URL) string { + if port := value.Port(); port != "" { + return port + } + switch strings.ToLower(value.Scheme) { + case "https": + return "443" + case "http": + return "80" + default: + return "" + } +} + +func isLoopbackHTTPURL(value *url.URL) bool { + if !strings.EqualFold(value.Scheme, "http") { + return false + } + host := value.Hostname() + if strings.EqualFold(host, "localhost") { + return true + } + ip := net.ParseIP(host) + return ip != nil && ip.IsLoopback() +} + +func (c *client) ListInstances(ctx context.Context, limit int, cursor string) (ListInstancesResponse, error) { + path := "/compute/instances" + query := url.Values{} + if limit > 0 { + query.Set("limit", fmt.Sprintf("%d", limit)) + } + if cursor != "" { + query.Set("cursor", cursor) + } + if encoded := query.Encode(); encoded != "" { + path += "?" + encoded + } + var out ListInstancesResponse + if err := c.do(ctx, http.MethodGet, path, nil, "", &out); err != nil { + return ListInstancesResponse{}, err + } + return out, nil +} + +func (c *client) GetInstance(ctx context.Context, id string) (ComputeInstance, error) { + id = strings.TrimSpace(id) + if id == "" { + return ComputeInstance{}, fmt.Errorf("fal instance id is required") + } + var out ComputeInstance + if err := c.do(ctx, http.MethodGet, "/compute/instances/"+url.PathEscape(id), nil, "", &out); err != nil { + return ComputeInstance{}, err + } + if strings.TrimSpace(out.ID) == "" { + return ComputeInstance{}, errors.New("fal get instance response is missing an instance id") + } + if out.ID != id { + return ComputeInstance{}, errors.New("fal get instance response id does not match the requested instance") + } + return out, nil +} + +func (c *client) CreateInstance(ctx context.Context, input CreateInstanceRequest, idempotencyKey string) (ComputeInstance, error) { + var out ComputeInstance + if err := c.do(ctx, http.MethodPost, "/compute/instances", input, strings.TrimSpace(idempotencyKey), &out); err != nil { + return ComputeInstance{}, err + } + return out, nil +} + +func (c *client) DeleteInstance(ctx context.Context, id string) error { + id = strings.TrimSpace(id) + if id == "" { + return fmt.Errorf("fal instance id is required") + } + return c.do(ctx, http.MethodDelete, "/compute/instances/"+url.PathEscape(id), nil, "", nil) +} + +func (c *client) do(ctx context.Context, method, path string, body any, idempotencyKey string, out any) error { + timeout := c.requestTimeout + if timeout <= 0 { + timeout = defaultRequestTimeout + } + requestCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + + var reader io.Reader + if body != nil { + data, err := json.Marshal(body) + if err != nil { + return err + } + reader = bytes.NewReader(data) + } + req, err := http.NewRequestWithContext(requestCtx, method, c.apiURL+path, reader) + if err != nil { + return err + } + req.Header.Set("Authorization", "Key "+c.apiKey) + req.Header.Set("Accept", "application/json") + if body != nil { + req.Header.Set("Content-Type", "application/json") + } + if idempotencyKey != "" { + req.Header.Set("Idempotency-Key", idempotencyKey) + } + resp, err := c.httpClient.Do(req) + if err != nil { + if errors.Is(err, errFalCrossOriginRedirect) || errors.Is(err, errFalUnsafeMutationRedirect) { + if errors.Is(err, errFalCrossOriginRedirect) { + return errFalCrossOriginRedirect + } + return errFalUnsafeMutationRedirect + } + return &falTransportError{cause: safeFalTransportCause(err), message: redactAPIErrorText(err.Error(), c.apiKey)} + } + defer resp.Body.Close() + data, readErr := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes+1)) + if readErr != nil { + return readErr + } + if len(data) > maxResponseBytes { + return fmt.Errorf("fal response exceeds %d bytes", maxResponseBytes) + } + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + return decodeAPIError(resp, data, c.apiKey) + } + if out != nil { + if len(bytes.TrimSpace(data)) == 0 { + return errors.New("decode fal data: empty response body") + } + if err := json.Unmarshal(data, out); err != nil { + return fmt.Errorf("decode fal data: %w", err) + } + } + return nil +} + +func decodeAPIError(resp *http.Response, data []byte, apiKey string) error { + apiErr := &APIError{StatusCode: resp.StatusCode, Status: strings.TrimSpace(redactAPIErrorText(resp.Status, apiKey))} + var body APIErrorBody + if len(bytes.TrimSpace(data)) > 0 && json.Unmarshal(data, &body) == nil { + apiErr.Type = strings.TrimSpace(redactAPIErrorText(body.Error.Type, apiKey)) + apiErr.Message = strings.TrimSpace(redactAPIErrorText(body.Error.Message, apiKey)) + apiErr.DocsURL = strings.TrimSpace(redactAPIErrorText(body.Error.DocsURL, apiKey)) + apiErr.RequestID = strings.TrimSpace(redactAPIErrorText(body.Error.RequestID, apiKey)) + } + return apiErr +} + +func redactAPIErrorText(value, apiKey string) string { + if apiKey != "" { + value = strings.ReplaceAll(value, apiKey, "") + if encoded, err := json.Marshal(apiKey); err == nil && len(encoded) >= 2 { + value = strings.ReplaceAll(value, string(encoded[1:len(encoded)-1]), "") + } + } + return shared.RedactErrorSecrets(value) +} diff --git a/internal/providers/fal/client_test.go b/internal/providers/fal/client_test.go new file mode 100644 index 000000000..c3dad61c3 --- /dev/null +++ b/internal/providers/fal/client_test.go @@ -0,0 +1,529 @@ +package fal + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +type roundTripFunc func(*http.Request) (*http.Response, error) + +func (fn roundTripFunc) RoundTrip(req *http.Request) (*http.Response, error) { + return fn(req) +} + +func TestClientUsesOfficialComputePathsAndKeyAuth(t *testing.T) { + var seen []string + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + seen = append(seen, r.Method+" "+r.URL.RequestURI()) + if got := r.Header.Get("Authorization"); got != "Key test-key" { + t.Fatalf("Authorization=%q, want Key auth", got) + } + switch { + case r.Method == http.MethodGet && r.URL.Path == "/compute/instances": + if r.URL.Query().Get("limit") != "5" || r.URL.Query().Get("cursor") != "Mg==" { + t.Fatalf("query=%s", r.URL.RawQuery) + } + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte(`{"next_cursor":null,"has_more":false,"instances":[{"id":"inst_abc123xyz","instance_type":"gpu_1x_h100_sxm5","region":"us-west","sector":"sector_1","ip":"203.0.113.42","status":"ready","creator_user_nickname":"developer"}]}`)) + case r.Method == http.MethodGet && r.URL.Path == "/compute/instances/inst_abc123xyz": + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte(`{"id":"inst_abc123xyz","instance_type":"gpu_1x_h100_sxm5","region":"us-west","sector":"sector_1","ip":"203.0.113.42","status":"ready","creator_user_nickname":"developer"}`)) + case r.Method == http.MethodPost && r.URL.Path == "/compute/instances": + if got := r.Header.Get("Idempotency-Key"); got != "idem-1" { + t.Fatalf("Idempotency-Key=%q", got) + } + var req CreateInstanceRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Fatal(err) + } + if req.InstanceType != InstanceTypeH100x8 || req.SSHKey != "ssh-ed25519 AAA test" || req.Sector != Sector2 { + t.Fatalf("create request=%#v", req) + } + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{"id":"inst_def456uvw","instance_type":"gpu_8x_h100_sxm5","region":"us-east","sector":"sector_2","status":"provisioning","creator_user_nickname":"developer"}`)) + case r.Method == http.MethodDelete && r.URL.Path == "/compute/instances/inst_def456uvw": + w.WriteHeader(http.StatusNoContent) + default: + t.Fatalf("unexpected request: %s %s", r.Method, r.URL.RequestURI()) + } + })) + defer server.Close() + + api, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: server.URL}}, Runtime{HTTP: server.Client()}) + if err != nil { + t.Fatal(err) + } + list, err := api.ListInstances(context.Background(), 5, "Mg==") + if err != nil { + t.Fatal(err) + } + if len(list.Instances) != 1 || list.Instances[0].ID != "inst_abc123xyz" || !list.Instances[0].Status.Known() { + t.Fatalf("list=%#v", list) + } + instance, err := api.GetInstance(context.Background(), "inst_abc123xyz") + if err != nil { + t.Fatal(err) + } + if instance.IP != "203.0.113.42" || instance.Status != InstanceStatusReady { + t.Fatalf("instance=%#v", instance) + } + created, err := api.CreateInstance(context.Background(), CreateInstanceRequest{InstanceType: InstanceTypeH100x8, SSHKey: "ssh-ed25519 AAA test", Sector: Sector2}, "idem-1") + if err != nil { + t.Fatal(err) + } + if created.ID != "inst_def456uvw" || created.Status != InstanceStatusProvisioning { + t.Fatalf("created=%#v", created) + } + if err := api.DeleteInstance(context.Background(), "inst_def456uvw"); err != nil { + t.Fatal(err) + } + want := []string{ + "GET /compute/instances?cursor=Mg%3D%3D&limit=5", + "GET /compute/instances/inst_abc123xyz", + "POST /compute/instances", + "DELETE /compute/instances/inst_def456uvw", + } + if strings.Join(seen, "\n") != strings.Join(want, "\n") { + t.Fatalf("requests=\n%s\nwant=\n%s", strings.Join(seen, "\n"), strings.Join(want, "\n")) + } +} + +func TestClientDecodesStandardErrorBody(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusForbidden) + _, _ = w.Write([]byte(`{"error":{"type":"authorization_error","message":"Access denied","docs_url":"https://fal.ai/docs","request_id":"req_123"}}`)) + })) + defer server.Close() + api, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: server.URL}}, Runtime{HTTP: server.Client()}) + if err != nil { + t.Fatal(err) + } + _, err = api.ListInstances(context.Background(), 0, "") + var apiErr *APIError + if !errors.As(err, &apiErr) { + t.Fatalf("err=%T %[1]v, want APIError", err) + } + if apiErr.StatusCode != http.StatusForbidden || apiErr.Type != "authorization_error" || apiErr.RequestID != "req_123" { + t.Fatalf("apiErr=%#v", apiErr) + } +} + +func TestClientGetInstanceRejectsMissingOrMismatchedResponseID(t *testing.T) { + for name, tc := range map[string]struct { + responseID string + want string + }{ + "empty": {want: "missing an instance id"}, + "whitespace": {responseID: " ", want: "missing an instance id"}, + "mismatch": {responseID: "inst_other", want: "does not match"}, + } { + t.Run(name, func(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + if err := json.NewEncoder(w).Encode(ComputeInstance{ID: tc.responseID}); err != nil { + t.Fatal(err) + } + })) + defer server.Close() + + api, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: server.URL}}, Runtime{HTTP: server.Client()}) + if err != nil { + t.Fatal(err) + } + if _, err := api.GetInstance(context.Background(), "inst_requested"); err == nil || !strings.Contains(err.Error(), tc.want) { + t.Fatalf("GetInstance error=%v, want %q", err, tc.want) + } + }) + } +} + +func TestClientBoundsStalledRequestsAndHonorsEarlierDeadlines(t *testing.T) { + for name, tc := range map[string]struct { + requestTimeout time.Duration + parentTimeout time.Duration + clientTimeout time.Duration + }{ + "request default": {requestTimeout: 20 * time.Millisecond}, + "parent": {requestTimeout: 500 * time.Millisecond, parentTimeout: 20 * time.Millisecond}, + "http client": {requestTimeout: 500 * time.Millisecond, clientTimeout: 20 * time.Millisecond}, + } { + t.Run(name, func(t *testing.T) { + httpClient := &http.Client{ + Timeout: tc.clientTimeout, + Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) { + <-req.Context().Done() + return nil, req.Context().Err() + }), + } + api, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: "https://api.fal.ai/v1"}}, Runtime{HTTP: httpClient}) + if err != nil { + t.Fatal(err) + } + falClient := api.(*client) + if falClient.requestTimeout != defaultRequestTimeout { + t.Fatalf("default request timeout=%s, want %s", falClient.requestTimeout, defaultRequestTimeout) + } + falClient.requestTimeout = tc.requestTimeout + + ctx := context.Background() + cancel := func() {} + if tc.parentTimeout > 0 { + ctx, cancel = context.WithTimeout(ctx, tc.parentTimeout) + } + defer cancel() + started := time.Now() + _, err = api.ListInstances(ctx, 0, "") + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("ListInstances error=%v, want context deadline exceeded", err) + } + if elapsed := time.Since(started); elapsed >= 300*time.Millisecond { + t.Fatalf("stalled request canceled after %s, want earlier deadline", elapsed) + } + }) + } +} + +func TestClientRedactsAPIKeyReflectedByErrorResponse(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusBadRequest) + _, _ = w.Write([]byte(`{"error":{"message":"rejected Key test-key"}}`)) + })) + defer server.Close() + api, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: server.URL}}, Runtime{HTTP: server.Client()}) + if err != nil { + t.Fatal(err) + } + _, err = api.ListInstances(context.Background(), 0, "") + if err == nil || strings.Contains(err.Error(), "test-key") || !strings.Contains(err.Error(), "") { + t.Fatalf("error=%v", err) + } +} + +func TestClientRedactsJSONEscapedAPIKeyFromStructuredErrorFields(t *testing.T) { + apiKey := `test-"key\suffix` + body, err := json.Marshal(APIErrorBody{Error: APIErrorDetail{ + Type: "authorization_" + apiKey, + Message: "rejected " + apiKey, + DocsURL: "https://example.test/docs?token=" + apiKey, + RequestID: "request_" + apiKey, + }}) + if err != nil { + t.Fatal(err) + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusForbidden) + _, _ = w.Write(body) + })) + defer server.Close() + + api, err := newClient(Config{Fal: FalConfig{APIKey: apiKey, APIURL: server.URL}}, Runtime{HTTP: server.Client()}) + if err != nil { + t.Fatal(err) + } + _, err = api.ListInstances(context.Background(), 0, "") + var apiErr *APIError + if !errors.As(err, &apiErr) { + t.Fatalf("error=%T %[1]v, want APIError", err) + } + encodedKey, err := json.Marshal(apiKey) + if err != nil { + t.Fatal(err) + } + escapedKey := string(encodedKey[1 : len(encodedKey)-1]) + for field, value := range map[string]string{ + "error": apiErr.Error(), + "type": apiErr.Type, + "message": apiErr.Message, + "docs_url": apiErr.DocsURL, + "request_id": apiErr.RequestID, + } { + if strings.Contains(value, apiKey) || strings.Contains(value, escapedKey) { + t.Fatalf("%s leaked API key: %q", field, value) + } + if !strings.Contains(value, "redacted") { + t.Fatalf("%s omitted redaction marker: %q", field, value) + } + } +} + +func TestClientRedactsUnstructuredErrorBody(t *testing.T) { + apiKey := `raw-"secret\key` + encodedKey, err := json.Marshal(apiKey) + if err != nil { + t.Fatal(err) + } + escapedKey := string(encodedKey[1 : len(encodedKey)-1]) + body := `malformed={"message":"reflected ` + escapedKey + ` token=secondary-secret` + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusBadRequest) + _, _ = w.Write([]byte(body)) + })) + defer server.Close() + + api, err := newClient(Config{Fal: FalConfig{APIKey: apiKey, APIURL: server.URL}}, Runtime{HTTP: server.Client()}) + if err != nil { + t.Fatal(err) + } + _, err = api.ListInstances(context.Background(), 0, "") + if err == nil { + t.Fatal("expected API error") + } + text := err.Error() + for _, secret := range []string{apiKey, escapedKey, "secondary-secret", "malformed"} { + if strings.Contains(text, secret) { + t.Fatalf("unstructured API error leaked %q: %q", secret, text) + } + } + if !strings.Contains(text, "400 Bad Request") { + t.Fatalf("unstructured API error omitted status: %q", text) + } +} + +func TestClientRedactsTransportAndStatusErrors(t *testing.T) { + apiKey := "transport-secret-key" + transportCause := errors.New("transport reflected " + apiKey) + httpClient := &http.Client{Transport: roundTripFunc(func(*http.Request) (*http.Response, error) { + return nil, transportCause + })} + api, err := newClient(Config{Fal: FalConfig{APIKey: apiKey, APIURL: "https://api.fal.ai/v1"}}, Runtime{HTTP: httpClient}) + if err != nil { + t.Fatal(err) + } + _, err = api.ListInstances(context.Background(), 0, "") + if errors.Is(err, transportCause) || errors.Unwrap(err) != nil || strings.Contains(err.Error(), apiKey) || !strings.Contains(err.Error(), "redacted") { + t.Fatalf("transport error=%v", err) + } + + statusErr := decodeAPIError(&http.Response{ + StatusCode: http.StatusBadRequest, + Status: "400 reflected " + apiKey, + }, nil, apiKey) + if strings.Contains(statusErr.Error(), apiKey) || !strings.Contains(statusErr.Error(), "redacted") { + t.Fatalf("status error=%v", statusErr) + } +} + +func TestAPIErrorFormatting(t *testing.T) { + var nilErr *APIError + if got := nilErr.Error(); got != "" { + t.Fatalf("nil error=%q", got) + } + for name, tc := range map[string]struct { + err *APIError + want string + }{ + "typed": {err: &APIError{Type: "quota", Message: "limit reached"}, want: "fal API quota: limit reached"}, + "status": {err: &APIError{Status: "Forbidden"}, want: "fal API Forbidden"}, + "message": {err: &APIError{Message: "bad request"}, want: "fal API bad request"}, + } { + t.Run(name, func(t *testing.T) { + if got := tc.err.Error(); got != tc.want { + t.Fatalf("Error()=%q want %q", got, tc.want) + } + }) + } +} + +func TestClientRejectsPlainHTTPExceptLoopback(t *testing.T) { + if _, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: "http://api.fal.ai/v1"}}, Runtime{}); err == nil { + t.Fatal("accepted non-loopback http") + } + _, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: "http://user:secret@api.fal.ai/v1"}}, Runtime{}) + if err == nil { + t.Fatal("accepted non-loopback http with userinfo") + } + message := err.Error() + if strings.Contains(message, "user") || strings.Contains(message, "secret") { + t.Fatalf("api url error leaked userinfo: %q", message) + } + if _, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: "http://127.0.0.1:8080/v1"}}, Runtime{}); err != nil { + t.Fatalf("loopback http rejected: %v", err) + } +} + +func TestClientRejectsHostlessAPIURL(t *testing.T) { + for _, apiURL := range []string{"https://:443/v1", "https:///v1"} { + if _, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: apiURL}}, Runtime{}); err == nil { + t.Fatalf("accepted hostless API URL %q", apiURL) + } + } +} + +func TestClientRejectsCredentialBearingBaseURLComponents(t *testing.T) { + for name, apiURL := range map[string]string{ + "userinfo": "https://user:secret@api.fal.ai/v1", + "query": "https://api.fal.ai/v1?token=secret", + "fragment": "https://api.fal.ai/v1#secret", + } { + t.Run(name, func(t *testing.T) { + _, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: apiURL}}, Runtime{}) + if err == nil { + t.Fatal("accepted credential-bearing API URL") + } + if message := err.Error(); strings.Contains(message, "secret") || strings.Contains(message, "user") || strings.Contains(message, "token") { + t.Fatalf("api url error leaked sensitive component: %q", message) + } + }) + } +} + +func TestClientRejectsEscapedBaseURLPath(t *testing.T) { + _, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: "https://api.fal.ai/v1%2Ftenant"}}, Runtime{}) + if err == nil || !strings.Contains(err.Error(), "unsupported escaped path") { + t.Fatalf("escaped API URL error=%v", err) + } +} + +func TestClientRejectsMethodRewritingMutationRedirects(t *testing.T) { + for _, method := range []string{http.MethodPost, http.MethodDelete} { + for _, statusCode := range []int{http.StatusMovedPermanently, http.StatusFound, http.StatusSeeOther} { + t.Run(method+"/"+http.StatusText(statusCode), func(t *testing.T) { + redirected := false + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/redirected" { + redirected = true + w.WriteHeader(http.StatusNoContent) + return + } + http.Redirect(w, r, "/redirected", statusCode) + })) + defer server.Close() + api, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: server.URL}}, Runtime{HTTP: server.Client()}) + if err != nil { + t.Fatal(err) + } + if method == http.MethodPost { + _, err = api.CreateInstance(context.Background(), CreateInstanceRequest{InstanceType: InstanceTypeH100x1, SSHKey: "ssh-ed25519 AAA test"}, "idem-1") + } else { + err = api.DeleteInstance(context.Background(), "inst_abc123xyz") + } + if !errors.Is(err, errFalUnsafeMutationRedirect) { + t.Fatalf("redirect err=%v, want %v", err, errFalUnsafeMutationRedirect) + } + if redirected { + t.Fatal("mutation redirect reached redirected handler") + } + }) + } + } +} + +func TestClientPreservesMutationMethodsAcrossSafeRedirects(t *testing.T) { + for _, method := range []string{http.MethodPost, http.MethodDelete} { + for _, statusCode := range []int{http.StatusTemporaryRedirect, http.StatusPermanentRedirect} { + t.Run(method+"/"+http.StatusText(statusCode), func(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/redirected" { + http.Redirect(w, r, "/redirected", statusCode) + return + } + if r.Method != method { + t.Fatalf("redirected method=%s, want %s", r.Method, method) + } + if method == http.MethodPost { + w.WriteHeader(http.StatusCreated) + _, _ = w.Write([]byte(`{"id":"inst_created","instance_type":"gpu_1x_h100_sxm5","region":"us-west","status":"provisioning"}`)) + return + } + w.WriteHeader(http.StatusNoContent) + })) + defer server.Close() + api, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: server.URL}}, Runtime{HTTP: server.Client()}) + if err != nil { + t.Fatal(err) + } + if method == http.MethodPost { + _, err = api.CreateInstance(context.Background(), CreateInstanceRequest{InstanceType: InstanceTypeH100x1, SSHKey: "ssh-ed25519 AAA test"}, "idem-1") + } else { + err = api.DeleteInstance(context.Background(), "inst_abc123xyz") + } + if err != nil { + t.Fatal(err) + } + }) + } + } +} + +func TestClientRejectsMalformedListInventoryResponses(t *testing.T) { + for name, body := range map[string]string{ + "empty": "", + "null": "null", + "empty object": `{}`, + "missing next cursor": `{"has_more":false,"instances":[]}`, + "missing has more": `{"next_cursor":null,"instances":[]}`, + "missing instances": `{"next_cursor":null,"has_more":false}`, + "null has more": `{"next_cursor":null,"has_more":null,"instances":[]}`, + "null instances": `{"next_cursor":null,"has_more":false,"instances":null}`, + } { + t.Run(name, func(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(body)) + })) + defer server.Close() + api, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: server.URL}}, Runtime{HTTP: server.Client()}) + if err != nil { + t.Fatal(err) + } + if _, err := api.ListInstances(context.Background(), 0, ""); err == nil || !strings.Contains(err.Error(), "decode fal data") { + t.Fatalf("ListInstances error=%v", err) + } + }) + } +} + +func TestClientAcceptsStructurallyCompleteEmptyInventory(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"next_cursor":null,"has_more":false,"instances":[]}`)) + })) + defer server.Close() + api, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: server.URL}}, Runtime{HTTP: server.Client()}) + if err != nil { + t.Fatal(err) + } + result, err := api.ListInstances(context.Background(), 0, "") + if err != nil { + t.Fatal(err) + } + if result.HasMore || result.NextCursor != nil || result.Instances == nil || len(result.Instances) != 0 { + t.Fatalf("result=%#v", result) + } +} + +func TestClientRefusesCrossOriginRedirectBeforeReplayingAuth(t *testing.T) { + var redirectedAuth string + untrusted := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { + redirectedAuth = r.Header.Get("Authorization") + })) + defer untrusted.Close() + trusted := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, untrusted.URL+"/compute/instances?token=redirect-secret", http.StatusTemporaryRedirect) + })) + defer trusted.Close() + api, err := newClient(Config{Fal: FalConfig{APIKey: "test-key", APIURL: trusted.URL}}, Runtime{HTTP: trusted.Client()}) + if err != nil { + t.Fatal(err) + } + _, err = api.ListInstances(context.Background(), 0, "") + if err == nil || !strings.Contains(err.Error(), "refused cross-origin redirect") { + t.Fatalf("redirect err=%v", err) + } + if strings.Contains(err.Error(), "redirect-secret") || strings.Contains(err.Error(), untrusted.URL) { + t.Fatalf("redirect error leaked target: %v", err) + } + if redirectedAuth != "" { + t.Fatalf("auth replayed to untrusted origin: %q", redirectedAuth) + } +} + +func TestUnknownStatusIsNotKnown(t *testing.T) { + if InstanceStatus("booting").Known() { + t.Fatal("unexpected status should not be known") + } +} diff --git a/internal/providers/fal/config.go b/internal/providers/fal/config.go new file mode 100644 index 000000000..1b7ed612f --- /dev/null +++ b/internal/providers/fal/config.go @@ -0,0 +1,64 @@ +package fal + +import ( + "regexp" + "strings" +) + +const ( + defaultAPIURL = "https://api.fal.ai/v1" + defaultInstanceType = "gpu_1x_h100_sxm5" + defaultUser = "ubuntu" + defaultWorkRoot = "/home/ubuntu/crabbox" +) + +var falSSHUserPattern = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9._-]{0,31}$`) + +func applyFalDefaults(cfg *Config) { + if cfg == nil { + return + } + cfg.Fal.APIURL = strings.TrimSpace(cfg.Fal.APIURL) + cfg.Fal.InstanceType = strings.TrimSpace(cfg.Fal.InstanceType) + cfg.Fal.User = strings.TrimSpace(cfg.Fal.User) + cfg.Fal.WorkRoot = strings.TrimSpace(cfg.Fal.WorkRoot) + cfg.SSHUser = strings.TrimSpace(cfg.SSHUser) + cfg.WorkRoot = strings.TrimSpace(cfg.WorkRoot) + if cfg.ServerTypeExplicit && strings.TrimSpace(cfg.ServerType) != "" { + cfg.Fal.InstanceType = strings.TrimSpace(cfg.ServerType) + } + if cfg.Fal.APIURL == "" { + cfg.Fal.APIURL = defaultAPIURL + } + if cfg.Fal.InstanceType == "" { + cfg.Fal.InstanceType = defaultInstanceType + } + if cfg.Fal.User == "" { + cfg.Fal.User = defaultUser + } + if cfg.Fal.WorkRoot == "" { + cfg.Fal.WorkRoot = defaultWorkRoot + } + cfg.Provider = providerName + if cfg.TargetOS == "" { + cfg.TargetOS = targetLinux + } + if cfg.SSHUser == "" { + cfg.SSHUser = cfg.Fal.User + } + if cfg.SSHPort == "" { + cfg.SSHPort = "22" + } + cfg.SSHFallbackPorts = nil + if cfg.WorkRoot == "" { + cfg.WorkRoot = cfg.Fal.WorkRoot + } + cfg.ServerType = cfg.Fal.InstanceType +} + +func validateFalSSHUser(user string) error { + if !falSSHUserPattern.MatchString(user) { + return exit(2, "provider=%s SSH user must be a valid Linux login name, got %q", providerName, user) + } + return nil +} diff --git a/internal/providers/fal/core.go b/internal/providers/fal/core.go new file mode 100644 index 000000000..9c42782fc --- /dev/null +++ b/internal/providers/fal/core.go @@ -0,0 +1,42 @@ +package fal + +import ( + "flag" + "strings" + + core "github.com/openclaw/crabbox/internal/cli" +) + +type Config = core.Config +type FalConfig = core.FalConfig +type ProviderSpec = core.ProviderSpec +type Runtime = core.Runtime +type Backend = core.Backend +type DoctorRequest = core.DoctorRequest +type DoctorResult = core.DoctorResult + +const ( + providerName = "fal" + targetLinux = core.TargetLinux +) + +func exit(code int, format string, args ...any) core.ExitError { + return core.Exit(code, format, args...) +} + +func flagWasSet(fs *flag.FlagSet, name string) bool { + return core.FlagWasSet(fs, name) +} + +func blank(value, fallback string) string { + return core.Blank(value, fallback) +} + +func isFalProviderName(provider string) bool { + switch strings.ToLower(strings.TrimSpace(provider)) { + case providerName, "fal-ai": + return true + default: + return false + } +} diff --git a/internal/providers/fal/flags.go b/internal/providers/fal/flags.go new file mode 100644 index 000000000..297c966d2 --- /dev/null +++ b/internal/providers/fal/flags.go @@ -0,0 +1,61 @@ +package fal + +import ( + "flag" + + core "github.com/openclaw/crabbox/internal/cli" +) + +type falFlagValues struct { + APIURL *string + InstanceType *string + Sector *string + User *string + WorkRoot *string +} + +// RegisterFalProviderFlags exposes only non-secret fal settings. The API key is +// intentionally env-only so it cannot leak through shell history or process +// listings. +func RegisterFalProviderFlags(fs *flag.FlagSet, defaults Config) any { + return falFlagValues{ + APIURL: fs.String("fal-api-url", defaults.Fal.APIURL, "fal Platform API URL"), + InstanceType: fs.String("fal-instance-type", defaults.Fal.InstanceType, "fal Compute instance type"), + Sector: fs.String("fal-sector", defaults.Fal.Sector, "fal Compute sector for supported multi-node instance types"), + User: fs.String("fal-user", defaults.Fal.User, "SSH user for fal Compute instances"), + WorkRoot: fs.String("fal-work-root", defaults.Fal.WorkRoot, "remote Crabbox work root on fal Compute instances"), + } +} + +func ApplyFalProviderFlags(cfg *Config, fs *flag.FlagSet, values any) error { + v, ok := values.(falFlagValues) + if !ok { + return nil + } + if flagWasSet(fs, "fal-api-url") { + cfg.Fal.APIURL = *v.APIURL + } + if flagWasSet(fs, "fal-instance-type") { + cfg.Fal.InstanceType = *v.InstanceType + } + if flagWasSet(fs, "fal-sector") { + cfg.Fal.Sector = *v.Sector + } + if flagWasSet(fs, "fal-user") { + cfg.Fal.User = *v.User + cfg.SSHUser = *v.User + core.MarkSSHUserExplicit(cfg) + } + if flagWasSet(fs, "fal-work-root") { + cfg.Fal.WorkRoot = *v.WorkRoot + cfg.WorkRoot = *v.WorkRoot + core.MarkWorkRootExplicit(cfg) + } + if isFalProviderName(cfg.Provider) { + if err := (Provider{}).ValidateConfig(*cfg); err != nil { + return err + } + applyFalDefaults(cfg) + } + return nil +} diff --git a/internal/providers/fal/operation_lock.go b/internal/providers/fal/operation_lock.go new file mode 100644 index 000000000..2a41861e7 --- /dev/null +++ b/internal/providers/fal/operation_lock.go @@ -0,0 +1,142 @@ +package fal + +import ( + "context" + "os" + "path/filepath" + "strings" + "sync" + "time" + + "github.com/gofrs/flock" + core "github.com/openclaw/crabbox/internal/cli" +) + +var falOperationLocks sync.Map + +var ensureFalClaimNamespace = core.EnsureCrabboxClaimNamespaceDurable + +type falOperationSemaphore struct { + token chan struct{} +} + +func newFalOperationSemaphore() *falOperationSemaphore { + semaphore := &falOperationSemaphore{token: make(chan struct{}, 1)} + semaphore.token <- struct{}{} + return semaphore +} + +func lockFalLeaseOperation(ctx context.Context, leaseID string) (func(), error) { + if err := validateFalLeaseID(leaseID); err != nil { + return nil, err + } + return lockFalOperation(ctx, leaseID+".fal-operation.lock", "fal lease "+leaseID) +} + +func lockFalAcquireLifetime(ctx context.Context, leaseID string) (func(), error) { + if err := validateFalLeaseID(leaseID); err != nil { + return nil, err + } + return lockFalOperation(ctx, leaseID+".fal-acquire.lock", "fal acquire "+leaseID) +} + +func tryLockFalAcquireLifetime(ctx context.Context, leaseID string) (func(), bool, error) { + if err := validateFalLeaseID(leaseID); err != nil { + return nil, false, err + } + return tryLockFalOperation(ctx, leaseID+".fal-acquire.lock") +} + +func validateFalLeaseID(leaseID string) error { + if !strings.HasPrefix(leaseID, "cbx_") || strings.TrimPrefix(leaseID, "cbx_") == "" || filepath.Base(leaseID) != leaseID || leaseID == "." { + return exit(2, "invalid fal lease id %q", leaseID) + } + return nil +} + +func lockFalSlugAllocation(ctx context.Context) (func(), error) { + return lockFalOperation(ctx, "fal-slug-allocation.lock", "fal slug allocation") +} + +func lockFalOperation(ctx context.Context, lockName, description string) (func(), error) { + lockPath, semaphore, err := prepareFalOperationLock(lockName) + if err != nil { + return nil, err + } + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-semaphore.token: + } + + fileLock := flock.New(lockPath, flock.SetPermissions(0o600)) + locked, err := fileLock.TryLockContext(ctx, 50*time.Millisecond) + if err != nil { + semaphore.token <- struct{}{} + if contextErr := ctx.Err(); contextErr != nil { + return nil, contextErr + } + return nil, err + } + if !locked { + semaphore.token <- struct{}{} + if err := ctx.Err(); err != nil { + return nil, err + } + return nil, exit(2, "lock %s was not acquired", description) + } + return falOperationUnlock(fileLock, semaphore), nil +} + +func tryLockFalOperation(ctx context.Context, lockName string) (func(), bool, error) { + if err := ctx.Err(); err != nil { + return nil, false, err + } + lockPath, semaphore, err := prepareFalOperationLock(lockName) + if err != nil { + return nil, false, err + } + select { + case <-semaphore.token: + default: + return nil, false, nil + } + fileLock := flock.New(lockPath, flock.SetPermissions(0o600)) + locked, err := fileLock.TryLock() + if err != nil { + semaphore.token <- struct{}{} + return nil, false, err + } + if !locked { + semaphore.token <- struct{}{} + return nil, false, nil + } + return falOperationUnlock(fileLock, semaphore), true, nil +} + +func prepareFalOperationLock(lockName string) (string, *falOperationSemaphore, error) { + stateDir, err := core.CrabboxStateDir() + if err != nil { + return "", nil, err + } + if err := ensureFalClaimNamespace(); err != nil { + return "", nil, exit(2, "create fal claim namespace: %v", err) + } + lockDir := filepath.Join(stateDir, "claim-locks") + if err := os.MkdirAll(lockDir, 0o700); err != nil { + return "", nil, exit(2, "create fal lock directory: %v", err) + } + lockPath := filepath.Join(lockDir, lockName) + value, _ := falOperationLocks.LoadOrStore(lockPath, newFalOperationSemaphore()) + return lockPath, value.(*falOperationSemaphore), nil +} + +func falOperationUnlock(fileLock *flock.Flock, semaphore *falOperationSemaphore) func() { + var once sync.Once + return func() { + once.Do(func() { + _ = fileLock.Unlock() + semaphore.token <- struct{}{} + }) + } +} diff --git a/internal/providers/fal/provider.go b/internal/providers/fal/provider.go new file mode 100644 index 000000000..9136fc4d9 --- /dev/null +++ b/internal/providers/fal/provider.go @@ -0,0 +1,243 @@ +package fal + +import ( + "context" + "crypto/sha256" + "flag" + "fmt" + "io" + "path" + "sort" + "strings" + "time" + + core "github.com/openclaw/crabbox/internal/cli" +) + +func init() { + core.RegisterProvider(Provider{}) +} + +type Provider struct{} + +func (Provider) Name() string { return providerName } + +func (Provider) Aliases() []string { return []string{"fal-ai"} } + +func (Provider) ProviderClaimScope(cfg core.Config) string { + endpoint := core.NormalizeProviderClaimEndpoint(cfg.Fal.APIURL) + if endpoint == "" { + return "" + } + return "endpoint:" + endpoint +} + +func (Provider) StopRoutingArgs(cfg core.Config, _ string) []string { + apiURL := strings.TrimSpace(cfg.Fal.APIURL) + if apiURL == "" { + return nil + } + return []string{"--fal-api-url", core.RoutingSafeURL(apiURL)} +} + +func (Provider) Spec() core.ProviderSpec { + return core.ProviderSpec{ + Name: providerName, + Family: providerName, + Kind: core.ProviderKindSSHLease, + Targets: []core.TargetSpec{{OS: core.TargetLinux}}, + Features: core.FeatureSet{core.FeatureSSH, core.FeatureCrabboxSync, core.FeatureCleanup}, + Coordinator: core.CoordinatorNever, + } +} + +func (Provider) RegisterFlags(fs *flag.FlagSet, defaults core.Config) any { + return RegisterFalProviderFlags(fs, defaults) +} + +func (Provider) ApplyFlags(cfg *core.Config, fs *flag.FlagSet, values any) error { + return ApplyFalProviderFlags(cfg, fs, values) +} + +func (Provider) PrepareLeaseClaimEndpoint(existing core.LeaseClaim, provider, slug string, server core.Server, _ bool) (core.Server, error) { + if !isFalProviderName(provider) { + return core.Server{}, core.Exit(2, "refusing to rewrite fal lease=%s as provider=%s", existing.LeaseID, provider) + } + if falDeletionInProgress(existing) { + return core.Server{}, core.Exit(4, "fal lease %s deletion is in progress; refusing endpoint rewrite", existing.LeaseID) + } + if slug != existing.Slug || server.Labels["lease"] != existing.LeaseID || server.Labels["slug"] != existing.Slug { + return core.Server{}, core.Exit(2, "refusing to rewrite fal lease=%s with mismatched claim identity", existing.LeaseID) + } + if existing.CloudID != "" && server.CloudID != "" && existing.CloudID != server.CloudID { + return core.Server{}, core.Exit(2, "refusing to rewrite fal lease=%s with stale instance identity", existing.LeaseID) + } + binding := strings.TrimSpace(existing.Labels[falCredentialBindingLabel]) + if binding == "" { + return core.Server{}, core.Exit(2, "fal lease %s has no credential binding; refusing endpoint rewrite", existing.LeaseID) + } + if supplied := strings.TrimSpace(server.Labels[falCredentialBindingLabel]); supplied != "" && supplied != binding { + return core.Server{}, core.Exit(2, "refusing to rewrite fal lease=%s with a different credential binding", existing.LeaseID) + } + lifetime := existing.Labels[falAcquireLifetimeLabel] + if supplied := server.Labels[falAcquireLifetimeLabel]; supplied != "" && supplied != lifetime { + return core.Server{}, core.Exit(2, "refusing to rewrite fal lease=%s with a different acquisition-lifetime marker", existing.LeaseID) + } + server.Labels = cloneLabels(server.Labels) + server.Labels[falCredentialBindingLabel] = binding + if lifetime == "" { + delete(server.Labels, falAcquireLifetimeLabel) + } else { + server.Labels[falAcquireLifetimeLabel] = lifetime + } + return server, nil +} + +func (p Provider) Configure(cfg core.Config, rt core.Runtime) (core.Backend, error) { + if err := p.ValidateConfig(cfg); err != nil { + return nil, err + } + applyFalDefaults(&cfg) + return &backend{spec: p.Spec(), cfg: cfg, rt: rt, clientFactory: newClient}, nil +} + +func (Provider) ValidateConfig(cfg core.Config) error { + for _, field := range []struct { + name string + value string + }{ + {name: "fal.apiUrl", value: cfg.Fal.APIURL}, + {name: "fal.instanceType", value: cfg.Fal.InstanceType}, + {name: "fal.user", value: cfg.Fal.User}, + {name: "fal.workRoot", value: cfg.Fal.WorkRoot}, + {name: "sshUser", value: cfg.SSHUser}, + {name: "workRoot", value: cfg.WorkRoot}, + } { + if field.value != "" && strings.TrimSpace(field.value) == "" { + return exit(2, "%s must not be blank", field.name) + } + } + applyFalDefaults(&cfg) + if cfg.TargetOS != "" && cfg.TargetOS != core.TargetLinux { + return exit(2, "provider=%s managed provisioning supports target=linux only", providerName) + } + if cfg.Tailscale.Enabled || string(cfg.Network) == "tailscale" { + return exit(2, "--tailscale is not supported for provider=%s; fal Compute exposes public SSH only", providerName) + } + if err := validateFalSSHUser(cfg.Fal.User); err != nil { + return err + } + if err := validateFalSSHUser(cfg.SSHUser); err != nil { + return err + } + if strings.TrimSpace(cfg.Fal.InstanceType) == "" { + return exit(2, "fal.instanceType must not be blank") + } + for _, field := range []struct { + name string + value string + }{ + {name: "fal.workRoot", value: cfg.Fal.WorkRoot}, + {name: "workRoot", value: cfg.WorkRoot}, + } { + if !path.IsAbs(field.value) { + return exit(2, "%s must be an absolute Linux path", field.name) + } + } + _, err := validateFalAPIURL(cfg.Fal.APIURL) + return err +} + +func (p Provider) ConfigureDoctor(cfg core.Config, rt core.Runtime) (core.DoctorBackend, error) { + backend, err := p.Configure(cfg, rt) + if err != nil { + return nil, err + } + doctor, ok := backend.(core.DoctorBackend) + if !ok { + return nil, core.Exit(2, "fal doctor backend unavailable") + } + return doctor, nil +} + +type backend struct { + spec ProviderSpec + cfg Config + rt Runtime + clientFactory func(Config, Runtime) (computeAPI, error) + persistCreateIntent func(string, string, Config, string, bool, time.Time, CreateInstanceRequest) (core.LeaseClaim, error) + persistRollbackClaim func(string, string, Config, string, string, string, bool) (core.LeaseClaim, error) + recoveryClaimReplacement func(core.LeaseClaim, Config, string, string, bool) (core.LeaseClaim, error) + removeLeaseKey func(string) error + ensureCreateKey func(Config, string) (string, string, error) + waitSSH func(context.Context, *core.SSHTarget, string, time.Duration) error + pollInterval time.Duration + pollTimeout time.Duration +} + +func (b *backend) Spec() ProviderSpec { return b.spec } + +func (b *backend) Doctor(ctx context.Context, _ DoctorRequest) (DoctorResult, error) { + if strings.TrimSpace(b.cfg.Fal.APIKey) == "" { + return DoctorResult{}, exit(2, "provider=%s requires fal credentials in environment", providerName) + } + client, err := b.clientFactory(b.cfg, b.rt) + if err != nil { + return DoctorResult{}, err + } + count, fingerprint, err := falInventoryFingerprint(ctx, client) + if err != nil { + return DoctorResult{}, exit(1, "fal auth check failed: %v", err) + } + return DoctorResult{ + Provider: providerName, + Message: fmt.Sprintf("auth=ready control_plane=ready inventory=ready inventory_count=%d inventory_fingerprint=%s api=list mutation=false runtime=unchecked", count, fingerprint), + }, nil +} + +func falInventoryFingerprint(ctx context.Context, client computeAPI) (int, string, error) { + ids, err := falInventoryIDs(ctx, client) + if err != nil { + return 0, "", err + } + sum := sha256.Sum256([]byte(strings.Join(ids, "\x00"))) + return len(ids), fmt.Sprintf("%x", sum), nil +} + +func falInventoryIDs(ctx context.Context, client computeAPI) ([]string, error) { + const maxPages = 100 + ids := make([]string, 0) + cursor := "" + seenCursors := map[string]struct{}{} + for page := 0; page < maxPages; page++ { + result, err := client.ListInstances(ctx, 100, cursor) + if err != nil { + return nil, err + } + for _, instance := range result.Instances { + id := strings.TrimSpace(instance.ID) + if id == "" { + return nil, exit(5, "fal inventory returned an instance without an id") + } + ids = append(ids, id) + } + if !result.HasMore { + sort.Strings(ids) + return ids, nil + } + if result.NextCursor == nil || strings.TrimSpace(*result.NextCursor) == "" { + return nil, exit(5, "fal inventory pagination omitted the next cursor") + } + next := strings.TrimSpace(*result.NextCursor) + if _, exists := seenCursors[next]; exists { + return nil, exit(5, "fal inventory pagination repeated a cursor") + } + seenCursors[next] = struct{}{} + cursor = next + } + return nil, exit(5, "fal inventory pagination exceeded %d pages", maxPages) +} + +func newDiscardRuntime() Runtime { + return Runtime{Stdout: io.Discard, Stderr: io.Discard} +} diff --git a/internal/providers/fal/provider_test.go b/internal/providers/fal/provider_test.go new file mode 100644 index 000000000..b7c81f818 --- /dev/null +++ b/internal/providers/fal/provider_test.go @@ -0,0 +1,393 @@ +package fal + +import ( + "context" + "flag" + "io" + "strings" + "testing" + "time" + + core "github.com/openclaw/crabbox/internal/cli" +) + +func TestProviderSpecAndAlias(t *testing.T) { + spec := Provider{}.Spec() + if spec.Name != providerName || spec.Family != providerName || spec.Kind != core.ProviderKindSSHLease || spec.Coordinator != core.CoordinatorNever { + t.Fatalf("unexpected spec: %#v", spec) + } + if len(spec.Targets) != 1 || spec.Targets[0].OS != core.TargetLinux { + t.Fatalf("targets=%#v, want linux only", spec.Targets) + } + for _, feature := range []core.Feature{core.FeatureSSH, core.FeatureCrabboxSync, core.FeatureCleanup} { + if !spec.Features.Has(feature) { + t.Fatalf("features=%v missing %s", spec.Features, feature) + } + } + aliases := Provider{}.Aliases() + if len(aliases) != 1 || aliases[0] != "fal-ai" { + t.Fatalf("aliases=%#v, want [fal-ai]", aliases) + } +} + +func TestProviderOwnsClaimScopeAndStopRouting(t *testing.T) { + p := Provider{} + cfg := core.BaseConfig() + cfg.Fal.APIURL = "https://user@API.EXAMPLE.TEST:443/v1/?region=west#section" + + if got, want := p.ProviderClaimScope(cfg), "endpoint:https://api.example.test/v1"; got != want { + t.Fatalf("claim scope=%q, want %q", got, want) + } + args := p.StopRoutingArgs(cfg, "lease-1") + if got, want := strings.Join(args, " "), "--fal-api-url https://API.EXAMPLE.TEST:443/v1/?region=west#section"; got != want { + t.Fatalf("stop routing args=%q, want %q", got, want) + } +} + +func TestPrepareLeaseClaimEndpointPreservesCredentialBinding(t *testing.T) { + existing := core.LeaseClaim{ + LeaseID: "cbx_abcdef123456", + Slug: "gpu-box", + Provider: providerName, + CloudID: "inst_owned", + Labels: map[string]string{ + falCredentialBindingLabel: "binding-a", + falAcquireLifetimeLabel: falAcquireLifetimeVersion, + }, + } + server := core.Server{ + CloudID: "inst_owned", + Provider: providerName, + Labels: map[string]string{ + "lease": existing.LeaseID, + "slug": existing.Slug, + }, + } + prepared, err := (Provider{}).PrepareLeaseClaimEndpoint(existing, providerName, existing.Slug, server, false) + if err != nil { + t.Fatal(err) + } + if prepared.Labels[falCredentialBindingLabel] != "binding-a" { + t.Fatalf("prepared labels=%#v", prepared.Labels) + } + if prepared.Labels[falAcquireLifetimeLabel] != falAcquireLifetimeVersion { + t.Fatalf("prepared lifetime marker=%q", prepared.Labels[falAcquireLifetimeLabel]) + } + changed := server + changed.Labels = cloneLabels(server.Labels) + changed.Labels[falCredentialBindingLabel] = "binding-b" + if _, err := (Provider{}).PrepareLeaseClaimEndpoint(existing, providerName, existing.Slug, changed, false); err == nil { + t.Fatal("expected credential-binding retarget rejection") + } + changed = server + changed.Labels = cloneLabels(server.Labels) + changed.Labels[falAcquireLifetimeLabel] = "2" + if _, err := (Provider{}).PrepareLeaseClaimEndpoint(existing, providerName, existing.Slug, changed, false); err == nil { + t.Fatal("expected lifetime-marker retarget rejection") + } + + legacy := existing + legacy.Labels = cloneLabels(existing.Labels) + delete(legacy.Labels, falAcquireLifetimeLabel) + supplied := server + supplied.Labels = cloneLabels(server.Labels) + supplied.Labels[falAcquireLifetimeLabel] = falAcquireLifetimeVersion + if _, err := (Provider{}).PrepareLeaseClaimEndpoint(legacy, providerName, legacy.Slug, supplied, false); err == nil { + t.Fatal("expected legacy lifetime-marker upgrade rejection") + } + prepared, err = (Provider{}).PrepareLeaseClaimEndpoint(legacy, providerName, legacy.Slug, server, false) + if err != nil { + t.Fatal(err) + } + if _, ok := prepared.Labels[falAcquireLifetimeLabel]; ok { + t.Fatalf("legacy endpoint rewrite added lifetime marker: %#v", prepared.Labels) + } + malformed := existing + malformed.Labels = cloneLabels(existing.Labels) + malformed.Labels[falAcquireLifetimeLabel] = " 1 " + prepared, err = (Provider{}).PrepareLeaseClaimEndpoint(malformed, providerName, malformed.Slug, server, false) + if err != nil { + t.Fatal(err) + } + if prepared.Labels[falAcquireLifetimeLabel] != " 1 " { + t.Fatalf("malformed existing lifetime marker was normalized: %#v", prepared.Labels) + } +} + +func TestIsFalProviderNameAcceptsAlias(t *testing.T) { + for _, name := range []string{"fal", "FAL", " fal-ai "} { + if !isFalProviderName(name) { + t.Fatalf("isFalProviderName(%q) = false, want true", name) + } + } + for _, name := range []string{"", "falai", "runpod"} { + if isFalProviderName(name) { + t.Fatalf("isFalProviderName(%q) = true, want false", name) + } + } +} + +func TestFalTokenFlagIsNotRegistered(t *testing.T) { + cfg := Config{} + cfg.Fal.APIKey = "secret-key" + fs := flag.NewFlagSet("test", flag.ContinueOnError) + Provider{}.RegisterFlags(fs, cfg) + for _, name := range []string{"fal-key", "fal-api-key", "fal-token", "fal-api-token"} { + if fs.Lookup(name) != nil { + t.Fatalf("fal API key surfaced as a flag --%s", name) + } + } + for _, name := range []string{"fal-api-url", "fal-instance-type", "fal-sector", "fal-user", "fal-work-root"} { + if fs.Lookup(name) == nil { + t.Fatalf("%s flag missing", name) + } + } +} + +func TestFalFlagsApplyNonSecretConfig(t *testing.T) { + fs := flag.NewFlagSet("test", flag.ContinueOnError) + fs.SetOutput(io.Discard) + values := Provider{}.RegisterFlags(fs, Config{}) + if err := fs.Parse([]string{ + "--fal-api-url", "https://api.example.test/v1", + "--fal-instance-type", string(InstanceTypeH100x8), + "--fal-sector", string(Sector3), + "--fal-user", "ubuntu", + "--fal-work-root", "/srv/crabbox", + }); err != nil { + t.Fatal(err) + } + cfg := Config{Provider: providerName, SSHUser: "old-user", WorkRoot: "/old/work-root"} + core.MarkSSHUserExplicit(&cfg) + core.MarkWorkRootExplicit(&cfg) + if err := (Provider{}).ApplyFlags(&cfg, fs, values); err != nil { + t.Fatal(err) + } + if cfg.Fal.APIURL != "https://api.example.test/v1" || + cfg.Fal.InstanceType != string(InstanceTypeH100x8) || + cfg.Fal.Sector != string(Sector3) || + cfg.Fal.User != "ubuntu" || + cfg.Fal.WorkRoot != "/srv/crabbox" { + t.Fatalf("fal flags not applied: %#v", cfg.Fal) + } + if cfg.SSHUser != "ubuntu" || cfg.WorkRoot != "/srv/crabbox" { + t.Fatalf("effective ssh config user=%q workRoot=%q", cfg.SSHUser, cfg.WorkRoot) + } + if cfg.Fal.APIKey != "" { + t.Fatalf("fal API key should stay env-only, got %q", cfg.Fal.APIKey) + } +} + +func TestConfigureRejectsUnsupportedTargetAndTailscale(t *testing.T) { + for name, cfg := range map[string]Config{ + "macos target": {TargetOS: "macos"}, + "tailscale": {TargetOS: targetLinux, Tailscale: core.TailscaleConfig{Enabled: true}}, + "network": {TargetOS: targetLinux, Network: "tailscale"}, + } { + t.Run(name, func(t *testing.T) { + _, err := Provider{}.Configure(cfg, newDiscardRuntime()) + if err == nil { + t.Fatal("expected error") + } + }) + } +} + +func TestValidateConfigRejectsUnsafeNonRuntimeConfig(t *testing.T) { + for name, cfg := range map[string]Config{ + "macos target": {TargetOS: "macos"}, + "tailscale": {TargetOS: targetLinux, Tailscale: core.TailscaleConfig{Enabled: true}}, + "ssh user": {TargetOS: targetLinux, Fal: FalConfig{User: "bad user"}}, + "blank fal user": {TargetOS: targetLinux, Fal: FalConfig{User: " "}}, + "blank ssh user": {TargetOS: targetLinux, SSHUser: " "}, + "api url": {TargetOS: targetLinux, Fal: FalConfig{APIURL: "http://api.fal.ai/v1"}}, + "blank api url": {TargetOS: targetLinux, Fal: FalConfig{APIURL: " "}}, + "instance type": {TargetOS: targetLinux, Fal: FalConfig{InstanceType: " "}}, + "fal work root": {TargetOS: targetLinux, Fal: FalConfig{WorkRoot: "relative/work"}}, + "blank fal work root": {TargetOS: targetLinux, Fal: FalConfig{WorkRoot: " "}}, + "work root": {TargetOS: targetLinux, WorkRoot: " "}, + } { + t.Run(name, func(t *testing.T) { + if err := (Provider{}).ValidateConfig(cfg); err == nil { + t.Fatal("expected validation error") + } + }) + } + if err := (Provider{}).ValidateConfig(Config{}); err != nil { + t.Fatalf("default config validation failed: %v", err) + } +} + +func TestConfigureRejectsUnsafeSSHUsers(t *testing.T) { + for _, user := range []string{ + "-oProxyCommand=sh", + "user@host", + "user name", + "user;touch-pwned", + strings.Repeat("a", 33), + } { + t.Run(user, func(t *testing.T) { + _, err := (Provider{}).Configure(Config{ + TargetOS: targetLinux, + Fal: FalConfig{ + APIKey: "test-key", + User: user, + }, + }, newDiscardRuntime()) + if err == nil || !strings.Contains(err.Error(), "valid Linux login name") { + t.Fatalf("user=%q err=%v", user, err) + } + }) + } +} + +func TestConfigureNormalizesSSHUsersBeforeValidationAndUse(t *testing.T) { + got, err := (Provider{}).Configure(Config{ + TargetOS: targetLinux, + SSHUser: " root ", + Fal: FalConfig{ + APIKey: "test-key", + User: " ubuntu ", + }, + }, newDiscardRuntime()) + if err != nil { + t.Fatal(err) + } + cfg := got.(*backend).cfg + if cfg.Fal.User != "ubuntu" || cfg.SSHUser != "root" { + t.Fatalf("normalized users fal=%q ssh=%q", cfg.Fal.User, cfg.SSHUser) + } +} + +func TestConfigureReturnsSSHLeaseBackend(t *testing.T) { + gotBackend, err := Provider{}.Configure(Config{TargetOS: targetLinux}, newDiscardRuntime()) + if err != nil { + t.Fatal(err) + } + sshBackend, ok := gotBackend.(core.SSHLeaseBackend) + if !ok { + t.Fatalf("backend %T does not implement SSHLeaseBackend", gotBackend) + } + cleanup, ok := gotBackend.(core.CleanupBackend) + if !ok { + t.Fatalf("backend %T does not implement CleanupBackend", gotBackend) + } + _ = sshBackend + _ = cleanup + if got := gotBackend.(*backend).Spec(); got.Name != providerName { + t.Fatalf("backend spec=%#v", got) + } + configured := gotBackend.(*backend).cfg + if configured.SSHUser != defaultUser || configured.Fal.User != defaultUser || configured.WorkRoot != defaultWorkRoot || configured.Fal.WorkRoot != defaultWorkRoot { + t.Fatalf("default ssh config=%#v", configured) + } + if _, err := (Provider{}).ConfigureDoctor(Config{TargetOS: targetLinux}, newDiscardRuntime()); err != nil { + t.Fatalf("ConfigureDoctor: %v", err) + } +} + +func TestBackendTimingAndClaimTargetHelpers(t *testing.T) { + b := &backend{pollInterval: 25 * time.Millisecond} + if got := b.effectivePollInterval(); got != 25*time.Millisecond { + t.Fatalf("poll interval=%s", got) + } + b.pollInterval = 0 + if got := b.effectivePollInterval(); got != falPollInterval { + t.Fatalf("default poll interval=%s", got) + } + + t.Setenv("XDG_STATE_HOME", t.TempDir()) + claim := core.LeaseClaim{ + LeaseID: "falbx_helper", + Slug: "helper", + Provider: providerName, + CloudID: "inst_helper", + SSHHost: "192.0.2.10", + SSHPort: 2222, + Labels: map[string]string{ + "name": "helper", + "state": "ready", + "server_type": string(InstanceTypeH100x1), + }, + } + target, err := leaseTargetFromClaim(claim, Config{SSHUser: "root"}, true) + if err != nil { + t.Fatal(err) + } + if target.LeaseID != claim.LeaseID || target.Server.CloudID != claim.CloudID || target.SSH.Host != claim.SSHHost || target.SSH.Port != "2222" { + t.Fatalf("target=%#v", target) + } + claim.Provider = "runpod" + if _, err := leaseTargetFromClaim(claim, Config{}, false); err == nil { + t.Fatal("expected provider mismatch") + } +} + +func TestDoctorReportsMissingAuthWithoutTokenNames(t *testing.T) { + gotBackend, err := Provider{}.Configure(Config{TargetOS: targetLinux}, newDiscardRuntime()) + if err != nil { + t.Fatal(err) + } + _, err = gotBackend.(*backend).Doctor(t.Context(), DoctorRequest{}) + if err == nil { + t.Fatal("expected missing auth error") + } + message := err.Error() + if !strings.Contains(message, "requires fal credentials in environment") { + t.Fatalf("missing auth error=%q", message) + } + for _, forbidden := range []string{"FAL_KEY", "CRABBOX_FAL_KEY", "secret"} { + if strings.Contains(message, forbidden) { + t.Fatalf("missing auth error leaked %q: %q", forbidden, message) + } + } +} + +func TestDoctorReadyMessageReportsCompleteInventoryFingerprint(t *testing.T) { + gotBackend, err := Provider{}.Configure(Config{ + TargetOS: targetLinux, + Fal: FalConfig{APIKey: "test-key", APIURL: "https://api.example.test/v1"}, + }, newDiscardRuntime()) + if err != nil { + t.Fatal(err) + } + backend := gotBackend.(*backend) + backend.clientFactory = func(Config, Runtime) (computeAPI, error) { + return stubComputeAPI{list: ListInstancesResponse{ + Instances: []ComputeInstance{{ID: "inst_2"}, {ID: "inst_1"}}, + }}, nil + } + result, err := backend.Doctor(t.Context(), DoctorRequest{}) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(result.Message, "auth=ready") || !strings.Contains(result.Message, "api=list") || !strings.Contains(result.Message, "inventory_count=2") { + t.Fatalf("doctor message missing readiness details: %q", result.Message) + } + for _, field := range strings.Fields(result.Message) { + if fingerprint, ok := strings.CutPrefix(field, "inventory_fingerprint="); ok && len(fingerprint) == 64 { + return + } + } + t.Fatalf("doctor message missing inventory fingerprint: %q", result.Message) +} + +type stubComputeAPI struct { + list ListInstancesResponse +} + +func (s stubComputeAPI) ListInstances(_ context.Context, _ int, _ string) (ListInstancesResponse, error) { + return s.list, nil +} + +func (stubComputeAPI) GetInstance(context.Context, string) (ComputeInstance, error) { + return ComputeInstance{}, nil +} + +func (stubComputeAPI) CreateInstance(context.Context, CreateInstanceRequest, string) (ComputeInstance, error) { + return ComputeInstance{}, nil +} + +func (stubComputeAPI) DeleteInstance(context.Context, string) error { + return nil +} diff --git a/internal/providers/fal/types.go b/internal/providers/fal/types.go new file mode 100644 index 000000000..5a273f9ec --- /dev/null +++ b/internal/providers/fal/types.go @@ -0,0 +1,124 @@ +package fal + +import ( + "bytes" + "encoding/json" + "fmt" +) + +type InstanceStatus string + +const ( + InstanceStatusReady InstanceStatus = "ready" + InstanceStatusInit InstanceStatus = "init" + InstanceStatusPending InstanceStatus = "pending" + InstanceStatusProvisioning InstanceStatus = "provisioning" + InstanceStatusStopped InstanceStatus = "stopped" + InstanceStatusUnknown InstanceStatus = "unknown" +) + +func (s InstanceStatus) Known() bool { + switch s { + case InstanceStatusReady, InstanceStatusInit, InstanceStatusPending, InstanceStatusProvisioning, InstanceStatusStopped, InstanceStatusUnknown: + return true + default: + return false + } +} + +type InstanceType string + +const ( + InstanceTypeH100x1 InstanceType = "gpu_1x_h100_sxm5" + InstanceTypeH100x8 InstanceType = "gpu_8x_h100_sxm5" +) + +type Sector string + +const ( + Sector1 Sector = "sector_1" + Sector2 Sector = "sector_2" + Sector3 Sector = "sector_3" +) + +type ComputeInstance struct { + ID string `json:"id"` + InstanceType InstanceType `json:"instance_type"` + Region string `json:"region"` + Sector Sector `json:"sector,omitempty"` + IP string `json:"ip,omitempty"` + Status InstanceStatus `json:"status"` + CreatorUserNickname string `json:"creator_user_nickname,omitempty"` +} + +type ListInstancesResponse struct { + NextCursor *string `json:"next_cursor"` + HasMore bool `json:"has_more"` + Instances []ComputeInstance `json:"instances"` +} + +func (r *ListInstancesResponse) UnmarshalJSON(data []byte) error { + var wire struct { + NextCursor json.RawMessage `json:"next_cursor"` + HasMore json.RawMessage `json:"has_more"` + Instances json.RawMessage `json:"instances"` + } + if err := json.Unmarshal(data, &wire); err != nil { + return err + } + for _, field := range []struct { + name string + raw json.RawMessage + }{ + {name: "next_cursor", raw: wire.NextCursor}, + {name: "has_more", raw: wire.HasMore}, + {name: "instances", raw: wire.Instances}, + } { + if len(field.raw) == 0 { + return fmt.Errorf("fal list response missing required field %q", field.name) + } + } + + var nextCursor *string + if !bytes.Equal(bytes.TrimSpace(wire.NextCursor), []byte("null")) { + var value string + if err := json.Unmarshal(wire.NextCursor, &value); err != nil { + return fmt.Errorf("fal list response field %q: %w", "next_cursor", err) + } + nextCursor = &value + } + var hasMore bool + if bytes.Equal(bytes.TrimSpace(wire.HasMore), []byte("null")) { + return fmt.Errorf("fal list response field %q must not be null", "has_more") + } + if err := json.Unmarshal(wire.HasMore, &hasMore); err != nil { + return fmt.Errorf("fal list response field %q: %w", "has_more", err) + } + if bytes.Equal(bytes.TrimSpace(wire.Instances), []byte("null")) { + return fmt.Errorf("fal list response field %q must not be null", "instances") + } + var instances []ComputeInstance + if err := json.Unmarshal(wire.Instances, &instances); err != nil { + return fmt.Errorf("fal list response field %q: %w", "instances", err) + } + + *r = ListInstancesResponse{NextCursor: nextCursor, HasMore: hasMore, Instances: instances} + return nil +} + +type CreateInstanceRequest struct { + InstanceType InstanceType `json:"instance_type"` + SSHKey string `json:"ssh_key"` + Sector Sector `json:"sector,omitempty"` +} + +type APIErrorBody struct { + Error APIErrorDetail `json:"error"` +} + +type APIErrorDetail struct { + Type string `json:"type"` + Message string `json:"message"` + DocsURL string `json:"docs_url,omitempty"` + RequestID string `json:"request_id,omitempty"` +} diff --git a/scripts/live-digitalocean-smoke.test.js b/scripts/live-digitalocean-smoke.test.js index b3a99b8c5..f3fd957d1 100644 --- a/scripts/live-digitalocean-smoke.test.js +++ b/scripts/live-digitalocean-smoke.test.js @@ -6,6 +6,9 @@ import { spawnSync } from "node:child_process"; import test from "node:test"; const repoRoot = path.resolve(import.meta.dirname, ".."); +const isolatedHome = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-do-home-")); +process.env.HOME = isolatedHome; +process.env.XDG_CONFIG_HOME = path.join(isolatedHome, ".config"); function writeExecutable(file, body) { fs.writeFileSync(file, body, "utf8"); diff --git a/scripts/live-fal-smoke.sh b/scripts/live-fal-smoke.sh new file mode 100755 index 000000000..bc5afd586 --- /dev/null +++ b/scripts/live-fal-smoke.sh @@ -0,0 +1,508 @@ +#!/usr/bin/env bash +set -euo pipefail + +provider_enabled() { + local list="${CRABBOX_LIVE_PROVIDERS:-fal}" + local item + IFS=',' read -ra items <<<"$list" + for item in "${items[@]}"; do + item="${item//[[:space:]]/}" + if [[ "$item" == "fal" || "$item" == "fal-ai" ]]; then + return 0 + fi + done + return 1 +} + +redact_output() { + FAL_SMOKE_REDACT_PRIMARY="${CRABBOX_FAL_KEY:-}" FAL_SMOKE_REDACT_FALLBACK="${FAL_KEY:-}" python3 -c ' +import os +import re +import sys + +body = sys.stdin.read() +for token in (os.environ.get("FAL_SMOKE_REDACT_PRIMARY", ""), os.environ.get("FAL_SMOKE_REDACT_FALLBACK", "")): + if token: + body = body.replace(token, "") +for field in ("token", "api_key", "apiKey", "fal_key", "falKey", "ssh_key", "sshKey", "public_key", "publicKey", "private_key", "privateKey"): + body = re.sub(rf"(\"{re.escape(field)}\"\s*:\s*\")[^\"]*(\")", rf"\1\2", body, flags=re.IGNORECASE) + body = re.sub(rf"({re.escape(field)}\s*[=: ]\s*)[^\",\s]+", rf"\1", body, flags=re.IGNORECASE) +body = re.sub(r"-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z ]*PRIVATE KEY-----", "", body) +body = re.sub(r"""https?://[^\s"]*(?:token|api_key|key)=[^\s"]+""", "", body, flags=re.IGNORECASE) +sys.stdout.write(body) +' +} + +classify_known_external_blocker() { + local command="$1" + local exit_status="$2" + local output="$3" + local classification="" + local lower + lower="$(printf '%s' "$output" | tr '[:upper:]' '[:lower:]')" + if [[ "$lower" == *billing* || + "$lower" == *"payment"* || + "$lower" == *credit*balance* || + "$lower" == *"automated top up"* || + "$lower" == *"automated top-up"* || + "$lower" == *"insufficient funds"* || + "$lower" == *"account-inactive"* || + "$lower" == *"account inactive"* || + "$lower" == *"inactive account"* ]]; then + classification="billing_blocked" + elif [[ "$lower" == *quota* || "$lower" == *"rate limit"* || "$lower" == *"too many requests"* || "$lower" == *"account limit"* || "$lower" == *"resource exhausted"* ]]; then + classification="quota_blocked" + elif [[ "$lower" == *capacity* || "$lower" == *"no instances available"* ]]; then + classification="capacity_blocked" + else + return 1 + fi + printf 'classification=%s command=%q exit=%s\n' "$classification" "$command" "$exit_status" >&2 + printf '%s\n' "$output" | redact_output >&2 + return 0 +} + +classify_known_preflight_auth_blocker() { + local command="$1" + local exit_status="$2" + local output="$3" + local lower + lower="$(printf '%s' "$output" | tr '[:upper:]' '[:lower:]')" + if [[ "$lower" != *"fal api"* ]] || + [[ "$lower" != *"authorization_error:"* && "$lower" != *unauthorized* && "$lower" != *forbidden* && "$lower" != *"invalid api key"* && "$lower" != *"invalid key"* ]]; then + return 1 + fi + printf 'classification=environment_blocked command=%q exit=%s reason=fal_api_auth\n' "$command" "$exit_status" >&2 + printf '%s\n' "$output" | redact_output >&2 + return 0 +} + +classify_validation_failure() { + local command="$1" + local exit_status="$2" + local output="$3" + printf 'classification=validation_failed command=%q exit=%s\n' "$command" "$exit_status" >&2 + printf '%s\n' "$output" | redact_output >&2 +} + +run_capture() { + local command="$1" + shift + local output + set +e + output="$("$@" 2>&1)" + local exit_status=$? + set -e + if [ "$exit_status" -ne 0 ]; then + if classify_known_external_blocker "$command" "$exit_status" "$output"; then + exit 0 + fi + classify_validation_failure "$command" "$exit_status" "$output" + exit "$exit_status" + fi + CAPTURED_OUTPUT="$(printf '%s\n' "$output" | redact_output)" +} + +run_capture_preflight() { + local command="$1" + shift + local output + set +e + output="$("$@" 2>&1)" + local exit_status=$? + set -e + if [ "$exit_status" -ne 0 ]; then + if classify_known_external_blocker "$command" "$exit_status" "$output" || + classify_known_preflight_auth_blocker "$command" "$exit_status" "$output"; then + exit 0 + fi + classify_validation_failure "$command" "$exit_status" "$output" + exit "$exit_status" + fi + CAPTURED_OUTPUT="$(printf '%s\n' "$output" | redact_output)" +} + +run_capture_validation() { + local command="$1" + shift + local output + set +e + output="$("$@" 2>&1)" + local exit_status=$? + set -e + if [ "$exit_status" -ne 0 ]; then + classify_validation_failure "$command" "$exit_status" "$output" + exit "$exit_status" + fi + CAPTURED_OUTPUT="$(printf '%s\n' "$output" | redact_output)" +} + +validate_list_json_contains_slug() { + local command="$1" + local output="$2" + local validation_output="" + local exit_status=0 + set +e + validation_output="$(CRABBOX_SMOKE_SLUG="$slug" python3 -c ' +import json +import os +import sys + +slug = os.environ["CRABBOX_SMOKE_SLUG"] +try: + payload = json.load(sys.stdin) +except Exception as exc: + print(f"invalid JSON: {exc}", file=sys.stderr) + sys.exit(1) + +def has_slug(value): + if isinstance(value, dict): + labels = value.get("labels") or value.get("tags") + if isinstance(labels, dict) and (labels.get("slug") == slug or labels.get("crabbox.slug") == slug): + return True + if value.get("slug") == slug or value.get("name") == slug or value.get("id") == slug or value.get("leaseId") == slug: + return True + return any(has_slug(child) for child in value.values()) + if isinstance(value, list): + return any(has_slug(child) for child in value) + return False + +if not has_slug(payload): + print(f"list JSON did not include slug {slug}", file=sys.stderr) + sys.exit(1) +' <<<"$output" 2>&1)" + exit_status=$? + set -e + if [ "$exit_status" -ne 0 ]; then + classify_validation_failure "$command" "$exit_status" "$validation_output" + exit "$exit_status" + fi +} + +validate_list_json_empty() { + local command="$1" + local output="$2" + local validation_output="" + local exit_status=0 + set +e + validation_output="$(python3 -c ' +import json +import sys + +try: + payload = json.load(sys.stdin) +except Exception as exc: + print(f"invalid JSON: {exc}", file=sys.stderr) + sys.exit(1) + +if payload != []: + print("fal Crabbox inventory is not empty", file=sys.stderr) + sys.exit(1) +' <<<"$output" 2>&1)" + exit_status=$? + set -e + if [ "$exit_status" -ne 0 ]; then + classify_validation_failure "$command" "$exit_status" "$validation_output" + exit "$exit_status" + fi +} + +validate_output_line() { + local command="$1" + local output="$2" + local expected="$3" + if ! grep -Fqx -- "$expected" <<<"$output"; then + classify_validation_failure "$command" 1 "expected output line: $expected" + exit 1 + fi +} + +is_fal_not_found_output() { + local output="$1" + [[ "$output" == *"lease/fal instance not found"* || "$output" == *"no local claim for fal lease"* || "$output" == *"not locally claimed"* ]] +} + +inventory_snapshot_from_doctor() { + local output="$1" + local count="" + local fingerprint="" + local field + for field in $output; do + case "$field" in + inventory_count=*) count="${field#inventory_count=}" ;; + inventory_fingerprint=*) fingerprint="${field#inventory_fingerprint=}" ;; + esac + done + if [[ ! "$count" =~ ^[0-9]+$ || ! "$fingerprint" =~ ^[0-9a-f]{64}$ ]]; then + return 1 + fi + printf '%s:%s\n' "$count" "$fingerprint" +} + +wait_for_provider_inventory_baseline() { + local attempts="${CRABBOX_LIVE_FAL_INVENTORY_POLL_ATTEMPTS:-65}" + local poll_seconds="${CRABBOX_LIVE_FAL_INVENTORY_POLL_SECONDS:-2}" + local required_matches="${1:-1}" + local minimum_elapsed_seconds="${2:-0}" + local started_at="$SECONDS" + local attempt + local matching=0 + local inventory_output="" + local inventory_status=1 + local current_provider_inventory="" + if [[ ! "$attempts" =~ ^[1-9][0-9]*$ ]]; then + attempts=65 + fi + if [[ ! "$poll_seconds" =~ ^[0-9]+([.][0-9]+)?$ ]]; then + poll_seconds=2 + fi + if [[ ! "$required_matches" =~ ^[1-9][0-9]*$ ]]; then + required_matches=1 + fi + if [[ ! "$minimum_elapsed_seconds" =~ ^[0-9]+$ ]]; then + minimum_elapsed_seconds=0 + fi + if [ "$attempts" -lt "$required_matches" ]; then + attempts="$required_matches" + fi + INVENTORY_VERIFY_OUTPUT="" + INVENTORY_VERIFY_STATUS=1 + for ((attempt = 1; attempt <= attempts; attempt++)); do + set +e + inventory_output="$(bin/crabbox doctor --provider fal 2>&1)" + inventory_status=$? + set -e + current_provider_inventory="" + if [ "$inventory_status" -eq 0 ]; then + current_provider_inventory="$(inventory_snapshot_from_doctor "$inventory_output" || true)" + fi + if [ "$inventory_status" -eq 0 ] && [ -n "$initial_provider_inventory" ] && [ "$current_provider_inventory" = "$initial_provider_inventory" ]; then + matching=$((matching + 1)) + if [ "$matching" -ge "$required_matches" ] && [ "$((SECONDS - started_at))" -ge "$minimum_elapsed_seconds" ]; then + INVENTORY_VERIFY_OUTPUT="$inventory_output" + INVENTORY_VERIFY_STATUS=0 + return 0 + fi + else + matching=0 + fi + if [ "$inventory_status" -eq 0 ]; then + inventory_status=1 + fi + INVENTORY_VERIFY_STATUS="$inventory_status" + INVENTORY_VERIFY_OUTPUT="provider inventory did not return to the pre-create baseline; expected=$initial_provider_inventory current=${current_provider_inventory:-unavailable}" + if [ -n "$inventory_output" ]; then + INVENTORY_VERIFY_OUTPUT+=$'\n'"$inventory_output" + fi + if [ "$attempt" -lt "$attempts" ]; then + sleep "$poll_seconds" + fi + done + return 1 +} + +verify_local_cleanup() { + local list_output="" + local list_status=1 + local validation_output="" + local current_local_key_inventory="" + LOCAL_VERIFY_OUTPUT="" + LOCAL_VERIFY_STATUS=1 + + set +e + list_output="$(bin/crabbox list --provider fal --json 2>&1)" + list_status=$? + set -e + if [ "$list_status" -ne 0 ]; then + LOCAL_VERIFY_STATUS="$list_status" + LOCAL_VERIFY_OUTPUT="$list_output" + return 1 + fi + set +e + validation_output="$(python3 -c ' +import json +import sys + +payload = json.load(sys.stdin) +if payload != []: + print("fal Crabbox inventory is not empty", file=sys.stderr) + sys.exit(1) +' <<<"$list_output" 2>&1)" + list_status=$? + set -e + if [ "$list_status" -ne 0 ]; then + LOCAL_VERIFY_STATUS="$list_status" + LOCAL_VERIFY_OUTPUT="$validation_output" + return 1 + fi + current_local_key_inventory="$(local_key_inventory_snapshot)" + if [ "$current_local_key_inventory" != "$initial_local_key_inventory" ]; then + LOCAL_VERIFY_OUTPUT="local fal lifecycle changed the per-lease key inventory under $local_keys_dir" + return 1 + fi + LOCAL_VERIFY_STATUS=0 + LOCAL_VERIFY_OUTPUT="$list_output" +} + +local_key_inventory_snapshot() { + if [ ! -d "$local_keys_dir" ]; then + return 0 + fi + find "$local_keys_dir" -type f -print | LC_ALL=C sort +} + +cleanup_armed=0 +slug="fal-smoke-$(date +%Y%m%d%H%M%S)-$$" +config_file="" +fal_key="${CRABBOX_FAL_KEY:-${FAL_KEY:-}}" +fal_instance_type="${CRABBOX_LIVE_FAL_INSTANCE_TYPE:-gpu_1x_h100_sxm5}" +fal_sector="${CRABBOX_LIVE_FAL_SECTOR:-}" +fal_api_url="${CRABBOX_LIVE_FAL_API_URL:-https://api.fal.ai/v1}" +initial_provider_inventory="" +initial_local_key_inventory="" +if [[ "$(uname -s)" == "Darwin" ]]; then + local_keys_dir="$HOME/Library/Application Support/crabbox/testboxes" +elif [ -n "${XDG_CONFIG_HOME:-}" ]; then + local_keys_dir="$XDG_CONFIG_HOME/crabbox/testboxes" +else + local_keys_dir="$HOME/.config/crabbox/testboxes" +fi + +cleanup() { + local exit_status=$? + if [ "$cleanup_armed" -eq 1 ]; then + local cleanup_output="" + local cleanup_status=1 + local attempt + local cleanup_attempts=65 + local cleanup_poll_seconds=2 + local ambiguous_matches="${CRABBOX_LIVE_FAL_AMBIGUOUS_BASELINE_OBSERVATIONS:-271}" + local ambiguous_minimum_seconds=540 + if [[ ! "$ambiguous_matches" =~ ^[1-9][0-9]*$ ]]; then + ambiguous_matches=271 + fi + if [ "${CRABBOX_LIVE_FAL_TEST_ONLY_SHORT_RECOVERY:-}" = "1" ]; then + ambiguous_minimum_seconds=0 + elif [ "$ambiguous_matches" -lt 271 ]; then + ambiguous_matches=271 + fi + for ((attempt = 1; attempt <= cleanup_attempts; attempt++)); do + set +e + cleanup_output="$(bin/crabbox stop --provider fal "$slug" 2>&1)" + cleanup_status=$? + set -e + if [ "$cleanup_status" -eq 0 ]; then + if wait_for_provider_inventory_baseline && verify_local_cleanup; then + cleanup_armed=0 + else + cleanup_status="${LOCAL_VERIFY_STATUS:-$INVENTORY_VERIFY_STATUS}" + cleanup_output+=$'\n'"${INVENTORY_VERIFY_OUTPUT:-}"$'\n'"${LOCAL_VERIFY_OUTPUT:-}" + fi + break + fi + if is_fal_not_found_output "$cleanup_output"; then + if wait_for_provider_inventory_baseline "$ambiguous_matches" "$ambiguous_minimum_seconds" && verify_local_cleanup; then + cleanup_armed=0 + else + cleanup_status="${LOCAL_VERIFY_STATUS:-$INVENTORY_VERIFY_STATUS}" + cleanup_output+=$'\nprovider and local state do not prove zero residue; manual reconciliation required\n'"${INVENTORY_VERIFY_OUTPUT:-}"$'\n'"${LOCAL_VERIFY_OUTPUT:-}" + fi + break + fi + if [ "$attempt" -lt "$cleanup_attempts" ]; then + sleep "$cleanup_poll_seconds" + fi + done + if [ "$cleanup_status" -ne 0 ] && [ "$cleanup_armed" -eq 1 ]; then + printf 'classification=cleanup_failed command=%q exit=%s slug=%s\n' "bin/crabbox stop --provider fal $slug" "$cleanup_status" "$slug" >&2 + printf '%s\n' "$cleanup_output" | redact_output >&2 + if [ "$exit_status" -eq 0 ]; then + exit_status="$cleanup_status" + fi + fi + fi + if [ -n "$config_file" ]; then + rm -f "$config_file" + fi + exit "$exit_status" +} +trap cleanup EXIT + +if [[ "${CRABBOX_LIVE:-}" != "1" ]]; then + printf 'classification=environment_blocked reason=CRABBOX_LIVE_not_enabled\n' + exit 0 +fi + +if ! provider_enabled; then + printf 'classification=environment_blocked reason=fal_not_selected providers=%q\n' "${CRABBOX_LIVE_PROVIDERS:-}" + exit 0 +fi + +if [[ -z "$fal_key" ]]; then + printf 'classification=environment_blocked reason=FAL_KEY_missing\n' + exit 0 +fi + +mkdir -p bin +go build -trimpath -o bin/crabbox ./cmd/crabbox + +config_file="$(mktemp)" +cat >"$config_file" <>"$config_file" +fi + +export CRABBOX_CONFIG="$config_file" +export CRABBOX_COORDINATOR= +export CRABBOX_FAL_KEY="$fal_key" + +run_capture_preflight "bin/crabbox doctor --provider fal" bin/crabbox doctor --provider fal +doctor_output="$CAPTURED_OUTPUT" +printf '%s\n' "$doctor_output" +if ! initial_provider_inventory="$(inventory_snapshot_from_doctor "$doctor_output")"; then + classify_validation_failure "bin/crabbox doctor --provider fal" 1 "doctor output omitted a complete provider inventory fingerprint" + exit 1 +fi +run_capture_preflight "bin/crabbox list --provider fal --json" bin/crabbox list --provider fal --json +initial_list_output="$CAPTURED_OUTPUT" +validate_list_json_empty "bin/crabbox list --provider fal --json" "$initial_list_output" +initial_local_key_inventory="$(local_key_inventory_snapshot)" +cleanup_armed=1 +warmup_args=(bin/crabbox warmup --provider fal --slug "$slug" --keep=false --fal-instance-type "$fal_instance_type") +if [[ -n "$fal_sector" ]]; then + warmup_args+=(--fal-sector "$fal_sector") +fi +warmup_args+=(--ttl 20m --idle-timeout 5m) +run_capture "${warmup_args[*]}" "${warmup_args[@]}" +run_capture_validation "bin/crabbox status --provider fal --id $slug --wait --wait-timeout 600s" bin/crabbox status --provider fal --id "$slug" --wait --wait-timeout 600s +run_capture_validation "bin/crabbox run --provider fal --id $slug --no-sync -- echo ok" bin/crabbox run --provider fal --id "$slug" --no-sync -- echo ok +run_output="$CAPTURED_OUTPUT" +validate_output_line "bin/crabbox run --provider fal --id $slug --no-sync -- echo ok" "$run_output" "ok" +run_capture_validation "bin/crabbox list --provider fal --json" bin/crabbox list --provider fal --json +list_output="$CAPTURED_OUTPUT" +printf '%s\n' "$list_output" +validate_list_json_contains_slug "bin/crabbox list --provider fal --json" "$list_output" +run_capture_validation "bin/crabbox stop --provider fal $slug" bin/crabbox stop --provider fal "$slug" +if ! wait_for_provider_inventory_baseline; then + classify_validation_failure "bin/crabbox doctor --provider fal (post-stop inventory baseline)" "$INVENTORY_VERIFY_STATUS" "$INVENTORY_VERIFY_OUTPUT" + exit 1 +fi +run_capture_validation "bin/crabbox cleanup --provider fal --dry-run" bin/crabbox cleanup --provider fal --dry-run +cleanup_output="$CAPTURED_OUTPUT" +if ! verify_local_cleanup; then + classify_validation_failure "local fal zero-residue verification" "$LOCAL_VERIFY_STATUS" "$LOCAL_VERIFY_OUTPUT" + exit 1 +fi +post_list_output="$LOCAL_VERIFY_OUTPUT" +cleanup_armed=0 +printf '%s\n' "$cleanup_output" +printf '%s\n' "$post_list_output" +printf 'classification=live_fal_smoke_passed slug=%s cleanup=complete\n' "$slug" diff --git a/scripts/live-fal-smoke.test.js b/scripts/live-fal-smoke.test.js new file mode 100644 index 000000000..375dfdebf --- /dev/null +++ b/scripts/live-fal-smoke.test.js @@ -0,0 +1,861 @@ +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { spawnSync } from "node:child_process"; +import test from "node:test"; + +const repoRoot = path.resolve(import.meta.dirname, ".."); + +function writeExecutable(file, body) { + fs.writeFileSync(file, body, "utf8"); + fs.chmodSync(file, 0o755); +} + +function prepareSmokeRepo(dir) { + const tempRoot = path.join(dir, "repo"); + const tempScripts = path.join(tempRoot, "scripts"); + const smokeScript = path.join(tempScripts, "live-fal-smoke.sh"); + fs.mkdirSync(tempScripts, { recursive: true }); + fs.copyFileSync(path.join(repoRoot, "scripts", "live-fal-smoke.sh"), smokeScript); + fs.chmodSync(smokeScript, 0o755); + return { tempRoot, smokeScript }; +} + +function isolatedEnv(dir) { + const home = path.join(dir, "home"); + const configHome = path.join(dir, "config"); + const stateHome = path.join(dir, "state"); + fs.mkdirSync(home, { recursive: true }); + return { + ...process.env, + HOME: home, + XDG_CONFIG_HOME: configHome, + XDG_STATE_HOME: stateHome, + CRABBOX_LIVE_FAL_TEST_ONLY_SHORT_RECOVERY: "1", + }; +} + +function writeGoStub(binDir, scriptBody) { + writeExecutable( + path.join(binDir, "go"), + `#!/usr/bin/env bash +set -euo pipefail +out="" +while [[ "$#" -gt 0 ]]; do + if [[ "$1" == "-o" ]]; then + out="$2" + shift 2 + continue + fi + shift +done +mkdir -p "$(dirname "$out")" +cat >"$out" <<'SCRIPT' +${scriptBody} +SCRIPT +chmod +x "$out" +`, + ); +} + +test("live fal smoke skips unless opted in", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-skip-")); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { ...isolatedEnv(dir), CRABBOX_LIVE: "", CRABBOX_FAL_KEY: "", FAL_KEY: "" }, + encoding: "utf8", + }); + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /classification=environment_blocked reason=CRABBOX_LIVE_not_enabled/); +}); + +test("live fal smoke skips unless provider filter selects fal", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-filter-")); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "aws,digitalocean", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + }, + encoding: "utf8", + }); + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /classification=environment_blocked reason=fal_not_selected/); +}); + +test("live fal smoke requires token before building", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-token-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + fs.mkdirSync(binDir, { recursive: true }); + writeExecutable(path.join(binDir, "go"), "#!/usr/bin/env bash\nexit 99\n"); + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "", + FAL_KEY: "", + }, + encoding: "utf8", + }); + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /classification=environment_blocked reason=FAL_KEY_missing/); +}); + +test("live fal smoke classifies actual preflight fal API auth errors as environment blocked", () => { + for (const message of [ + "fal API authorization_error: Authentication required", + "fal API authorization_error: Access denied", + ]) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-auth-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + fs.mkdirSync(binDir, { recursive: true }); + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +if [[ "$1" == "doctor" ]]; then + printf '${message}\n' >&2 + exit 2 +fi +exit 99 +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 0, result.stdout + result.stderr); + assert.match(result.stderr, /classification=environment_blocked .*reason=fal_api_auth/); + assert.match(result.stderr, new RegExp(message)); + assert.doesNotMatch(result.stderr, /classification=validation_failed/); + } +}); + +test("live fal smoke runs guarded lifecycle and redacts token", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const calls = path.join(dir, "calls.log"); + const slugFile = path.join(dir, "slug.txt"); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +printf '%s\\n' "$*" >>"${calls}" +if [[ "\${CRABBOX_FAL_KEY:-}" != "test-secret-token" ]]; then + printf 'missing token\\n' >&2 + exit 91 +fi +case "$1" in + doctor) + printf 'auth=ready control_plane=ready inventory=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 api=list mutation=false api_key=test-secret-token url=https://example.test/?token=abc\\n' + ;; + warmup) + printf '%s\\n' "$5" >"${slugFile}" + ;; + status) + printf 'status=ready\\n' + ;; + run) + printf 'ok\\n' + ;; + list) + slug="$(cat "${slugFile}" 2>/dev/null || true)" + if [[ -z "$slug" || -f "${slugFile}.stopped" ]]; then + printf '[]\\n' + else + printf '[{"labels":{"slug":"%s"},"provider":"fal"}]\\n' "$slug" + fi + ;; + stop) + printf stopped >"${slugFile}.stopped" + ;; + cleanup) + printf 'skip server id=none name=none reason=missing labels\\n' + ;; + *) + printf 'unexpected args: %s\\n' "$*" >&2 + exit 99 + ;; +esac +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 0, result.stdout + result.stderr); + assert.match(result.stdout, /classification=live_fal_smoke_passed/); + assert.doesNotMatch(result.stdout + result.stderr, /test-secret-token|token=abc/); + + const seen = fs.readFileSync(calls, "utf8").trim().split("\n"); + assert.equal(seen[0], "doctor --provider fal"); + assert.equal(seen[1], "list --provider fal --json"); + assert.match(seen[2], /^warmup --provider fal --slug fal-smoke-\d{14}-\d+ --keep=false --fal-instance-type gpu_1x_h100_sxm5 --ttl 20m --idle-timeout 5m$/); + assert.match(seen[3], /^status --provider fal --id fal-smoke-\d{14}-\d+ --wait --wait-timeout 600s$/); + assert.match(seen[4], /^run --provider fal --id fal-smoke-\d{14}-\d+ --no-sync -- echo ok$/); + assert.equal(seen[5], "list --provider fal --json"); + assert.match(seen[6], /^stop --provider fal fal-smoke-\d{14}-\d+$/); + assert.equal(seen[7], "doctor --provider fal"); + assert.equal(seen[8], "cleanup --provider fal --dry-run"); + assert.equal(seen[9], "list --provider fal --json"); +}); + +test("live fal smoke polls raw inventory until asynchronous deletion completes", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-async-delete-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const doctorCalls = path.join(dir, "doctor-calls.log"); + const slugFile = path.join(dir, "slug.txt"); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +case "$1" in + doctor) + count="$(wc -l <"${doctorCalls}" 2>/dev/null || printf 0)" + printf 'doctor\n' >>"${doctorCalls}" + if [[ "$count" -eq 1 ]]; then + printf 'auth=ready inventory_count=1 inventory_fingerprint=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n' + else + printf 'auth=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n' + fi + ;; + list) + slug="$(cat "${slugFile}" 2>/dev/null || true)" + if [[ -z "$slug" || -f "${slugFile}.stopped" ]]; then + printf '[]\n' + else + printf '[{"labels":{"slug":"%s"},"provider":"fal"}]\n' "$slug" + fi + ;; + warmup) + printf '%s\n' "$5" >"${slugFile}" + ;; + status) + printf 'status=ready\n' + ;; + run) + printf 'ok\n' + ;; + stop) + printf stopped >"${slugFile}.stopped" + ;; + cleanup) + printf 'clean\n' + ;; + *) + exit 99 + ;; +esac +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + CRABBOX_LIVE_FAL_INVENTORY_POLL_ATTEMPTS: "3", + CRABBOX_LIVE_FAL_INVENTORY_POLL_SECONDS: "0", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 0, result.stdout + result.stderr); + assert.match(result.stdout, /classification=live_fal_smoke_passed/); + assert.equal(fs.readFileSync(doctorCalls, "utf8"), "doctor\ndoctor\ndoctor\n"); +}); + +test("live fal smoke fails when raw inventory lingers after a successful stop", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-lingering-delete-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const doctorCalls = path.join(dir, "doctor-calls.log"); + const slugFile = path.join(dir, "slug.txt"); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +case "$1" in + doctor) + count="$(wc -l <"${doctorCalls}" 2>/dev/null || printf 0)" + printf 'doctor\n' >>"${doctorCalls}" + if [[ "$count" -eq 0 ]]; then + printf 'auth=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n' + else + printf 'auth=ready inventory_count=1 inventory_fingerprint=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n' + fi + ;; + list) + slug="$(cat "${slugFile}" 2>/dev/null || true)" + if [[ -z "$slug" || -f "${slugFile}.stopped" ]]; then + printf '[]\n' + else + printf '[{"labels":{"slug":"%s"},"provider":"fal"}]\n' "$slug" + fi + ;; + warmup) + printf '%s\n' "$5" >"${slugFile}" + ;; + status) + printf 'status=ready\n' + ;; + run) + printf 'ok\n' + ;; + stop) + printf stopped >"${slugFile}.stopped" + ;; + *) + exit 99 + ;; +esac +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + CRABBOX_LIVE_FAL_INVENTORY_POLL_ATTEMPTS: "2", + CRABBOX_LIVE_FAL_INVENTORY_POLL_SECONDS: "0", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 1, result.stdout + result.stderr); + assert.match(result.stderr, /classification=validation_failed/); + assert.match(result.stderr, /classification=cleanup_failed/); + assert.match(result.stderr, /provider inventory did not return to the pre-create baseline/); + assert.doesNotMatch(result.stdout, /classification=live_fal_smoke_passed/); +}); + +test("live fal smoke attempts cleanup after partial failure", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-fail-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const stopped = path.join(dir, "stopped.log"); + const calls = path.join(dir, "calls.log"); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +printf '%s\\n' "$*" >>"${calls}" +if [[ "$1" == "doctor" ]]; then + printf 'auth=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\n' + exit 0 +fi +if [[ "$1" == "list" ]]; then + printf '[]\\n' + exit 0 +fi +if [[ "$1" == "warmup" ]]; then + printf 'capacity unavailable after partial create\\n' >&2 + exit 37 +fi +if [[ "$1" == "stop" ]]; then + printf '%s\\n' "$4" >>"${stopped}" + exit 0 +fi +exit 99 +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 0, result.stdout + result.stderr); + assert.match(result.stderr, /classification=capacity_blocked/); + assert.match(result.stderr, /capacity unavailable after partial create/); + assert.match(fs.readFileSync(stopped, "utf8"), /^fal-smoke-\d{14}-\d+\n$/); + assert.doesNotMatch(fs.readFileSync(calls, "utf8"), /^cleanup /m); +}); + +test("live fal smoke treats SSH and malformed-request warmup errors as validation failures", () => { + for (const [label, message] of [ + ["ssh-permission", "ssh: Permission denied (publickey)"], + ["missing-field", "missing required field: instance_type"], + ]) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), `crabbox-live-fal-${label}-`)); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +case "$1" in + doctor) + printf 'auth=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n' + ;; + list) + printf '[]\n' + ;; + warmup) + printf '${message}\n' >&2 + exit 37 + ;; + stop) + printf 'lease/fal instance not found or not locally claimed\n' >&2 + exit 4 + ;; + *) + exit 99 + ;; +esac +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + CRABBOX_LIVE_FAL_AMBIGUOUS_BASELINE_OBSERVATIONS: "1", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 37, `${label}: ${result.stdout}${result.stderr}`); + assert.match(result.stderr, /classification=validation_failed/, label); + assert.match(result.stderr, new RegExp(message.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")), label); + assert.doesNotMatch(result.stderr, /classification=(environment|billing|quota|capacity)_blocked/, label); + } +}); + +test("live fal smoke classifies compute credit requirements as billing blockers", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-credit-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const doctorCalls = path.join(dir, "doctor-calls.log"); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +case "$1" in + doctor) + printf 'doctor\n' >>"${doctorCalls}" + printf 'auth=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\n' + ;; + list) + printf '[]\\n' + ;; + warmup) + printf 'In order to start a compute instance, you have to have at least $500 in your fal credit balance per instance, as well as an automated top up with a trigger at $500.\\n' >&2 + exit 37 + ;; + stop) + printf 'lease/fal instance not found or not locally claimed\\n' >&2 + exit 4 + ;; + *) + exit 99 + ;; +esac +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + CRABBOX_LIVE_FAL_INVENTORY_POLL_ATTEMPTS: "1", + CRABBOX_LIVE_FAL_INVENTORY_POLL_SECONDS: "0", + CRABBOX_LIVE_FAL_AMBIGUOUS_BASELINE_OBSERVATIONS: "1", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 0, result.stdout + result.stderr); + assert.match(result.stderr, /classification=billing_blocked/); + assert.match(result.stderr, /credit balance/); + assert.doesNotMatch(result.stderr, /classification=cleanup_failed/); + assert.equal(fs.readFileSync(doctorCalls, "utf8"), "doctor\ndoctor\n"); + + const productionWindowResult = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + CRABBOX_LIVE_FAL_TEST_ONLY_SHORT_RECOVERY: "", + CRABBOX_LIVE_FAL_INVENTORY_POLL_ATTEMPTS: "1", + CRABBOX_LIVE_FAL_INVENTORY_POLL_SECONDS: "0", + CRABBOX_LIVE_FAL_AMBIGUOUS_BASELINE_OBSERVATIONS: "1", + }, + encoding: "utf8", + }); + + assert.equal(productionWindowResult.status, 1, productionWindowResult.stdout + productionWindowResult.stderr); + assert.match(productionWindowResult.stderr, /classification=billing_blocked/); + assert.match(productionWindowResult.stderr, /classification=cleanup_failed/); +}); + +test("live fal smoke fails cleanup when provider inventory changed after an unclaimed create", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-inventory-change-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const doctorCalls = path.join(dir, "doctor-calls.log"); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +case "$1" in + doctor) + count="$(wc -l <"${doctorCalls}" 2>/dev/null || printf 0)" + printf 'doctor\n' >>"${doctorCalls}" + if [[ "$count" -eq 0 ]]; then + printf 'auth=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\n' + else + printf 'auth=ready inventory_count=1 inventory_fingerprint=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\n' + fi + ;; + list) + printf '[]\\n' + ;; + warmup) + printf 'compute credit balance required\\n' >&2 + exit 37 + ;; + stop) + printf 'lease/fal instance not found or not locally claimed\\n' >&2 + exit 4 + ;; + *) + exit 99 + ;; +esac +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + CRABBOX_LIVE_FAL_INVENTORY_POLL_ATTEMPTS: "1", + CRABBOX_LIVE_FAL_INVENTORY_POLL_SECONDS: "0", + CRABBOX_LIVE_FAL_AMBIGUOUS_BASELINE_OBSERVATIONS: "1", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 1, result.stdout + result.stderr); + assert.match(result.stderr, /classification=billing_blocked/); + assert.match(result.stderr, /classification=cleanup_failed/); + assert.match(result.stderr, /manual reconciliation required/); +}); + +test("live fal smoke does not accept a transient baseline after an ambiguous create", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-delayed-create-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const doctorCalls = path.join(dir, "doctor-calls.log"); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +case "$1" in + doctor) + count="$(wc -l <"${doctorCalls}" 2>/dev/null || printf 0)" + printf 'doctor\n' >>"${doctorCalls}" + if [[ "$count" -le 1 ]]; then + printf 'auth=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n' + else + printf 'auth=ready inventory_count=1 inventory_fingerprint=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n' + fi + ;; + list) + printf '[]\n' + ;; + warmup) + printf 'compute credit balance required after ambiguous create\n' >&2 + exit 37 + ;; + stop) + printf 'lease/fal instance not found or not locally claimed\n' >&2 + exit 4 + ;; + *) + exit 99 + ;; +esac +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + CRABBOX_LIVE_FAL_INVENTORY_POLL_ATTEMPTS: "3", + CRABBOX_LIVE_FAL_INVENTORY_POLL_SECONDS: "0", + CRABBOX_LIVE_FAL_AMBIGUOUS_BASELINE_OBSERVATIONS: "3", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 1, result.stdout + result.stderr); + assert.match(result.stderr, /classification=cleanup_failed/); + assert.match(result.stderr, /provider and local state do not prove zero residue/); + assert.equal(fs.readFileSync(doctorCalls, "utf8"), "doctor\ndoctor\ndoctor\ndoctor\n"); +}); + +test("live fal smoke retains recovery state when local key cleanup is incomplete", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-local-residue-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const keyStore = + process.platform === "darwin" + ? path.join(dir, "home", "Library", "Application Support", "crabbox", "testboxes") + : path.join(dir, "config", "crabbox", "testboxes"); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +case "$1" in + doctor) + printf 'auth=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n' + ;; + list) + printf '[]\n' + ;; + warmup) + mkdir -p "$CRABBOX_TEST_KEY_STORE/cbx_fixture" + printf fixture >"$CRABBOX_TEST_KEY_STORE/cbx_fixture/id_ed25519" + printf 'capacity unavailable after partial create\n' >&2 + exit 37 + ;; + stop) + exit 0 + ;; + *) + exit 99 + ;; +esac +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + XDG_CONFIG_HOME: path.join(dir, "config"), + XDG_STATE_HOME: path.join(dir, "state"), + CRABBOX_TEST_KEY_STORE: keyStore, + CRABBOX_LIVE_FAL_INVENTORY_POLL_ATTEMPTS: "1", + CRABBOX_LIVE_FAL_INVENTORY_POLL_SECONDS: "0", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 1, result.stdout + result.stderr); + assert.match(result.stderr, /classification=cleanup_failed/); + assert.match(result.stderr, /local fal lifecycle changed the per-lease key inventory/); + assert.equal(fs.existsSync(path.join(keyStore, "cbx_fixture", "id_ed25519")), true); +}); + +test("live fal smoke treats post-create lifecycle failure as validation", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-status-fail-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const stopped = path.join(dir, "stopped.log"); + const slugFile = path.join(dir, "slug.txt"); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +case "$1" in + doctor) + printf 'auth=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\n' + ;; + list) + printf '[]\\n' + ;; + warmup) + printf '%s\\n' "$5" >"${slugFile}" + ;; + status) + printf 'status never became ready\\n' >&2 + exit 42 + ;; + stop) + printf '%s\\n' "$4" >>"${stopped}" + ;; + *) + exit 99 + ;; +esac +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 42, result.stdout + result.stderr); + assert.match(result.stderr, /classification=validation_failed/); + assert.match(result.stderr, /status never became ready/); + assert.match(fs.readFileSync(stopped, "utf8"), /^fal-smoke-\d{14}-\d+\n$/); +}); + +test("live fal smoke requires remote command output", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-fal-output-")); + const binDir = path.join(dir, "bin"); + const { tempRoot, smokeScript } = prepareSmokeRepo(dir); + const stopped = path.join(dir, "stopped.log"); + const slugFile = path.join(dir, "slug.txt"); + fs.mkdirSync(binDir, { recursive: true }); + + writeGoStub( + binDir, + `#!/usr/bin/env bash +set -euo pipefail +case "$1" in + doctor) + printf 'auth=ready inventory_count=0 inventory_fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n' + ;; + list) + printf '[]\n' + ;; + warmup) + printf '%s\n' "$5" >"${slugFile}" + ;; + status) + printf 'status=ready\n' + ;; + run) + printf 'unexpected-output\n' + ;; + stop) + printf '%s\n' "$4" >>"${stopped}" + ;; + *) + exit 99 + ;; +esac +`, + ); + + const result = spawnSync("bash", [smokeScript], { + cwd: tempRoot, + env: { + ...isolatedEnv(dir), + PATH: `${binDir}${path.delimiter}${process.env.PATH ?? ""}`, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "test-secret-token", + FAL_KEY: "", + }, + encoding: "utf8", + }); + + assert.equal(result.status, 1, result.stdout + result.stderr); + assert.match(result.stderr, /classification=validation_failed/); + assert.match(result.stderr, /expected output line: ok/); + assert.match(fs.readFileSync(stopped, "utf8"), /^fal-smoke-\d{14}-\d+\n$/); +}); diff --git a/scripts/live-lambda-smoke.test.js b/scripts/live-lambda-smoke.test.js index 950394ca7..bd1b15947 100644 --- a/scripts/live-lambda-smoke.test.js +++ b/scripts/live-lambda-smoke.test.js @@ -6,6 +6,9 @@ import { spawnSync } from "node:child_process"; import test from "node:test"; const repoRoot = path.resolve(import.meta.dirname, ".."); +const isolatedHome = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-lambda-home-")); +process.env.HOME = isolatedHome; +process.env.XDG_CONFIG_HOME = path.join(isolatedHome, ".config"); function writeExecutable(file, body) { fs.writeFileSync(file, body, "utf8"); diff --git a/scripts/live-linode-smoke.test.js b/scripts/live-linode-smoke.test.js index eaaa07f97..23a0d2505 100644 --- a/scripts/live-linode-smoke.test.js +++ b/scripts/live-linode-smoke.test.js @@ -6,6 +6,9 @@ import { spawnSync } from "node:child_process"; import test from "node:test"; const repoRoot = path.resolve(import.meta.dirname, ".."); +const isolatedHome = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-linode-home-")); +process.env.HOME = isolatedHome; +process.env.XDG_CONFIG_HOME = path.join(isolatedHome, ".config"); function writeExecutable(file, body) { fs.writeFileSync(file, body, "utf8"); diff --git a/scripts/live-scaleway-smoke.test.js b/scripts/live-scaleway-smoke.test.js index 3dce6f5ce..2f3cfcb68 100644 --- a/scripts/live-scaleway-smoke.test.js +++ b/scripts/live-scaleway-smoke.test.js @@ -6,6 +6,9 @@ import { spawnSync } from "node:child_process"; import test from "node:test"; const repoRoot = path.resolve(import.meta.dirname, ".."); +const isolatedHome = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-scaleway-home-")); +process.env.HOME = isolatedHome; +process.env.XDG_CONFIG_HOME = path.join(isolatedHome, ".config"); function writeExecutable(file, body) { fs.writeFileSync(file, body, "utf8"); diff --git a/scripts/live-smoke.sh b/scripts/live-smoke.sh index 84b0205eb..88620b059 100755 --- a/scripts/live-smoke.sh +++ b/scripts/live-smoke.sh @@ -1422,6 +1422,10 @@ if has_provider nvidia-brev || has_provider brev || has_provider nvidia; then CRABBOX_NVIDIA_BREV_LIVE=1 "$root/scripts/live-nvidia-brev-smoke.sh" fi +if has_provider fal || has_provider fal-ai; then + "$root/scripts/live-fal-smoke.sh" +fi + if has_provider anthropic-sandbox-runtime || has_provider srt; then "$root/scripts/live-anthropic-sandbox-runtime-smoke.sh" fi diff --git a/scripts/live-smoke.test.js b/scripts/live-smoke.test.js index 95bd9f8f4..b932ed29b 100644 --- a/scripts/live-smoke.test.js +++ b/scripts/live-smoke.test.js @@ -40,6 +40,26 @@ test("OpenSandbox live smoke dispatches to the provider-specific script", () => assert.match(result.stderr, /admin active-lease check skipped/); }); +test("fal live smoke dispatches to the provider-specific script", () => { + const result = spawnSync("bash", ["scripts/live-smoke.sh"], { + cwd: repoRoot, + env: { + ...process.env, + CRABBOX_LIVE: "1", + CRABBOX_LIVE_COORDINATOR: "0", + CRABBOX_LIVE_PROVIDERS: "fal", + CRABBOX_FAL_KEY: "", + FAL_KEY: "", + CRABBOX_LIVE_REPO: repoRoot, + }, + encoding: "utf8", + }); + + assert.equal(result.status, 0, result.stdout + result.stderr); + assert.match(result.stdout, /classification=environment_blocked reason=FAL_KEY_missing/); + assert.match(result.stderr, /admin active-lease check skipped/); +}); + test("Proxmox live smoke dispatches to the provider-specific proof script", () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-proxmox-")); const fakeCrabbox = path.join(dir, "crabbox"); diff --git a/scripts/live-vultr-smoke.test.js b/scripts/live-vultr-smoke.test.js index 858cb0e29..bce61409b 100644 --- a/scripts/live-vultr-smoke.test.js +++ b/scripts/live-vultr-smoke.test.js @@ -6,6 +6,9 @@ import { spawnSync } from "node:child_process"; import test from "node:test"; const repoRoot = path.resolve(import.meta.dirname, ".."); +const isolatedHome = fs.mkdtempSync(path.join(os.tmpdir(), "crabbox-live-vultr-home-")); +process.env.HOME = isolatedHome; +process.env.XDG_CONFIG_HOME = path.join(isolatedHome, ".config"); function writeExecutable(file, body) { fs.writeFileSync(file, body, "utf8");