Shine-neko · Oct 8, 2026 · Oct 8, 2026 · Oct 8, 2026 · Oct 8, 2026 · Oct 8, 2026
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -22,7 +22,7 @@ clap = { version = "4.5.1", features = ["derive"] }
 # Generates the shell completion scripts emitted by `ring completions <shell>`
 # from the same `Command` tree the CLI is parsed with, so the two can't drift.
 clap_complete = "4"
-chrono = "0.4"
+chrono = { version = "0.4", features = ["serde"] }
 uuid = { version = "1.4.1", features = ["v4"] }
 sqlx = { version = "0.8", features = ["runtime-tokio", "sqlite", "migrate"] }
 reqwest = { version = "0.12.11", features = ["json"] }

diff --git a/documentation/concepts/deployment-status-lifecycle.md b/documentation/concepts/deployment-status-lifecycle.md
@@ -23,18 +23,18 @@ All values serialize as `snake_case`, identically on the wire (JSON), in the CLI
 | Status | Cause | Terminal? |
 |---|---|---|
 | `failed` | A job exited non-zero / crashed; **or** a readiness check never turned green before the deadline on a non-rolling deployment; **or** (Cloud Hypervisor) firmware not found. | **Terminal** |
-| `crash_loop_back_off` | A worker's container/VM kept dying and `restart_count` reached `MAX_RESTART_COUNT` (5). The reconciler stops trying. | **Terminal** |
-| `insufficient_resources` | The host doesn't have enough memory for the deployment's request. A retry can't conjure RAM, so Ring stops. | **Terminal** |
+| `crash_loop_back_off` | A worker's instances keep dying. On Docker and Podman the worker is waiting for its next start (see [restart policy](#restart-policy)); on the other runtimes it reached 5 failed attempts. | Retried (Docker, Podman) |
+| `insufficient_resources` | The host doesn't have enough free memory for the deployment's request right now. Retried, since memory gets freed. | Retried |
 | `image_pull_back_off` | The image couldn't be pulled (tag not found, registry auth, `image_pull_policy: Never` forbidding a pull, transient network). | Retried |
 | `create_container_error` | The runtime rejected container creation (invalid mount, unsupported option, a port conflict the daemon surfaces at create time). | Retried |
 | `network_error` | Creating the namespace network/bridge failed. | Retried |
 | `config_error` | A mounted config (or a key within it) doesn't exist in the namespace. | Retried |
 | `file_system_error` | An IO error handling volumes or temp config files. | Retried |
 | `error` | Generic runtime fallback: a stats fetch, a JSON parse, a VM-start failure, or any error not classified above. | Retried |
 
-**Terminal vs retried.** A *terminal* status is never reconciled again, because the deployment is done. A *retried* failure status stays in the reconcile loop: each tick re-attempts the apply, bumping `restart_count`, until either it succeeds (back to `creating` → `running`) or the counter hits `MAX_RESTART_COUNT` and a worker flips to `crash_loop_back_off`. The reconciler explicitly polls `pending`, `creating`, `running`, `deleted`, and the five *retried* error states; `completed`, `failed`, `crash_loop_back_off`, and `insufficient_resources` are left out on purpose.
+**Terminal vs retried.** A *terminal* status is never reconciled again, because the deployment is done: only `completed` and `failed` are. Every other status stays in the reconcile loop, and a failed one is retried on the backoff curve described in [restart policy](#restart-policy) until it succeeds (back to `creating` → `running`).
 
-> **Recover from a terminal or stuck state** by re-applying a fixed manifest: `ring apply` resets `restart_count` and re-enters the lifecycle from the top. The restart counter is cumulative over the deployment's lifetime, not a sliding window.
+> **Recover from a terminal state** by re-applying a fixed manifest: `ring apply` resets `restart_count` and re-enters the lifecycle from the top.
 
 ## Worker lifecycle
 
@@ -45,13 +45,12 @@ pending → creating ──────────────→ running ─
               │     (gate: ready)      │
               │                        ├──→ deleted              (you delete it)
               │←── readiness not green │
-              │    (held here)         └──→ crash_loop_back_off  (restart_count ≥ 5)
+              │    (held here)         └──→ crash_loop_back_off  (an instance died;
+              │                                                  waiting for the next start)
               │
               └──→ image_pull_back_off / create_container_error / network_error /
-                   config_error / file_system_error / error   (retried; → crash_loop_back_off
-                   after MAX_RESTART_COUNT, or → running once resolved)
-
-              insufficient_resources  (terminal — host out of memory)
+                   config_error / file_system_error / insufficient_resources / error
+                   (waiting for the next start; → creating once it succeeds)
 ```
 
 - **`creating → running`** happens as soon as the container/VM is up **unless** the deployment declares a `readiness: true` health check, in which case the [readiness gate](#the-readiness-gate) holds it in `creating` until ready.
@@ -66,7 +65,7 @@ A `kind: job` runs one instance to completion (`replicas` is ignored).
 pending → creating → running ──→ completed   (exit 0 / clean guest shutdown)
                           │
                           ├──→ failed         (non-zero exit, OOM, signal, host-side timeout)
-                          └──→ failed         (restart_count ≥ MAX_RESTART_COUNT)
+                          └──→ failed         (5 failed starts)
 ```
 
 - On Cloud Hypervisor the host can't read the guest's exit code, so any clean VM shutdown is `completed`. Use a worker if you need precise exit-code semantics on CH.
@@ -82,18 +81,15 @@ While in `creating`, only the readiness checks run (recorded for the gate to rea
 
 Without any readiness check, the legacy behaviour is preserved: `running` as soon as the container is up. See [Health checks (design) → the readiness gate](/documentation/concepts/health-checks-design#the-readiness-gate) for the full mechanics.
 
-## Restart counter and `crash_loop_back_off`
-
-Ring tracks a cumulative `restart_count` per deployment. It is bumped when:
+## Restart policy
 
-- A worker's container dies unexpectedly (Docker `die`/`oom`/`kill` events, or a CH VM going unresponsive), unless the shutdown was intentional (a delete/scale-down).
-- A *retried* error status re-attempts its apply and fails again.
+`restart_count` counts failed attempts since the last reset: an instance that died unexpectedly (not one Ring stopped itself on a delete or scale-down), a start that failed, or an instance removed by a liveness check.
 
-Once `restart_count` reaches `MAX_RESTART_COUNT` (5), the next tick flips a **worker** to `crash_loop_back_off` (terminal) and a **job** to `failed` (terminal): the reconciler stops retrying, protecting the host from a tight crash loop. The counter is **cumulative for the deployment's lifetime**, not a sliding window; `ring apply` with a fixed manifest resets it.
+On **Docker and Podman**, a worker is never abandoned. Each failure pushes its next start out on a randomized exponential backoff, from up to 10 s after the first failure to at most 5 minutes, and the counter resets after 10 minutes of uninterrupted running. Failures that need an operator (a missing config, a rejected spec, an exit code `126`/`127`) wait up to 5 minutes from the first attempt on, and are still retried, so fixing the cause is picked up without a re-apply. The backoff is stored in the database and survives a `ring server` restart. See [Reconciliation → restart policy](/documentation/concepts/reconciliation#restart-policy) and [`[server.restart]`](/documentation/reference/config-toml#server-restart).
 
-**Permanent failures skip the budget.** Some failures cannot fix themselves on a retry: the image genuinely doesn't exist, a referenced config or key is absent, the container/VM spec is rejected, the Firecracker kernel or Cloud Hypervisor firmware is missing at its configured path, or the host is out of memory. Rather than bumping the counter by one and burning five reconcile cycles to reach the same conclusion, Ring classifies these as terminal and lands on the matching status (`image_pull_back_off`, `config_error`, `create_container_error`, `failed`, `insufficient_resources`) on the next tick. Transient failures — a pull that died mid-flight, a busy port, a network setup race — still bump by one and retry within the budget. The classification is shared by every runtime, so Docker, Podman, containerd, Cloud Hypervisor and Firecracker converge identically.
+On **containerd, Cloud Hypervisor and Firecracker**, and for **jobs** on every runtime, the previous budget still applies: once `restart_count` reaches 5, a worker lands in `crash_loop_back_off` and a job in `failed`, and the reconciler stops retrying until the manifest is re-applied. Failures that cannot fix themselves (a missing image, config or firmware, a rejected spec) exhaust the budget at once.
 
-Counters live in memory only, so restarting `ring server` clears them, so each `(deployment, instance, check)` triple starts back at zero after a server restart.
+Health-check failure counters live in memory only, so each `(deployment, instance, check)` triple starts back at zero after a server restart.
 
 ## Observing the status
 

diff --git a/documentation/concepts/reconciliation.md b/documentation/concepts/reconciliation.md
@@ -78,9 +78,17 @@ In both cases, the missing instance is recreated automatically on the next tick.
 
 ## Restart policy
 
-Ring tracks restart attempts per deployment. Past `MAX_RESTART_COUNT` (currently 5) failed boots, the deployment lands in `crash_loop_back_off` and the reconciler stops trying. The counter is **cumulative for the lifetime of the deployment**, not a sliding window; fix the underlying issue and re-apply the manifest to reset.
+On Docker and Podman, a worker is **never abandoned**. It is a service: whatever ended it (a crash, a clean exit, a failed start, a liveness check), Ring starts it again, and slows down instead of giving up.
 
-This protects the host from a tight crash loop pegging Docker / Cloud Hypervisor.
+- Each failure counts one attempt in `restart_count`, and the next start waits a random delay between zero and `min(cap, base × 2^(attempt − 1))`: up to 10 s after the first failure, 20 s after the second, and so on up to 5 minutes. The randomness ("full jitter") spreads retries out, so workloads that failed together, after a host reboot for instance, do not all come back in the same second.
+- While a worker waits, it shows `crash_loop_back_off` (or the status of the start error, such as `image_pull_back_off`), and the reconciler keeps watching it: it is started again as soon as the delay elapses, without a re-apply. Creating a missing config, or pushing the missing image, is enough.
+- Failures that need an operator (a missing config, a spec the runtime rejects, an exit code `126`/`127` meaning the program cannot run) wait up to `cap` from the first attempt on.
+- Once the worker has run for `stable_after` (10 minutes) without a failure, `restart_count` resets to 0.
+- The delay, the running time and the last exit (code, time and the last 80 lines or 2 KiB of output) are stored in the database, so a `ring server` restart does not reset anyone's backoff.
+
+All of this is tunable in [`[server.restart]`](/documentation/reference/config-toml#server-restart), including `on_exhaustion = "fail"` for an operator who prefers a worker to stop after `max_attempts` failures.
+
+Jobs, and workers on containerd, Cloud Hypervisor and Firecracker, still use the previous budget: past 5 failed attempts, a worker lands in `crash_loop_back_off` and a job in `failed`, and the reconciler stops trying until the manifest is re-applied. Their retries are spaced on the same backoff curve.
 
 ## What survives a `ring server` restart
 
@@ -93,6 +101,7 @@ Every input to the loop lives in SQLite:
 
 Two things **don't** survive:
 
+- **Restart state survives**: the next start time, the running time and the last exit are stored with the deployment.
 - **Health-check failure counters**: they live in memory. After a restart, each `(deployment, instance, check)` triple starts back at zero. A flapping service won't trigger `on_failure` immediately after a server restart.
 - **In-flight runtime operations**: if `ring server` crashes mid-`apply`, the partial state is detected on the next tick and the reconciler converges.
 

diff --git a/documentation/design/restart-policy.md b/documentation/design/restart-policy.md
@@ -261,9 +261,9 @@ Clients of the API (CLI, dashboard, webhook consumers) must be updated together
 Each step is a separate pull request that builds, passes the suite, and can ship on its own.
 
 1. **Policy module.** `scheduler/restart.rs` with its tests: backoff curve and jitter bounds, reset after `stable_after`, worker vs job exhaustion, `on_exhaustion = "fail"`, start-error starting points, resolution of a manifest override over the server defaults. Not wired yet.
-2. **Persisted restart state.** Migration for `next_attempt_at`, `running_since`, `last_termination`. The scheduler reads and writes them; `RetryBackoff` and `HealthyWindow` are removed. The `[restart]` server section and the manifest `restart` block are parsed and validated.
-3. **Runtime interface.** `observe` / `start_instance` / `stop_instance`, implemented for Docker and Podman. Scaling, rolling updates and the readiness gate move into the scheduler. The policy module is wired in for these runtimes, and liveness kills are counted as restarts.
-4. **Remaining runtimes.** containerd, then Firecracker and Cloud Hypervisor, each deleting its own `handle_*` and restart-count code.
+2. **Docker and Podman workers.** Persisted restart state (`next_attempt_at`, `running_since`, `last_termination`), the `[restart]` server section, and the instance interface (`observe` / `start_instance` / `stop_instance` / `discard_instance`) implemented for Docker and Podman. The scheduler reconciles their workers with the policy module and counts liveness kills as restarts. `RetryBackoff` is removed; the other runtimes keep their `apply` path but get their retries spaced on the persisted backoff. Persisting the state alone would not have shipped on its own: the runtimes still decided when to give up, and a longer reset window without that change would have abandoned workers that are restarted indefinitely today.
+3. **Jobs and manifest overrides.** `backoff_limit` and the `restart` block in the manifest, then Docker and Podman jobs on the policy module.
+4. **Remaining runtimes.** containerd, then Firecracker and Cloud Hypervisor, each deleting its own `handle_*` and restart-count code. `HealthyWindow` goes with the last one.
 5. **Phase and reason.** Migration of the status column, API, CLI, dashboard and webhook changes, documentation (`deployment-status-lifecycle.md`, `reconciliation.md`, troubleshooting).
 6. **Image digest reuse and start rate limiting.**
 7. **Metrics and events.**

diff --git a/documentation/help/troubleshooting.md b/documentation/help/troubleshooting.md
@@ -155,14 +155,14 @@ See [how-to: deploy with secrets → private registry credentials](/documentatio
 
 ### `crash_loop_back_off`
 
-The container has crashed more than `MAX_RESTART_COUNT` times. Look at:
+The worker's instances keep dying. On Docker and Podman, Ring keeps restarting it, waiting longer between attempts (up to 5 minutes); on the other runtimes it gave up after 5 attempts. Look at:
 
 ```bash
-ring deployment events <DEPLOYMENT_ID> --level warning   # crashes show up here
+ring deployment events <DEPLOYMENT_ID> --level error     # one container_crashed per exit, with its code
 ring deployment logs <DEPLOYMENT_ID> --tail 200          # what the app said
 ```
 
-After fixing the root cause, re-apply the manifest. Ring resets the counter on a fresh apply.
+On Docker and Podman, fixing the cause (a config, a dependency) is enough: the next attempt picks it up. Re-apply the manifest to start again right away with a fresh counter, or scale the worker to 0 to stop the retries.
 
 ### `insufficient_resources`
 
@@ -173,9 +173,9 @@ ring deployment events <DEPLOYMENT_ID> --level error --limit 5
 # insufficient host memory for 'web': needs 4096 MiB but only 1800 MiB is available — …
 ```
 
-This status is **terminal**: Ring does not retry, because the memory isn't going to reappear on its own. Two ways out:
+Ring retries on the backoff curve, so the deployment starts by itself once enough memory is free. To get it running sooner:
 
-- Free memory on the host (stop other workloads), then re-apply the manifest.
+- Free memory on the host (stop other workloads).
 - Lower the deployment's `resources.requests.memory` (or `resources.limits.memory` if no request is set) to fit, then re-apply.
 
 The check compares against memory available *at that moment*; it's a guard against gross over-asks, not a precise reservation system. CPU is not gated; CPU overcommit is allowed.

diff --git a/documentation/reference/api.md b/documentation/reference/api.md
@@ -145,13 +145,13 @@ List deployments.
 | `completed` | Job exited `0` (jobs only) | Yes |
 | `failed` | Job failed, or readiness deadline exceeded, or firmware not found | Yes |
 | `deleted` | Marked for teardown; instances being removed, then purged | No |
-| `crash_loop_back_off` | Worker hit `MAX_RESTART_COUNT` (5) restarts | Yes |
+| `crash_loop_back_off` | Worker instances keep dying; waiting for the next start (Docker, Podman) or gave up after 5 attempts (other runtimes) | No on Docker/Podman |
 | `image_pull_back_off` | Image couldn't be pulled (tag, auth, policy, network) | No (retried) |
 | `create_container_error` | Runtime rejected container creation | No (retried) |
 | `network_error` | Namespace network/bridge creation failed | No (retried) |
 | `config_error` | A mounted config or key doesn't exist | No (retried) |
 | `file_system_error` | IO error on volumes or temp config files | No (retried) |
-| `insufficient_resources` | Host out of memory for the deployment's request | Yes |
+| `insufficient_resources` | Host out of memory for the deployment's request | No (retried) |
 | `error` | Generic runtime fallback (stats, JSON, VM start, unclassified) | No (retried) |
 
 **Examples:**

diff --git a/documentation/reference/config-toml.md b/documentation/reference/config-toml.md
@@ -14,6 +14,7 @@ A context describes one client→server connection; it has no business deciding
 ```toml
 [server]                                  # daemon config (shared)
 [server.scheduler]                        # optional
+[server.restart]                          # optional
 [server.dashboard]                        # optional
 [server.exec]                             # opt-in: enabled = true
 [server.telemetry.traces]                 # opt-in: enabled = true
@@ -72,6 +73,20 @@ The daemon's own configuration, shared by every context in the file. All subsect
 |---|---|---|---|---|
 | `interval` | int (seconds) | no | `10` | Reconciliation tick interval. Overridden by `RING_SCHEDULER_INTERVAL` if set |
 
+### `[server.restart]`
+
+How the scheduler restarts a failed worker. See [Reconciliation → restart policy](/documentation/concepts/reconciliation#restart-policy). Durations accept `s`, `m`, `h` and `d` (`"10s"`, `"5m"`, `"1h30m"`). A value Ring cannot parse stops the server at startup.
+
+| Field | Type | Required | Default | Purpose |
+|---|---|---|---|---|
+| `base` | duration | no | `"10s"` | Upper bound of the delay before the first retry |
+| `cap` | duration | no | `"5m"` | Upper bound of the delay between two retries, however many failed. Must not be lower than `base` |
+| `stable_after` | duration | no | `"10m"` | Uninterrupted running time after which the attempt counter resets |
+| `on_exhaustion` | string | no | `"backoff"` | `"backoff"` keeps retrying at `cap` forever; `"fail"` marks the worker `failed` once its failures exceed `max_attempts` |
+| `max_attempts` | int | no | `5` | Only read when `on_exhaustion = "fail"` |
+
+Applies to workers on Docker and Podman. Other runtimes and jobs still give up after 5 failed attempts.
+
 ### `[server.dashboard]`
 
 | Field | Type | Required | Default | Purpose |