Shine-neko · Oct 19, 2026 · Oct 8, 2026 · Oct 8, 2026 · Oct 8, 2026 · Oct 8, 2026
diff --git a/documentation/concepts/deployment-status-lifecycle.md b/documentation/concepts/deployment-status-lifecycle.md
@@ -64,8 +64,8 @@ A `kind: job` runs one instance to completion (`replicas` is ignored).
 ```
 pending → creating → running ──→ completed   (exit 0 / clean guest shutdown)
                           │
-                          ├──→ failed         (non-zero exit, OOM, signal, host-side timeout)
-                          └──→ failed         (5 failed starts)
+                          └──→ failed         (non-zero exit, OOM, signal; once
+                                               `restart.backoff_limit` more runs failed too)
 ```
 
 - On Cloud Hypervisor the host can't read the guest's exit code, so any clean VM shutdown is `completed`. Use a worker if you need precise exit-code semantics on CH.
@@ -87,7 +87,9 @@ Without any readiness check, the legacy behaviour is preserved: `running` as soo
 
 On **Docker and Podman**, a worker is never abandoned. Each failure pushes its next start out on a randomized exponential backoff, from up to 10 s after the first failure to at most 5 minutes, and the counter resets after 10 minutes of uninterrupted running. Failures that need an operator (a missing config, a rejected spec, an exit code `126`/`127`) wait up to 5 minutes from the first attempt on, and are still retried, so fixing the cause is picked up without a re-apply. The backoff is stored in the database and survives a `ring server` restart. See [Reconciliation → restart policy](/documentation/concepts/reconciliation#restart-policy) and [`[server.restart]`](/documentation/reference/config-toml#server-restart).
 
-On **containerd, Cloud Hypervisor and Firecracker**, and for **jobs** on every runtime, the previous budget still applies: once `restart_count` reaches 5, a worker lands in `crash_loop_back_off` and a job in `failed`, and the reconciler stops retrying until the manifest is re-applied. Failures that cannot fix themselves (a missing image, config or firmware, a rejected spec) exhaust the budget at once.
+A **job** on Docker and Podman completes on exit 0. A run that exits non-zero fails it, unless its manifest sets `restart.backoff_limit`, in which case it is run again up to that many times on the same backoff. A job that cannot start (an image that cannot be pulled, a missing config, not enough memory) ran nothing, so it is retried without limit and does not count against `backoff_limit`. See [`restart`](/documentation/reference/manifest#restart).
+
+On **containerd, Cloud Hypervisor and Firecracker**, the previous budget still applies: once `restart_count` reaches 5, a worker lands in `crash_loop_back_off` and a job in `failed`, and the reconciler stops retrying until the manifest is re-applied. Failures that cannot fix themselves (a missing image, config or firmware, a rejected spec) exhaust the budget at once.
 
 Health-check failure counters live in memory only, so each `(deployment, instance, check)` triple starts back at zero after a server restart.
 

diff --git a/documentation/concepts/reconciliation.md b/documentation/concepts/reconciliation.md
@@ -86,9 +86,11 @@ On Docker and Podman, a worker is **never abandoned**. It is a service: whatever
 - Once the worker has run for `stable_after` (10 minutes) without a failure, `restart_count` resets to 0.
 - The delay, the running time and the last exit (code, time and the last 80 lines or 2 KiB of output) are stored in the database, so a `ring server` restart does not reset anyone's backoff.
 
-All of this is tunable in [`[server.restart]`](/documentation/reference/config-toml#server-restart), including `on_exhaustion = "fail"` for an operator who prefers a worker to stop after `max_attempts` failures.
+All of this is tunable in [`[server.restart]`](/documentation/reference/config-toml#server-restart), including `on_exhaustion = "fail"` for an operator who prefers a worker to stop after `max_attempts` failures, and per deployment with the manifest's [`restart`](/documentation/reference/manifest#restart) block.
 
-Jobs, and workers on containerd, Cloud Hypervisor and Firecracker, still use the previous budget: past 5 failed attempts, a worker lands in `crash_loop_back_off` and a job in `failed`, and the reconciler stops trying until the manifest is re-applied. Their retries are spaced on the same backoff curve.
+A job runs once: a run that exits non-zero fails it, unless the manifest sets `restart.backoff_limit`. A job that cannot start is retried without limit, since nothing ran.
+
+Workers and jobs on containerd, Cloud Hypervisor and Firecracker still use the previous budget: past 5 failed attempts, a worker lands in `crash_loop_back_off` and a job in `failed`, and the reconciler stops trying until the manifest is re-applied. Their retries are spaced on the same backoff curve.
 
 ## What survives a `ring server` restart
 

diff --git a/documentation/design/restart-policy.md b/documentation/design/restart-policy.md
@@ -118,7 +118,7 @@ It has no I/O and is tested exhaustively without a runtime.
 | Kind | Restart on | Exhaustion |
 |---|---|---|
 | `worker` | Any exit, any start error | Never. Keeps retrying at the backoff cap. |
-| `job` | Non-zero exit, start error | `failed` after `backoff_limit` attempts (default 6, as Kubernetes) |
+| `job` | Non-zero exit, start error | `failed` once runs that exited non-zero exceed `backoff_limit` (default 0). Start errors never exhaust a job |
 
 A worker exiting `0` is restarted like any other exit. Only a job can complete.
 
@@ -150,7 +150,7 @@ on_exhaustion = "backoff"
 max_attempts = 5            # only read when on_exhaustion = "fail"
 ```
 
-Jobs take `backoff_limit` from their manifest.
+Jobs take `backoff_limit` from the manifest's `restart` block. It defaults to 0, not to Kubernetes' 6: a job that half-ran (a migration, a dump) is not safe to run again unless its author says so. A start error, on the other hand, never counts against it: nothing ran, so the job is retried on the backoff curve without limit, as Kubernetes keeps a pod in `ImagePullBackOff` without counting it.
 
 **Per-deployment overrides**
 

diff --git a/documentation/how-to/perform-rolling-update.md b/documentation/how-to/perform-rolling-update.md
@@ -11,6 +11,7 @@ For the underlying mechanism (parent/child deployments, readiness gate, drain lo
 | Manifest has ≥1 `health_checks:` entry, exactly one active deployment shares `name`+`namespace`, no `--force` | **Rolling update** |
 | `--force` is set | Immediate replacement |
 | No health checks declared | Immediate replacement |
+| `kind: job` | Immediate replacement (a job runs once, there is nothing to hand over) |
 | Multiple active deployments share `name`+`namespace` | Immediate replacement (clean up duplicates first) |
 | Manifest publishes a host port (`ports[].published`) | Immediate replacement (**recreate**), see below |
 

diff --git a/documentation/how-to/run-a-job.md b/documentation/how-to/run-a-job.md
@@ -34,6 +34,24 @@ Status transitions:
 
 `completed` and `failed` jobs stay in the database: they're history, not active deployments. Prune them with `ring namespace prune <namespace>`.
 
+A job that could not even start (an image that cannot be pulled, a missing config, not enough memory) is not `failed`: nothing ran, so Ring keeps retrying it on the backoff curve, up to 5 minutes apart, and runs it as soon as the cause is fixed.
+
+### Retry a failed run
+
+By default a job that ran and exited non-zero is **not** run again: it may have half-done its work, and only you know whether running it twice is safe. For a job that is, ask for retries:
+
+```yaml
+deployments:
+  sync:
+    name: sync
+    kind: job
+    image: registry.example.com/sync:1.4
+    restart:
+      backoff_limit: 3   # up to 3 more runs after the first failure
+```
+
+The runs are spaced by the same backoff as workers (up to 10 s after the first failure, then doubling). See [`restart`](/documentation/reference/manifest#restart).
+
 ## Inject configuration and secrets
 
 Jobs accept the same `environment:` block as workers:
@@ -188,7 +206,7 @@ A unique date in the name keeps multiple backfills coexisting in the database fo
 
 - **No cron.** Ring does not schedule jobs on a recurring time. Trigger from cron, GitHub Actions, or any external scheduler. For periodic work inside Ring, run a long-lived worker that wakes itself up.
 - **No parallelism.** `replicas: 4` on a job runs **one** instance, not four. For fan-out, deploy multiple jobs with distinct names or use a worker consuming from a queue.
-- **No automatic retry.** A `failed` job stays failed until you act.
+- **No retry unless asked.** A job that ran and failed stays `failed` unless its manifest sets `restart.backoff_limit`. On Docker and Podman only; on the other runtimes a failed job is never run again.
 - **No timeout / deadline.** A job that hangs runs until `ring deployment delete`. Plan timeouts inside your job's command.
 - **Logs live with the container.** Once you prune the deployment, the underlying Docker container goes away and the logs go with it. Ship logs out (Loki, Fluent Bit, journald → a collector) before pruning if you need long retention.
 - **Cloud Hypervisor:** clean guest shutdown = `completed`, regardless of the workload's actual exit code. Ring can't see the guest's main-process exit from the host. If exit-code precision matters, prefer Docker. See [Runtimes](/documentation/concepts/runtimes#quick-comparison).

diff --git a/documentation/reference/config-toml.md b/documentation/reference/config-toml.md
@@ -85,7 +85,7 @@ How the scheduler restarts a failed worker. See [Reconciliation → restart poli
 | `on_exhaustion` | string | no | `"backoff"` | `"backoff"` keeps retrying at `cap` forever; `"fail"` marks the worker `failed` once its failures exceed `max_attempts` |
 | `max_attempts` | int | no | `5` | Only read when `on_exhaustion = "fail"` |
 
-Applies to workers on Docker and Podman. Other runtimes and jobs still give up after 5 failed attempts.
+Applies on Docker and Podman; a manifest can override any key for one deployment with its [`restart`](/documentation/reference/manifest#restart) block. On the other runtimes, workers and jobs still give up after 5 failed attempts.
 
 ### `[server.dashboard]`
 

diff --git a/documentation/reference/manifest.md b/documentation/reference/manifest.md
@@ -139,6 +139,7 @@ A map of deployment declarations. A manifest may carry `configs:` alone, to upda
 | `kind` | enum | `worker` | `worker` (long-running) or `job` (one-shot). On CH, a job moves to `completed` when the guest powers off cleanly; the workload's exit code is not surfaced. See [how-to: run a job](/documentation/how-to/run-a-job). |
 | `replicas` | integer | `1` | Number of instances. Jobs always run a single instance regardless. When `autoscale` is set, this is the starting count, not a fixed one. |
 | `autoscale` | object | unset | Adjust the instance count from observed CPU. Opt-in: without it the count never changes on its own. See [autoscale](#autoscale). |
+| `restart` | object | unset | How this deployment is restarted after a failure, over the server's `[server.restart]`. See [restart](#restart). |
 | `command` | string list | `[]` | Override the image's entrypoint/CMD. **Docker only**, rejected at the API on the CH runtime. |
 | `environment` | map | `{}` | Environment variables, either plain values or `secretRef` references. See [environment](#environment). |
 | `volumes` | object list | `[]` | Volume mounts. See [volumes](#volumes). |
@@ -427,6 +428,31 @@ Rejected combinations, reported at `ring apply` time:
 - **`network.mode: host` with `max` above 1** — every instance would compete for the same host ports.
 - **the `containerd` runtime on a cgroup v1 host** — containerd reports CPU from cgroup v2 only, so a CPU target would be measured against a constant zero and walk the deployment down to `min`.
 
+## `restart`
+
+Tune how Ring restarts this deployment after a failure. Every key is optional and falls back to the server's [`[server.restart]`](/documentation/reference/config-toml#server-restart):
+
+```yaml
+restart:
+  cap: 30s            # a critical worker that should come back quickly
+  stable_after: 30m   # a slow starter, reset its counter later
+```
+
+| Field | Applies to | Description |
+|---|---|---|
+| `base` | all | Upper bound of the delay before the first retry, e.g. `10s` |
+| `cap` | all | Upper bound of the delay between two retries. Must not be lower than `base`, whether `base` comes from this block or from the server |
+| `stable_after` | workers | Uninterrupted running time after which the attempt counter resets |
+| `on_exhaustion` | workers | `backoff` (default) keeps retrying at `cap` forever; `fail` marks the worker `failed` once its failures exceed `max_attempts` |
+| `max_attempts` | workers | Read when `on_exhaustion` is `fail` |
+| `backoff_limit` | jobs | How many times a run that exited non-zero is run again. Default `0`: a failed job is not run again |
+
+Durations take `s`, `m`, `h` and `d` units (`"90s"`, `"1h30m"`). Ring rejects at `ring apply` time a block that does not parse, a `cap` below the effective `base`, an unknown key, `backoff_limit` on a worker and `on_exhaustion`, `max_attempts` or `stable_after` on a job.
+
+A job that cannot even start (an image that cannot be pulled, a missing config, not enough memory) is retried without limit on the backoff curve: nothing ran, so nothing can be run twice. `backoff_limit` only counts runs that exited non-zero.
+
+Applies on Docker and Podman. On the other runtimes, workers and jobs still give up after 5 failed attempts.
+
 ## `health_checks`
 
 A list of probe definitions. Each probe runs independently with its own counter and its own failure action. Three types: `tcp`, `http`, `command`.

diff --git a/migrations/20220101000028_restart_manifest.sql b/migrations/20220101000028_restart_manifest.sql
@@ -0,0 +1,9 @@
+-- Per-deployment restart settings and job run failures.
+--
+--   `restart`       the manifest's `restart` block, JSON, NULL when it has
+--                   none: every key it leaves out falls back to the server's
+--                   `[server.restart]`.
+--   `run_failures`  runs of a job that exited non-zero, bounded by its
+--                   `backoff_limit`. Start failures do not count: nothing ran.
+ALTER TABLE deployment ADD COLUMN restart JSON DEFAULT NULL;
+ALTER TABLE deployment ADD COLUMN run_failures INTEGER NOT NULL DEFAULT 0;