vigil is a service supervisor and container init daemon written in Rust. It manages multiple processes inside a single container — starting, stopping, restarting, and health-checking them — and exposes a REST API over a Unix socket (and optionally HTTPS) for programmatic control. It runs unprivileged, e.g. on OpenShift with a random UID.
Important
Upgrading from v3? v4 changes defaults (environment, access over TLS, shutdown) and reads layer files strictly. See the migration guide.
| Feature | Description |
|---|---|
| PID 1 / zombie reaper | Safe to run as container entrypoint; reaps orphaned child processes automatically |
| YAML layer config | Declarative service & check definitions; multiple layers are merged in order |
| Service supervision | Start, stop, restart with exponential backoff; healthy-time resets the backoff |
| Oneshot jobs and cron schedules | type: oneshot for migrations and jobs, schedule (cron, with time zone) instead of a cron daemon, run-timeout for jobs that hang |
| Readiness | ready: process | notify | check: dependents start when a service is ready (sd_notify READY=1 or a passing check) |
| Custom stop signal | Per-service stop-signal (SIGTERM, SIGUSR1, SIGHUP, …) + kill-delay |
| Startup ordering | after: / before: / requires: dependency graph |
| Health checks | HTTP, TCP, and exec checks with configurable period/timeout/threshold/delay; levels alive and ready |
/v1/health |
One endpoint for Kubernetes liveness and readiness probes, no credentials needed (?level=alive|ready) |
| on-check-failure | Automatic service restart (or shutdown) when a check goes down |
| Alerting | HTTP(S) notifications on check state transitions, service failures (on-service) and invalid layers (on-vigild) — Alertmanager, CloudEvents, OTLP logs, or generic webhook; env:VAR values for url/headers/labels/proxy under the same pass-env* rules as services, checked when loading (an unset or not allowed variable is an error); proxy support; configurable retry with backoff; body-template uses minijinja (Jinja2-compatible) for custom webhook payloads |
| on-success / on-failure | Per-service exit policies: restart, ignore, shutdown, failure-shutdown, success-shutdown |
| Exit-code propagation | vigild exits with the managed service's actual exit code |
| Log streaming | stdout/stderr captured per service; vigil logs and SSE follow (vigil logs -f) |
| Stack traces | logs-multiline (java, python, indent or a regex) joins a stack trace and its log line into one entry; vigil-log-relay --multiline does the same for pod logs and vigild streams |
| Log push | logs-push-socket / logs-push-addr: vigild streams ndjson to a collector (Filebeat, Fluent Bit, …) |
| vigil-log-relay | Forwards vigild's log stream or Kubernetes pod logs to a TCP sink |
| Environment and secrets | Services get only a minimal environment unless pass-env* allows more; minijinja templates ({{ file() }}, {{ mem() }}, {{ cpu() }}) and env-file in command and environment |
| watch-files | Restart or signal a service when a mounted Secret or config file changes |
| Prometheus metrics | GET /v1/metrics (OpenMetrics): services, checks, health, alerts, layer reloads |
| REST API | Full HTTP API over Unix socket; Swagger UI at /docs (assets loaded from unpkg.com — requires internet access; use /openapi.json in air-gapped environments) |
| OpenAPI | Auto-generated spec via utoipa; served at /openapi.json |
| TLS listener | Optional HTTPS API (--tls-addr); auto-generates self-signed cert |
| Identity management | Named principals with open/metrics/read/write/admin access levels (Unix UID, HTTP Basic, TLS client certificate); defined in layer files (password hash or CA from a file, e.g. a Kubernetes Secret) or added over the API |
| Signal forwarding | SIGHUP / SIGUSR1 / SIGUSR2 forwarded to all running service process groups; vigil services signal SIGHUP nginx for single services |
| Replan | Hot-reload layer YAML files without restarting the daemon; changed services restart; --watch-layers reloads by itself (e.g. a mounted ConfigMap); vigil plan shows the merged plan, secrets masked |
| Strict configuration | An unknown key or invalid value in a layer file is an error that names it; nothing is silently ignored |
| mimalloc | High-performance allocator enabled globally |
vigil-rs/
├── crates/
│ ├── vigil-types/ # Shared types (plan, API, identity, signal) — no async
│ ├── vigil-files/ # How all binaries read configured files (secrets, env files, certificates)
│ ├── vigil-tls/ # TLS material for all binaries: CA bundles, cert + key, client settings
│ ├── vigild/ # Daemon binary — axum HTTP server, overlord, service actors
│ ├── vigil/ # CLI binary — hyper (Unix socket) and reqwest (HTTPS) client
│ └── vigil-log-relay/ # Standalone log forwarder — K8s pods / HTTP / Unix socket → TCP sink
└── examples/
├── full-container/ # vigild + h2o (single service)
├── hug/ # vigild + HAProxy + controller (multi-service)
├── kubernetes-pod-logs/ # vigil-log-relay + Filebeat on Kubernetes
└── ... # filebeat, fluentbit, vector, php-caddy, quarkus, …
# Core binaries (daemon + CLI)
cargo build --release --bin vigild --bin vigil
# Include the log forwarder
cargo build --release --bin vigild --bin vigil --bin vigil-log-relay
# All workspace binaries
cargo build --releasevigild --layers-dir /etc/vigil/layers
# Unprivileged (e.g. OpenShift): runtime files under /tmp/vigil
vigild --layers-dir /etc/vigil/layers --path-prefix /tmp/vigilWarning
Keep --path-prefix short: at most 78 bytes. Unix socket paths are
limited to 107 bytes by Linux, and vigild puts its sockets under the
prefix. See Socket path limit.
# Default: connect via Unix socket
vigil services list
vigil checks list
vigil checks run myservice-alive # run a check now; exit code 1 when it fails
vigil logs -f
vigil services start myservice
vigil services restart myservice
vigil services stop myservice
# Same runtime directory as vigild (if not /run/vigil)
VIGIL_PATH_PREFIX=/tmp/vigil vigil services list
# Connect via HTTPS (vigild --tls-addr); needs an identity, see TLS docs
vigil --url https://myhost:8443 --cacert ca.pem --user deploy:secret services list
# HTTPS with auto-generated self-signed cert
vigil --url https://myhost:8443 --insecure --user deploy:secret services list# Unix socket
curl --unix-socket /run/vigil/vigild.sock http://localhost/v1/system-info
curl --unix-socket /run/vigil/vigild.sock http://localhost/v1/services
# HTTPS: open endpoints without credentials, the rest with an identity
curl -k https://myhost:8443/v1/health
curl -k -u deploy:secret https://myhost:8443/v1/servicesServices and health checks are defined in YAML layer files. Multiple layers are merged in order — later layers override earlier ones.
# /etc/vigil/layers/001-app.yaml
summary: My application
services:
myapp:
summary: Main application process
command: /usr/local/bin/myapp --config /etc/myapp/config.yaml
startup: enabled
stop-signal: SIGTERM
kill-delay: 10s
on-success: restart
on-failure: restart
backoff-delay: 1s
backoff-factor: 2.0
backoff-limit: 30s
on-check-failure:
myapp-alive: restart
sidecar:
summary: Helper process (starts after myapp)
command: /usr/local/bin/sidecar
startup: enabled
after:
- myapp
on-success: restart
on-failure: restart
checks:
myapp-alive:
level: alive
startup: enabled
delay: 3s # wait before first check (default: 3s)
period: 10s
timeout: 3s
threshold: 3
http:
url: http://localhost:8080/healthz# HTTP check
http:
url: https://localhost:8080/healthz
headers: # optional — arbitrary key/value map
Authorization: "Bearer token"
# tls: # optional: CAs, client certificate, insecure
# ca: /etc/vigil/internal-ca.pem
# proxy: # optional; without it, a direct connection
# url: http://proxy.example.com:3128
# TCP check
tcp:
host: localhost
port: 5432
# Exec check (exit code 0 = healthy)
exec:
command: pg_isready -U postgres
service-context: myapp # inherit env/user/group from servicevigild is designed to run as PID 1 inside a container.
ENTRYPOINT ["/usr/local/bin/vigild", "--layers-dir", "/etc/vigil/layers"]On OpenShift (random UID, /run not writable) set
ENV VIGIL_PATH_PREFIX=/tmp/vigil; see
Container Setup.
Interact with the running container:
podman exec <ctr> vigil services list
podman exec <ctr> vigil logs -f
podman exec <ctr> vigil services restart myappSee the examples/ directory for complete, buildable examples.
| dumb-init / tini | vigil-rs | |
|---|---|---|
| Purpose | Minimal PID 1 shim | Full service supervisor |
| Manages multiple processes | ❌ (single process only) | ✅ |
| Restart on failure | ❌ | ✅ with backoff |
| Health checks | ❌ | ✅ HTTP / TCP / exec |
| HTTP check headers | ❌ | ✅ arbitrary key/value map |
HTTP check TLS and proxy (tls: / proxy:) |
❌ | ✅ custom CAs, client certificate, proxy |
| Alerting (Alertmanager / CloudEvents / OTLP) | ❌ | ✅ |
| Runtime API | ❌ | ✅ REST over Unix socket and HTTPS |
| Configuration | None (CLI args) | YAML layers |
| Signal forwarding | ✅ | ✅ |
| Zombie reaping | ✅ | ✅ |
| Binary size | ~20 KB | ~16 MB (vigild, stripped) |
When to use dumb-init/tini: You have exactly one process and only need zombie reaping + signal forwarding. Zero configuration overhead.
When to use vigil: You run multiple processes and want health checks, automatic restarts, and programmatic control.
| s6-overlay | vigil-rs | |
|---|---|---|
| Language | C | Rust |
| Configuration | Directory-based (/etc/s6-overlay/s6-rc.d/) |
YAML layers |
| Health checks | ❌ (external tooling) | ✅ built-in |
| Readiness | ✅ notification fd | ✅ ready: notify | check |
| HTTP check headers | ❌ | ✅ arbitrary key/value map |
HTTP check TLS and proxy (tls: / proxy:) |
❌ | ✅ custom CAs, client certificate, proxy |
| Alerting (Alertmanager / CloudEvents / OTLP) | ❌ | ✅ |
| Runtime API | ❌ (CLI tools s6-svc, s6-rc) |
✅ REST over Unix socket and HTTPS |
| Log routing | ✅ s6-log per service, rotation to files |
log buffer, API stream, ndjson push to collectors, vigil-log-relay, stack traces as one entry |
| Dependency ordering | ✅ (dependencies.d/) |
✅ (after: / before: / requires:) |
| Custom stop signal | ✅ down-signal file, timeout-kill |
✅ stop-signal, kill-delay |
| Used by | linuxserver.io images | - |
When to use s6-overlay: Classic multi-service containers (nginx + cron + sshd) where you don't need runtime control. Battle-tested, extremely small.
When to use vigil: You need health checks, per-service restart policies, dynamic reconfiguration (replan), and a REST API to query or control services from outside the container.
| supervisord | go-supervisord | vigil-rs | |
|---|---|---|---|
| Language | Python | Go | Rust |
| Configuration | INI file (+ [include]) |
INI file (+ [include]) |
YAML layers, merged in order |
| Custom stop signal | ✅ stopsignal |
✅ stopsignal |
✅ stop-signal |
| Startup ordering | priority only |
✅ depends_on |
✅ after / before / requires |
| Health checks | ❌ | ✅ liveness: HTTP, TCP, script; restart, stop or script on failure | ✅ HTTP / TCP / exec, levels alive / ready, /v1/health |
| HTTP check headers, TLS, proxy | ❌ | headers, client cert, CA | ✅ headers, custom CAs, client certificate, proxy |
| Alerting | via event listeners (e.g. superlance) | via event listeners | ✅ Alertmanager / CloudEvents / OTLP / webhook |
| API | XML-RPC over HTTP (Unix socket or TCP) | XML-RPC + REST, web GUI | JSON REST over Unix socket and HTTPS, OpenAPI |
| Prometheus metrics | ❌ | ✅ | ✅ |
| Zombie reaping as PID 1 | ✅ waitpid(-1) reaps every child |
✅ | ✅ |
| Layer merging / replan | ❌ (reread / update) |
❌ | ✅ |
vigil-rs started as a Rust rewrite of Pebble. The layer format and the API look alike, but they are not interchangeable (see below).
| Pebble | vigil-rs | |
|---|---|---|
| Language | Go | Rust |
| PID 1 / zombie reaper | ✅ always a child subreaper | ✅ as PID 1 or with --reaper |
| Custom stop signal | ❌ | ✅ stop-signal |
| Oneshot services, cron schedules | ❌ | ✅ type: oneshot, schedule |
| Readiness | ❌ (a start succeeds after 1 s) | ✅ ready: process | notify | check |
| Templating, env files, environment isolation | ❌ (services inherit the full environment) | ✅ minijinja, env-file, pass-env* |
| Exit code of the daemon | 0, 10 (service failure), 11 (check failure) | the service's own exit code (--exit-code propagate), or 0/10 |
| HTTP check headers | ✅ | ✅ |
HTTP check TLS and proxy (tls: / proxy:) |
❌ | ✅ custom CAs, client certificate, proxy |
Check delay field |
❌ | ✅ |
/v1/health endpoint |
✅ | ✅ (404 for a name that is not a check) |
| Log targets | ✅ Loki, OpenTelemetry, syslog | ndjson push per service (Unix/TCP), vigil-log-relay |
| Alerting (webhook / Alertmanager / CloudEvents / OTLP) | ❌ | ✅ |
| Signals to single services (API) | ✅ | ✅ POST /v1/signals |
| exec and files API; notices | ✅ | ❌ |
Asynchronous changes (/v1/changes, /wait) |
✅ | ❌ (service actions are synchronous) |
| TLS API listener | ✅ --https |
✅ --tls-addr, optional mTLS |
| Identities | API, persisted | API (in memory) and layer files |
| Prometheus metrics | ✅ | ✅ (more series) |
| OpenAPI / Swagger UI | ❌ | ✅ |
Moving from Pebble. Simple layer files load unchanged, but some keys behave differently:
- Checks without
startup:do not run in vigil-rs (Pebble starts them); setstartup: enabled. backoff-limitis where vigil-rs gives up (stateerror); Pebble keeps retrying at that interval.requiresorders the start and stops the dependent when the dependency stops; it does not start the dependency.- Services get only
PATH,TZ,HOME,USERandLOGNAMEfrom vigild's environment unlesspass-env*allows more. log-targets:,workloads:and layer subdirectories (NNN-name.d/) are not supported.- The REST API has the same shape for
/v1/services,/v1/checksand/v1/system-info, but/v1/logsreturns a JSON envelope (withfollow=truean NDJSON stream without one), service actions return a finished change instead of an asynchronous one, and unknown fields and query parameters are rejected. Pebble clients need adjusting.
Single web server supervised by vigild. Demonstrates health checks, on-check-failure restart, and the vigil CLI.
podman build -f examples/full-container/Containerfile -t vigil-h2o .
podman run --rm --network host --name vigil-h2o vigil-h2oRuns vigild (service supervisor) and vigil-log-relay (K8s pod log forwarder) as sidecars alongside Filebeat in a single pod. vigil-log-relay watches a namespace via the K8s Watch API, streams pod logs, and forwards them as ndjson to Filebeat over a local TCP connection.
K8s stream query parameter (stdout/stderr separation)
Kubernetes ≥ 1.32 knows a stream=Stdout / stream=Stderr query parameter
on the pod log endpoint, allowing stdout and stderr to be collected as separate
labeled streams. It only works with the feature gate PodLogsQuerySplitStreams,
which is off by default (also on 1.35), and some distributions forbid it via
admission control.
On K8s ≥ 1.32 vigil-log-relay therefore tries the parameter first. If the API
server answers HTTP 422 "stream: Forbidden: may not be specified", the relay
logs one warning and switches to the combined stream ("stream": "output")
for all pods; no log lines are lost. To skip that first attempt, set
--no-stream-param (or NO_STREAM_PARAM=true):
env:
- name: NO_STREAM_PARAM
value: "true"HTTP/2 multiplexing
The kube client (hyper + rustls-tls) currently uses HTTP/1.1 — one TCP
connection per pod stream. HTTP/2 multiplexing is tracked in
kube-rs PR #1823 (draft). Use
--max-log-requests to cap concurrent connections when watching large namespaces.
Two services: HAProxy (with graceful SIGUSR1 drain) and a controller that starts after HAProxy. Demonstrates:
after:startup orderingstop-signal: SIGUSR1for graceful drain- exec health check via HAProxy Unix socket
on-check-failurerestart policy
podman build -f examples/hug/Containerfile -t vigil-hug .
podman run --rm --network host --name vigil-hug vigil-hugInteract:
podman exec vigil-hug vigil services list
podman exec vigil-hug vigil checks list
podman exec vigil-hug vigil logscargo test --workspaceEnd-to-end integration tests are written in the VTest2
.vtc format — the same test framework used by the Varnish HTTP cache. Each
test starts a real vigild process (and sometimes vigil-log-relay) with a
temporary layers directory and Unix socket, drives it through the HTTP API and
the vigil CLI, and asserts on the responses.
32 test files cover: system info, service/check/alert/identity CRUD, log streaming and follow, TLS listener, reaper, mTLS, alert body templates, log-relay socket and URL sources, TCP reconnect, JSON log format, HTTPS transport, service alerts, layer watching, shutdown exit codes, identities in layer files, check failure reasons and metrics, signals, replan restarts, request validation, and more.
# Run all VTest2 tests (requires VTest2 binary)
tests/vtest/run.sh
# Single test, verbose
tests/vtest/run.sh -v tests/vtest/v0001_system_info.vtc
# Combined coverage report (unit + integration + VTest2)
tests/vtest/run.sh --coverage # text summary
tests/vtest/run.sh --coverage --html # HTML report
# vigil-log-relay --kubernetes against a real cluster (creates and removes
# the namespace vigil-relay-test)
KUBECONFIG=~/.kube/test-cluster.yml tests/k8s/relay.sh
# Coverage including the cluster test
K8S_TEST=1 KUBECONFIG=~/.kube/test-cluster.yml tests/vtest/run.sh --coverageSee tests/vtest/README.md for prerequisites,
patterns, and gotchas.
Line coverage across all crates: ~96 % (measured with
llvm-cov via K8S_TEST=1 tests/vtest/run.sh --coverage).
Without a cluster (K8S_TEST unset) the source_k8s module of
vigil-log-relay stays uncovered and the total is about 93.5 %.
vigil-rs deliberately splits into two separate binaries (plus the optional
vigil-log-relay) instead of one binary that acts as daemon or client
depending on the subcommand.
┌──────────────────────────────────────────────────────────────┐
│ Container / Host │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ vigild (PID 1) │ │
│ │ │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ Overlord │ │ Service │ │ Check │ │ │
│ │ │ (actor) │──▶│ Actors │ │ Actors │ │ │
│ │ └─────┬──────┘ └────────────┘ └────────────┘ │ │
│ │ │ │ │
│ │ ┌─────▼──────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ axum API │ │ LogStore │ │ TLS API │ │ │
│ │ │ (HTTP/1.1)│ │ (broadcast)│ │ (opt.) │ │ │
│ │ └──┬─────────┘ └────────────┘ └─────┬──────┘ │ │
│ └─────┼─────────────────────────────────────┼─────────┘ │
│ │ Unix socket │ TCP/TLS │
│ │ /run/vigil/vigild.sock │ 0.0.0.0:8443 │
│ │ │ │
│ ┌─────▼──────────────┐ ┌─────────────────▼──────────┐ │
│ │ vigil (CLI) │ │ vigil (CLI) / │ │
│ │ podman exec ... │ │ curl / K8s probes │ │
│ └────────────────────┘ └────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
A single binary, mode-switched by subcommand (app run starts the daemon,
app services talks to it), must always carry the full CLI argument
parsing and output formatting, even when running as PID 1 where none of that
code is ever needed.
vigil-rs's approach — strict separation:
vigild |
vigil |
|
|---|---|---|
| Role | Daemon / PID 1 | CLI client |
| Dependencies | axum, rustls, tokio, rcgen, reqwest (alerts, HTTP checks), … | hyper-util, reqwest |
| Runs as | PID 1 in container | podman exec / host / CI |
| Contains | API server, supervisor logic | HTTP client, output formatting |
| Transports | Unix socket + optional TLS TCP | Unix socket or HTTPS |
| Final image | Required | Optional |
Benefits of the split:
-
Smaller attack surface in production.
vigildin the container image does not contain CLI parsing code or any client-side formatting logic.vigilcan be excluded from images where no interactive access is needed. -
Independent versioning. The CLI and daemon can evolve at different speeds. A newer
vigilclient can talk to an oldervigildas long as the API contract (viavigil-types) holds. -
Shared types as the contract. The
vigil-typescrate is the single source of truth for all API request/response types. Both binaries depend on it — the daemon serializes with it, the client deserializes with it. Adding a field in one place propagates automatically to both sides with full compile-time verification. -
No PID 1 bloat. In production containers
vigildis PID 1 and runs for the entire container lifetime. Keeping it lean (no subcommand parsing, no output formatting) means fewer dependencies and a faster startup. -
Mirrors real-world patterns. Container orchestrators (Kubernetes, Nomad) interact with the daemon via the REST API, not the CLI. The CLI (
vigil) is a convenience tool for developers — it is optional and does not need to exist in a production image at all.
vigild uses an actor-per-service model built on Tokio:
main
└─ Overlord (tokio task)
├─ ServiceActor "haproxy" (tokio task, mpsc mailbox)
├─ ServiceActor "hug" (tokio task, mpsc mailbox)
├─ CheckActor "check-haproxy" (tokio task, mpsc mailbox)
└─ LogStore (broadcast channel → SSE clients)
- Overlord owns the plan and routes commands from the API to the right actor. It is the only component that reads layer YAML files.
- ServiceActors manage a single child process each: spawn, signal, wait,
backoff, restart. They are state machines (
Inactive,Starting,Active,Stopping,Backoff,Error,Donefor a finished oneshot). - CheckActors run health checks on a timer and emit events back to the
Overlord when a check transitions to
Down. - LogStore holds a ring buffer of recent log lines and a broadcast channel
for live SSE streaming to
vigil logs -fclients. - Service and check state belongs to its actor and changes only through
typed
mpsc/oneshotmessages. Stores shared with the API (log buffer, identities, metrics) use locks.
vigil-rs is dual-licensed:
| Use case | License |
|---|---|
| Open-source projects, internal infrastructure | AGPL-3.0 — free |
| Closed-source products, SaaS, corporate AGPL policy exception | Commercial |
AGPL-3.0 in short: You can use vigil-rs freely as long as modifications and derivative works that are made available over a network are published under the same license. If you embed vigil-rs in a closed-source product or do not want to open-source your service, you need a commercial license.
Contact al-virgilrs@none.at for commercial licensing.