Skip to content
git001Public

About

vigil-rs is a PID 1 / container init daemon written in Rust. It supervises multiple processes, runs health checks, and exposes a REST API over a Unix socket with native zombie-reaping and per-service stop signals.

Topics

Resources

Contributing

Stars

27 stars

Watchers

0 watching

Forks

Repository files navigation

vigil-rs

vigil is a service supervisor and container init daemon written in Rust. It manages multiple processes inside a single container — starting, stopping, restarting, and health-checking them — and exposes a REST API over a Unix socket (and optionally HTTPS) for programmatic control. It runs unprivileged, e.g. on OpenShift with a random UID.

Important

Upgrading from v3? v4 changes defaults (environment, access over TLS, shutdown) and reads layer files strictly. See the migration guide.


Features

Feature Description
PID 1 / zombie reaper Safe to run as container entrypoint; reaps orphaned child processes automatically
YAML layer config Declarative service & check definitions; multiple layers are merged in order
Service supervision Start, stop, restart with exponential backoff; healthy-time resets the backoff
Oneshot jobs and cron schedules type: oneshot for migrations and jobs, schedule (cron, with time zone) instead of a cron daemon, run-timeout for jobs that hang
Readiness ready: process | notify | check: dependents start when a service is ready (sd_notify READY=1 or a passing check)
Custom stop signal Per-service stop-signal (SIGTERM, SIGUSR1, SIGHUP, …) + kill-delay
Startup ordering after: / before: / requires: dependency graph
Health checks HTTP, TCP, and exec checks with configurable period/timeout/threshold/delay; levels alive and ready
/v1/health One endpoint for Kubernetes liveness and readiness probes, no credentials needed (?level=alive|ready)
on-check-failure Automatic service restart (or shutdown) when a check goes down
Alerting HTTP(S) notifications on check state transitions, service failures (on-service) and invalid layers (on-vigild) — Alertmanager, CloudEvents, OTLP logs, or generic webhook; env:VAR values for url/headers/labels/proxy under the same pass-env* rules as services, checked when loading (an unset or not allowed variable is an error); proxy support; configurable retry with backoff; body-template uses minijinja (Jinja2-compatible) for custom webhook payloads
on-success / on-failure Per-service exit policies: restart, ignore, shutdown, failure-shutdown, success-shutdown
Exit-code propagation vigild exits with the managed service's actual exit code
Log streaming stdout/stderr captured per service; vigil logs and SSE follow (vigil logs -f)
Stack traces logs-multiline (java, python, indent or a regex) joins a stack trace and its log line into one entry; vigil-log-relay --multiline does the same for pod logs and vigild streams
Log push logs-push-socket / logs-push-addr: vigild streams ndjson to a collector (Filebeat, Fluent Bit, …)
vigil-log-relay Forwards vigild's log stream or Kubernetes pod logs to a TCP sink
Environment and secrets Services get only a minimal environment unless pass-env* allows more; minijinja templates ({{ file() }}, {{ mem() }}, {{ cpu() }}) and env-file in command and environment
watch-files Restart or signal a service when a mounted Secret or config file changes
Prometheus metrics GET /v1/metrics (OpenMetrics): services, checks, health, alerts, layer reloads
REST API Full HTTP API over Unix socket; Swagger UI at /docs (assets loaded from unpkg.com — requires internet access; use /openapi.json in air-gapped environments)
OpenAPI Auto-generated spec via utoipa; served at /openapi.json
TLS listener Optional HTTPS API (--tls-addr); auto-generates self-signed cert
Identity management Named principals with open/metrics/read/write/admin access levels (Unix UID, HTTP Basic, TLS client certificate); defined in layer files (password hash or CA from a file, e.g. a Kubernetes Secret) or added over the API
Signal forwarding SIGHUP / SIGUSR1 / SIGUSR2 forwarded to all running service process groups; vigil services signal SIGHUP nginx for single services
Replan Hot-reload layer YAML files without restarting the daemon; changed services restart; --watch-layers reloads by itself (e.g. a mounted ConfigMap); vigil plan shows the merged plan, secrets masked
Strict configuration An unknown key or invalid value in a layer file is an error that names it; nothing is silently ignored
mimalloc High-performance allocator enabled globally

Workspace layout

vigil-rs/
├── crates/
│   ├── vigil-types/      # Shared types (plan, API, identity, signal) — no async
│   ├── vigil-files/      # How all binaries read configured files (secrets, env files, certificates)
│   ├── vigil-tls/        # TLS material for all binaries: CA bundles, cert + key, client settings
│   ├── vigild/           # Daemon binary — axum HTTP server, overlord, service actors
│   ├── vigil/            # CLI binary — hyper (Unix socket) and reqwest (HTTPS) client
│   └── vigil-log-relay/  # Standalone log forwarder — K8s pods / HTTP / Unix socket → TCP sink
└── examples/
    ├── full-container/        # vigild + h2o (single service)
    ├── hug/                   # vigild + HAProxy + controller (multi-service)
    ├── kubernetes-pod-logs/   # vigil-log-relay + Filebeat on Kubernetes
    └── ...                    # filebeat, fluentbit, vector, php-caddy, quarkus, …

Quick start

Build

# Core binaries (daemon + CLI)
cargo build --release --bin vigild --bin vigil

# Include the log forwarder
cargo build --release --bin vigild --bin vigil --bin vigil-log-relay

# All workspace binaries
cargo build --release

Run the daemon

vigild --layers-dir /etc/vigil/layers

# Unprivileged (e.g. OpenShift): runtime files under /tmp/vigil
vigild --layers-dir /etc/vigil/layers --path-prefix /tmp/vigil

Warning

Keep --path-prefix short: at most 78 bytes. Unix socket paths are limited to 107 bytes by Linux, and vigild puts its sockets under the prefix. See Socket path limit.

Use the CLI

# Default: connect via Unix socket
vigil services list
vigil checks list
vigil checks run myservice-alive   # run a check now; exit code 1 when it fails
vigil logs -f
vigil services start myservice
vigil services restart myservice
vigil services stop myservice

# Same runtime directory as vigild (if not /run/vigil)
VIGIL_PATH_PREFIX=/tmp/vigil vigil services list

# Connect via HTTPS (vigild --tls-addr); needs an identity, see TLS docs
vigil --url https://myhost:8443 --cacert ca.pem --user deploy:secret services list

# HTTPS with auto-generated self-signed cert
vigil --url https://myhost:8443 --insecure --user deploy:secret services list

API via curl

# Unix socket
curl --unix-socket /run/vigil/vigild.sock http://localhost/v1/system-info
curl --unix-socket /run/vigil/vigild.sock http://localhost/v1/services

# HTTPS: open endpoints without credentials, the rest with an identity
curl -k https://myhost:8443/v1/health
curl -k -u deploy:secret https://myhost:8443/v1/services

Layer configuration

Services and health checks are defined in YAML layer files. Multiple layers are merged in order — later layers override earlier ones.

# /etc/vigil/layers/001-app.yaml
summary: My application

services:

  myapp:
    summary: Main application process
    command: /usr/local/bin/myapp --config /etc/myapp/config.yaml
    startup: enabled
    stop-signal: SIGTERM
    kill-delay: 10s
    on-success: restart
    on-failure: restart
    backoff-delay: 1s
    backoff-factor: 2.0
    backoff-limit: 30s
    on-check-failure:
      myapp-alive: restart

  sidecar:
    summary: Helper process (starts after myapp)
    command: /usr/local/bin/sidecar
    startup: enabled
    after:
      - myapp
    on-success: restart
    on-failure: restart

checks:

  myapp-alive:
    level: alive
    startup: enabled
    delay: 3s       # wait before first check (default: 3s)
    period: 10s
    timeout: 3s
    threshold: 3
    http:
      url: http://localhost:8080/healthz

Supported check types

# HTTP check
http:
  url: https://localhost:8080/healthz
  headers:                              # optional — arbitrary key/value map
    Authorization: "Bearer token"
  # tls:                                # optional: CAs, client certificate, insecure
  #   ca: /etc/vigil/internal-ca.pem
  # proxy:                              # optional; without it, a direct connection
  #   url: http://proxy.example.com:3128

# TCP check
tcp:
  host: localhost
  port: 5432

# Exec check (exit code 0 = healthy)
exec:
  command: pg_isready -U postgres
  service-context: myapp   # inherit env/user/group from service

Container usage

vigild is designed to run as PID 1 inside a container.

ENTRYPOINT ["/usr/local/bin/vigild", "--layers-dir", "/etc/vigil/layers"]

On OpenShift (random UID, /run not writable) set ENV VIGIL_PATH_PREFIX=/tmp/vigil; see Container Setup.

Interact with the running container:

podman exec <ctr> vigil services list
podman exec <ctr> vigil logs -f
podman exec <ctr> vigil services restart myapp

See the examples/ directory for complete, buildable examples.


Comparison: container init systems

vs. dumb-init / tini

dumb-init / tini vigil-rs
Purpose Minimal PID 1 shim Full service supervisor
Manages multiple processes ❌ (single process only) ✅
Restart on failure ❌ ✅ with backoff
Health checks ❌ ✅ HTTP / TCP / exec
HTTP check headers ❌ ✅ arbitrary key/value map
HTTP check TLS and proxy (tls: / proxy:) ❌ ✅ custom CAs, client certificate, proxy
Alerting (Alertmanager / CloudEvents / OTLP) ❌ ✅
Runtime API ❌ ✅ REST over Unix socket and HTTPS
Configuration None (CLI args) YAML layers
Signal forwarding ✅ ✅
Zombie reaping ✅ ✅
Binary size ~20 KB ~16 MB (vigild, stripped)

When to use dumb-init/tini: You have exactly one process and only need zombie reaping + signal forwarding. Zero configuration overhead.

When to use vigil: You run multiple processes and want health checks, automatic restarts, and programmatic control.


vs. s6-overlay

s6-overlay vigil-rs
Language C Rust
Configuration Directory-based (/etc/s6-overlay/s6-rc.d/) YAML layers
Health checks ❌ (external tooling) ✅ built-in
Readiness ✅ notification fd ✅ ready: notify | check
HTTP check headers ❌ ✅ arbitrary key/value map
HTTP check TLS and proxy (tls: / proxy:) ❌ ✅ custom CAs, client certificate, proxy
Alerting (Alertmanager / CloudEvents / OTLP) ❌ ✅
Runtime API ❌ (CLI tools s6-svc, s6-rc) ✅ REST over Unix socket and HTTPS
Log routing ✅ s6-log per service, rotation to files log buffer, API stream, ndjson push to collectors, vigil-log-relay, stack traces as one entry
Dependency ordering ✅ (dependencies.d/) ✅ (after: / before: / requires:)
Custom stop signal ✅ down-signal file, timeout-kill ✅ stop-signal, kill-delay
Used by linuxserver.io images -

When to use s6-overlay: Classic multi-service containers (nginx + cron + sshd) where you don't need runtime control. Battle-tested, extremely small.

When to use vigil: You need health checks, per-service restart policies, dynamic reconfiguration (replan), and a REST API to query or control services from outside the container.


vs. supervisord / go-supervisord

supervisord go-supervisord vigil-rs
Language Python Go Rust
Configuration INI file (+ [include]) INI file (+ [include]) YAML layers, merged in order
Custom stop signal ✅ stopsignal ✅ stopsignal ✅ stop-signal
Startup ordering priority only ✅ depends_on ✅ after / before / requires
Health checks ❌ ✅ liveness: HTTP, TCP, script; restart, stop or script on failure ✅ HTTP / TCP / exec, levels alive / ready, /v1/health
HTTP check headers, TLS, proxy ❌ headers, client cert, CA ✅ headers, custom CAs, client certificate, proxy
Alerting via event listeners (e.g. superlance) via event listeners ✅ Alertmanager / CloudEvents / OTLP / webhook
API XML-RPC over HTTP (Unix socket or TCP) XML-RPC + REST, web GUI JSON REST over Unix socket and HTTPS, OpenAPI
Prometheus metrics ❌ ✅ ✅
Zombie reaping as PID 1 ✅ waitpid(-1) reaps every child ✅ ✅
Layer merging / replan ❌ (reread / update) ❌ ✅

vs. Canonical Pebble

vigil-rs started as a Rust rewrite of Pebble. The layer format and the API look alike, but they are not interchangeable (see below).

Pebble vigil-rs
Language Go Rust
PID 1 / zombie reaper ✅ always a child subreaper ✅ as PID 1 or with --reaper
Custom stop signal ❌ ✅ stop-signal
Oneshot services, cron schedules ❌ ✅ type: oneshot, schedule
Readiness ❌ (a start succeeds after 1 s) ✅ ready: process | notify | check
Templating, env files, environment isolation ❌ (services inherit the full environment) ✅ minijinja, env-file, pass-env*
Exit code of the daemon 0, 10 (service failure), 11 (check failure) the service's own exit code (--exit-code propagate), or 0/10
HTTP check headers ✅ ✅
HTTP check TLS and proxy (tls: / proxy:) ❌ ✅ custom CAs, client certificate, proxy
Check delay field ❌ ✅
/v1/health endpoint ✅ ✅ (404 for a name that is not a check)
Log targets ✅ Loki, OpenTelemetry, syslog ndjson push per service (Unix/TCP), vigil-log-relay
Alerting (webhook / Alertmanager / CloudEvents / OTLP) ❌ ✅
Signals to single services (API) ✅ ✅ POST /v1/signals
exec and files API; notices ✅ ❌
Asynchronous changes (/v1/changes, /wait) ✅ ❌ (service actions are synchronous)
TLS API listener ✅ --https ✅ --tls-addr, optional mTLS
Identities API, persisted API (in memory) and layer files
Prometheus metrics ✅ ✅ (more series)
OpenAPI / Swagger UI ❌ ✅

Moving from Pebble. Simple layer files load unchanged, but some keys behave differently:

  • Checks without startup: do not run in vigil-rs (Pebble starts them); set startup: enabled.
  • backoff-limit is where vigil-rs gives up (state error); Pebble keeps retrying at that interval.
  • requires orders the start and stops the dependent when the dependency stops; it does not start the dependency.
  • Services get only PATH, TZ, HOME, USER and LOGNAME from vigild's environment unless pass-env* allows more.
  • log-targets:, workloads: and layer subdirectories (NNN-name.d/) are not supported.
  • The REST API has the same shape for /v1/services, /v1/checks and /v1/system-info, but /v1/logs returns a JSON envelope (with follow=true an NDJSON stream without one), service actions return a finished change instead of an asynchronous one, and unknown fields and query parameters are rejected. Pebble clients need adjusting.

Examples

examples/full-container — vigild + h2o

Single web server supervised by vigild. Demonstrates health checks, on-check-failure restart, and the vigil CLI.

podman build -f examples/full-container/Containerfile -t vigil-h2o .
podman run --rm --network host --name vigil-h2o vigil-h2o

examples/kubernetes-pod-logs — vigil-log-relay + Filebeat on Kubernetes

Runs vigild (service supervisor) and vigil-log-relay (K8s pod log forwarder) as sidecars alongside Filebeat in a single pod. vigil-log-relay watches a namespace via the K8s Watch API, streams pod logs, and forwards them as ndjson to Filebeat over a local TCP connection.

K8s stream query parameter (stdout/stderr separation)

Kubernetes ≥ 1.32 knows a stream=Stdout / stream=Stderr query parameter on the pod log endpoint, allowing stdout and stderr to be collected as separate labeled streams. It only works with the feature gate PodLogsQuerySplitStreams, which is off by default (also on 1.35), and some distributions forbid it via admission control.

On K8s ≥ 1.32 vigil-log-relay therefore tries the parameter first. If the API server answers HTTP 422 "stream: Forbidden: may not be specified", the relay logs one warning and switches to the combined stream ("stream": "output") for all pods; no log lines are lost. To skip that first attempt, set --no-stream-param (or NO_STREAM_PARAM=true):

env:
  - name: NO_STREAM_PARAM
    value: "true"

HTTP/2 multiplexing

The kube client (hyper + rustls-tls) currently uses HTTP/1.1 — one TCP connection per pod stream. HTTP/2 multiplexing is tracked in kube-rs PR #1823 (draft). Use --max-log-requests to cap concurrent connections when watching large namespaces.

examples/hug — vigild + HAProxy + controller

Two services: HAProxy (with graceful SIGUSR1 drain) and a controller that starts after HAProxy. Demonstrates:

  • after: startup ordering
  • stop-signal: SIGUSR1 for graceful drain
  • exec health check via HAProxy Unix socket
  • on-check-failure restart policy
podman build -f examples/hug/Containerfile -t vigil-hug .
podman run --rm --network host --name vigil-hug vigil-hug

Interact:

podman exec vigil-hug vigil services list
podman exec vigil-hug vigil checks list
podman exec vigil-hug vigil logs

Testing

Unit and integration tests

cargo test --workspace

VTest2 end-to-end tests

End-to-end integration tests are written in the VTest2 .vtc format — the same test framework used by the Varnish HTTP cache. Each test starts a real vigild process (and sometimes vigil-log-relay) with a temporary layers directory and Unix socket, drives it through the HTTP API and the vigil CLI, and asserts on the responses.

32 test files cover: system info, service/check/alert/identity CRUD, log streaming and follow, TLS listener, reaper, mTLS, alert body templates, log-relay socket and URL sources, TCP reconnect, JSON log format, HTTPS transport, service alerts, layer watching, shutdown exit codes, identities in layer files, check failure reasons and metrics, signals, replan restarts, request validation, and more.

# Run all VTest2 tests (requires VTest2 binary)
tests/vtest/run.sh

# Single test, verbose
tests/vtest/run.sh -v tests/vtest/v0001_system_info.vtc

# Combined coverage report (unit + integration + VTest2)
tests/vtest/run.sh --coverage          # text summary
tests/vtest/run.sh --coverage --html   # HTML report

# vigil-log-relay --kubernetes against a real cluster (creates and removes
# the namespace vigil-relay-test)
KUBECONFIG=~/.kube/test-cluster.yml tests/k8s/relay.sh

# Coverage including the cluster test
K8S_TEST=1 KUBECONFIG=~/.kube/test-cluster.yml tests/vtest/run.sh --coverage

See tests/vtest/README.md for prerequisites, patterns, and gotchas.

Coverage

Line coverage across all crates: ~96 % (measured with llvm-cov via K8S_TEST=1 tests/vtest/run.sh --coverage). Without a cluster (K8S_TEST unset) the source_k8s module of vigil-log-relay stays uncovered and the total is about 93.5 %.


Architecture

Two binaries: vigild and vigil

vigil-rs deliberately splits into two separate binaries (plus the optional vigil-log-relay) instead of one binary that acts as daemon or client depending on the subcommand.

┌──────────────────────────────────────────────────────────────┐
│  Container / Host                                            │
│                                                              │
│  ┌─────────────────────────────────────────────────────┐     │
│  │  vigild  (PID 1)                                    │     │
│  │                                                     │     │
│  │  ┌────────────┐   ┌────────────┐   ┌────────────┐   │     │
│  │  │  Overlord  │   │  Service   │   │   Check    │   │     │
│  │  │  (actor)   │──▶│  Actors    │   │   Actors   │   │     │
│  │  └─────┬──────┘   └────────────┘   └────────────┘   │     │
│  │        │                                            │     │
│  │  ┌─────▼──────┐   ┌────────────┐   ┌────────────┐   │     │
│  │  │  axum API  │   │  LogStore  │   │  TLS API   │   │     │
│  │  │  (HTTP/1.1)│   │ (broadcast)│   │  (opt.)    │   │     │
│  │  └──┬─────────┘   └────────────┘   └─────┬──────┘   │     │
│  └─────┼─────────────────────────────────────┼─────────┘     │
│        │ Unix socket                          │ TCP/TLS      │
│        │ /run/vigil/vigild.sock               │ 0.0.0.0:8443 │
│        │                                      │              │
│  ┌─────▼──────────────┐     ┌─────────────────▼──────────┐   │
│  │  vigil  (CLI)      │     │  vigil  (CLI) /            │   │
│  │  podman exec ...   │     │  curl / K8s probes         │   │
│  └────────────────────┘     └────────────────────────────┘   │
└──────────────────────────────────────────────────────────────┘

Why two binaries instead of one?

A single binary, mode-switched by subcommand (app run starts the daemon, app services talks to it), must always carry the full CLI argument parsing and output formatting, even when running as PID 1 where none of that code is ever needed.

vigil-rs's approach — strict separation:

vigild vigil
Role Daemon / PID 1 CLI client
Dependencies axum, rustls, tokio, rcgen, reqwest (alerts, HTTP checks), … hyper-util, reqwest
Runs as PID 1 in container podman exec / host / CI
Contains API server, supervisor logic HTTP client, output formatting
Transports Unix socket + optional TLS TCP Unix socket or HTTPS
Final image Required Optional

Benefits of the split:

  1. Smaller attack surface in production. vigild in the container image does not contain CLI parsing code or any client-side formatting logic. vigil can be excluded from images where no interactive access is needed.

  2. Independent versioning. The CLI and daemon can evolve at different speeds. A newer vigil client can talk to an older vigild as long as the API contract (via vigil-types) holds.

  3. Shared types as the contract. The vigil-types crate is the single source of truth for all API request/response types. Both binaries depend on it — the daemon serializes with it, the client deserializes with it. Adding a field in one place propagates automatically to both sides with full compile-time verification.

  4. No PID 1 bloat. In production containers vigild is PID 1 and runs for the entire container lifetime. Keeping it lean (no subcommand parsing, no output formatting) means fewer dependencies and a faster startup.

  5. Mirrors real-world patterns. Container orchestrators (Kubernetes, Nomad) interact with the daemon via the REST API, not the CLI. The CLI (vigil) is a convenience tool for developers — it is optional and does not need to exist in a production image at all.

Internal daemon architecture

vigild uses an actor-per-service model built on Tokio:

main
 └─ Overlord (tokio task)
     ├─ ServiceActor "haproxy"  (tokio task, mpsc mailbox)
     ├─ ServiceActor "hug"      (tokio task, mpsc mailbox)
     ├─ CheckActor  "check-haproxy" (tokio task, mpsc mailbox)
     └─ LogStore    (broadcast channel → SSE clients)
  • Overlord owns the plan and routes commands from the API to the right actor. It is the only component that reads layer YAML files.
  • ServiceActors manage a single child process each: spawn, signal, wait, backoff, restart. They are state machines (Inactive, Starting, Active, Stopping, Backoff, Error, Done for a finished oneshot).
  • CheckActors run health checks on a timer and emit events back to the Overlord when a check transitions to Down.
  • LogStore holds a ring buffer of recent log lines and a broadcast channel for live SSE streaming to vigil logs -f clients.
  • Service and check state belongs to its actor and changes only through typed mpsc / oneshot messages. Stores shared with the API (log buffer, identities, metrics) use locks.

License

vigil-rs is dual-licensed:

Use case License
Open-source projects, internal infrastructure AGPL-3.0 — free
Closed-source products, SaaS, corporate AGPL policy exception Commercial

AGPL-3.0 in short: You can use vigil-rs freely as long as modifications and derivative works that are made available over a network are published under the same license. If you embed vigil-rs in a closed-source product or do not want to open-source your service, you need a commercial license.

Contact al-virgilrs@none.at for commercial licensing.

About

vigil-rs is a PID 1 / container init daemon written in Rust. It supervises multiple processes, runs health checks, and exposes a REST API over a Unix socket with native zombie-reaping and per-service stop signals.

Topics

Resources

Contributing

Stars

27 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages