Makes the stack operable, not just runnable: authentication, self-monitoring,
alert delivery, backups, and infrastructure as code.
Added
- Authenticated ingestion: OTLP now requires a bearer token
(OTLP_AUTH_TOKEN). - Self-monitoring: node-exporter, a Stack Health dashboard, and a
disk-space alert that backstops Loki and Tempo (neither can cap its own
size). - Alert delivery: Alertmanager posts to any webhook
(ALERT_WEBHOOK_URL), and an always-firingWatchdogpings a dead man's
switch (HEARTBEAT_URL) so even a dead host gets noticed. - Backup and restore:
just backup/just restore, tested by wiping
every volume and recovering. - The Cloudflare edge as code:
infra/(OpenTofu) manages the tunnel,
DNS, and an email-PIN Access policy in front of Grafana. - Operator and onboarding docs: a runbook plus copy-paste telemetry
templates for new projects. - Logs Overview dashboard for OTLP logs, an Active Alerts table on Stack
Health, and Stack Health as the Grafana home page.
Changed
- Prometheus storage is now capped by size as well as time (15 GB).
- Tempo upgraded to 3.0, migrating off the removed ingester/compactor
config and verified in place against live traffic.