observability.example.yaml

---
# Template for secrets/observability.sops.yaml.
#
# Copy, fill in real values, and encrypt:
#
#   make secrets-init          # first time only — creates the age keypair
#   make secrets-edit          # opens the encrypted file in $EDITOR
#
# The encrypted result is committed; this template is not a secret and stays
# in plaintext so the required keys are discoverable without a decryption key.

# Grafana initial admin login. Changing this after first boot has no effect —
# Grafana only reads it when it creates the admin user. Change it in the UI, or
# wipe the grafana-data volume.
GRAFANA_ADMIN_USER: admin
GRAFANA_ADMIN_PASSWORD: change-me-to-something-long

# Shared secret between Grafana and the screenshot renderer (`make screenshots`).
# Grafana refuses to start at all once a rendering server is configured and this
# is left at its built-in default, so it is required even though the renderer
# itself only runs during a capture. Generate one with `make gen-secret`.
GRAFANA_RENDERER_TOKEN: change-me-to-something-long

# speedtest-tracker's Laravel APP_KEY (#914): it encrypts the app's sessions
# and stored credentials. Must be "base64:" followed by 32 random bytes in
# base64 — gen-secret.sh cannot make that shape. Set it without the value ever
# reaching the terminal:
#   sops set secrets/observability.sops.yaml '["SPEEDTEST_APP_KEY"]' "\"base64:$(openssl rand -base64 32)\""
# Changing it later logs everyone out and makes stored notification
# credentials unreadable; the test results themselves are not encrypted.
SPEEDTEST_APP_KEY: base64:change-me

# The ingest proxy's bearer tokens (#182, ADR-0067). Prometheus and Loki are
# published on loopback only, and `caddy` serves 9090 and 3100 on this host's
# address to a client that presents one of these.
#
# One per agent, so one host can be revoked without re-keying the others. An
# agent token can push (remote-write, Loki push) and do nothing else. The
# reader token can query and do nothing else. It is the one Glance and Home
# Assistant on trinity use, so it is also copied into
# secrets/sensitive.sops.yaml (GLANCE_PROMETHEUS_TOKEN, and
# HA_PROMETHEUS_AUTHORIZATION as "Bearer <token>"). Neither role can reach the admin API or Loki's delete
# API.
#
# render-config.sh refuses a token under 32 characters, or two that are equal.
# Generate four at once:
#
#   make gen-secret ARGS="--length 48 --count 4"
#
# scripts/deploy-agent.sh reads INGEST_TOKEN_<HOST> for the host it deploys to,
# with the host upper-cased and anything that is not a letter or digit mapped
# to `_`. A new host needs a key here and a line in stacks/observability/
# Caddyfile (docs/runbooks/add-monitored-device.md).
INGEST_TOKEN_ORACLE: change-me-to-something-long-oracle-0000000000000
INGEST_TOKEN_TRINITY: change-me-to-something-long-trinity-000000000000
INGEST_TOKEN_SARUMAN: change-me-to-something-long-saruman-000000000000
INGEST_TOKEN_READER: change-me-to-something-long-reader-0000000000000

# One SNMP community per device, not one shared across all of them.
#
# SNMPv2c sends the community in cleartext on the wire, so anyone with a port
# on the management VLAN can read it off a single packet. A distinct string per
# device limits what one captured packet grants, and lets you rotate a single
# device without an outage across the whole fleet.
#
# These grant read access to firewall rules, interface counters and hardware
# health. Treat them as credentials, not as configuration.
SNMP_COMMUNITY_PFSENSE: change-me-pfsense
SNMP_COMMUNITY_MOKERLINK: change-me-mokerlink

# A device polled over SNMPv3 authPriv (ADR-0036) has two keys instead of a
# community — its authentication and privacy passphrases — named after the
# same auth label. The iLO and the UPS card both moved on 2026-09-20; the
# firewall and the switch stay on v2c, for reasons the ADR gives, so the two
# shapes above and below are both live. The user name is not a secret and
# lives in snmp-exporter/generator.yaml. `make gen-secret ARGS=--snmp` prints
# whichever shape each device currently has.
SNMP_AUTHPASS_APC: change-me-apc-auth
SNMP_PRIVPASS_APC: change-me-apc-priv
SNMP_AUTHPASS_ILO: change-me-ilo-auth
SNMP_PRIVPASS_ILO: change-me-ilo-priv

# Where Alertmanager delivers notifications. Any webhook receiver works — ntfy,
# Gotify, a Home Assistant automation, or a Discord/Slack incoming webhook.
#
# THREE URLS, THREE DESTINATIONS. These were one URL under three receiver names
# until #66, which made `urgent` and `default` differ only in how often they
# repeated: a UPS on battery and a slow scrape arrived in the same place. The
# routing tree in alertmanager/alertmanager.yaml decides which alert is urgent;
# only a separate destination can make that difference audible, because that is
# where the per-topic sound and do-not-disturb settings live.
#
# Since #136 these three are the sensitive tier's own ntfy, one topic each.
# The topic names are NTFY_TOPIC_ALERTS / _URGENT / _SECURITY in
# secrets/sensitive.sops.yaml, where ntfy grants access to them; copy each
# name here, after the host, and end every URL with ?template=homelab, the
# repository's own rendering in stacks/sensitive/ntfy/templates/, so the phone
# shows a title and one line instead of JSON. Nothing about the
# token goes in the URL — it is ALERTMANAGER_NTFY_TOKEN below, sent as a
# header, because Caddy's access log records the URL.
ALERTMANAGER_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-alerts-topic?template=homelab
ALERTMANAGER_URGENT_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-urgent-topic?template=homelab
ALERTMANAGER_SECURITY_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-security-topic?template=homelab

# The same urgent and security channels, AGAIN, on ntfy.sh — for a phone that
# is not on the home network and so cannot reach the in-house server (#136).
# On ntfy.sh the topic name is the credential (SECURITY.md), so these are two
# long random names, and they are the topics these two channels used before
# the in-house server existed: moving the old values here is the whole
# migration. The bare topic URL and nothing after it: render-config.sh
# appends the inline template it builds from
# stacks/sensitive/ntfy/templates/homelab.yml, replacing any query here, so
# the ntfy.sh copies look like the in-house ones.
ALERTMANAGER_URGENT_EXTERNAL_URL: https://ntfy.sh/change-me-to-a-long-random-topic
ALERTMANAGER_SECURITY_EXTERNAL_URL: https://ntfy.sh/change-me-to-another-long-random-topic

# The bearer token the three in-house receivers present, and the SAME value as
# NTFY_ALERTMANAGER_TOKEN in secrets/sensitive.sops.yaml — two copies because
# the two hosts' age keys open two different files, on purpose. `tk_` and 29
# lower-case letters and digits; make it on any host with docker:
#
#   docker run --rm "$(COMPOSE_FILE=stacks/sensitive/compose.yaml \
#     ./scripts/image-for.sh ntfy)" token generate
#
# It can only publish to the three topics, and cannot read them.
ALERTMANAGER_NTFY_TOKEN: tk_changemechangemechangeme00

# The dead man's switch (#67). NOT an ntfy topic — this is a cron-monitor ping
# URL from something that lives OFF this host: healthchecks.io, Cronitor, an
# Uptime Kuma push monitor, or your own endpoint on another machine.
#
# The Watchdog rule fires forever and Alertmanager posts here every 5 minutes.
# Its arrival means nothing. Its absence means the alerting path is broken, and
# only something outside this stack can notice that — a watcher on the
# monitoring host fails at the same moment as the thing it is watching.
#
# Set the external check's period to 5m and its grace to 15m, and read
# docs/runbooks/verify-the-alert-path.md before changing either: the period is
# tied to `repeat_interval` on the heartbeat route in alertmanager.yaml, and
# raising one without the other produces a check that alarms on a healthy stack
# or one that never alarms at all.
ALERTMANAGER_HEARTBEAT_URL: https://hc-ping.example.invalid/00000000-0000-0000-0000-000000000000