Grafana loses all data on every cluster rebuild despite Velero backup

[nbrieussel]

Symptom

Every fresh cluster rebuild gives an empty Grafana — no dashboards, no manually-created content — despite a Velero backup schedule (grafana-data) specifically set up for this. Confirmed recurring, not a one-off.

What's already been ruled in / found (live-debugged 2026-08-20, not yet fixed)

Two separate persistence mechanisms are in play, and they need to be understood separately:

1. Terraform-managed Grafana config (06-monitoring/grafana/bootstrap + 06-monitoring/grafana/managed) — this is NOT stored on Grafana's PVC at all, it's stored as Grafana API objects (service accounts, datasources, dashboards) that Terraform creates via the grafana provider, tracked in Terraform state (Scaleway-hosted, survives cluster rebuilds fine) and replayed hourly by the terraform-apply CronWorkflows (grafana-bootstrap at :20, grafana-managed at :30). In theory this alone should make Grafana's declared config (SAs, Prometheus/Loki/Tempo datasources, grafana_dashboard.defaults) self-healing on every rebuild, independent of the PVC. Not yet confirmed whether this reconciliation actually runs cleanly end-to-end on a fresh cluster — worth checking CronWorkflow run history/logs for grafana-bootstrap/grafana-managed on the next rebuild.

2. Anything NOT Terraform-managed (dashboards/panels/alerts a human adds by hand via the UI, user accounts, preferences) — this only exists in Grafana's own SQLite DB on its PVC (services/platform/monitoring/chart's kube-prometheus-stack.grafana.persistence), and depends entirely on the Velero restore actually working. This is the part that looks broken:

   • Velero's grafana-data schedule (gitops repo services/platform/velero/chart/values-scaleway.yaml) uses the kopia node-agent file-system backup (not CSI snapshots) — deployNodeAgent: true, scoped via a labelSelector on the Grafana pod + that pod's backup.velero.io/backup-volumes: storage annotation. The Grafana pod's own annotation looks correct (confirmed live).
   • The restore side (gitops repo services/platform/monitoring/grafana-chart/templates/grafana-restore-configmap.yaml, a PreSync-hook Job) creates a Velero Restore object scoped to "includedResources": ["persistentvolumeclaims", "persistentvolumes"] — it does not include pods.
   • Per Velero's own restore mechanics (confirmed via docs research), kopia/restic file-system-backup restore is triggered when the restore process encounters a Pod that has an associated PodVolumeBackup — it injects an init container into that restoring pod to run the actual PodVolumeRestore (the file-level data copy). Without pods in the Restore's includedResources, there is nothing to trigger that mechanism — the PVC/PV objects get recreated (bound, correct size), but the underlying volume data is very likely never actually copied back. This matches the symptom exactly: PVC exists, restore Job reports "Completed", but Grafana comes up empty.
   • Confirmed live: the grafana-pvc-restore Restore object existed and its owning Job reported success, yet the resulting PVC (grafana, 2Gi, Bound) is effectively empty from Grafana's perspective.

Working hypothesis: fix likely involves either (a) including pods in the Restore's includedResources (needs care — Grafana's pod is Deployment-managed, so restoring a raw Pod object mid-Helm-sync may conflict/be ignored; needs testing against Velero's documented pattern for this), or (b) switching this specific backup from kopia fs-backup to CSI VolumeSnapshot-based backup/restore (doesn't need a pod at restore time, works directly against the PV), or (c) some other Velero-documented pattern for exactly this "restore-before-the-owning-workload-exists" PreSync-hook use case. Needs a proper read of https://velero.io/docs/main/file-system-backup/ and https://velero.io/docs/main/restore-reference/ before picking an approach.

Other prod-readiness follow-ups surfaced this session (not blocking, tracked here for later)

• Stale letsencrypt-staging CA overrides never removed after the Gateway's cert flipped back to letsencrypt-prod (2026-08-20, gitops repo services/platform/gateway/config/values-scaleway.yaml) — three spots still hardcode the old staging CA chain for server-side OIDC calls to Dex, which now breaks those integrations instead of fixing them:
  • infrastructure repo 10-cluster/scaleway/argocd.tf (rootCA)
  • gitops repo services/platform/monitoring/grafana-config (tls_client_ca + configmap, enabled: true in values-scaleway.yaml)
  • infrastructure repo 05-secrets/openbao/managed/main.tf (oidc_discovery_ca_pem)
• No robust mechanism to switch the cluster-wide cert issuer (prod ↔ staging) without hand-editing N hardcoded CA bundles across two repos every time — worth designing something more resilient (e.g. trust the system CA store by default and only special-case staging behind one toggle) instead of the current copy-pasted-PEM-per-consumer pattern.
• Argo Workflows UI: fetching a workflow's logs still 403s (/api/v1/workflows/.../log) even after the SSO RBAC fix — likely missing pods/log in the argo-workflows-admin ClusterRole. Narrower issue, rest of the UI works.
• terraform-apply's wave-6 placement (carried over from Argo Workflows CronWorkflow automation + SSO UI: tracking #73) is still a blunt fix — openbao-managed doesn't actually need to wait that long (only depends on OpenBao itself, wave 0); proper per-root dependency/health gating is real follow-up work.
• Orphaned Secret argo-workflows-sso-admin-token (old, wrongly-named SA-token Secret from before the RBAC fix) may still be sitting unpruned in the argo-workflows namespace — minor cleanup, verify and delete if so.

Context

Two PRs already merged archiving progress up to this point: infra #71, gitops #37. This cluster (and this whole setup) is explicitly not prod-ready yet — this issue exists to keep the remaining gaps visible rather than lost once the session context clears.