Add Velero backup/restore for cert-manager secrets across cluster rebuilds - #20
Merged
Conversation
CNCF-incubating, Apache-2.0 (chosen over AppsCode's KubeStash to avoid any paid-tier ceiling). Scope is deliberately narrow: this cluster is otherwise 100% declarative from git, so the only thing worth backing up is the handful of Secrets Kubernetes controllers mint at runtime — specifically cert-manager's issued wildcard TLS cert and ACME account key, which get wiped on every full cluster rebuild and force a brand new Let's Encrypt order (what hit the prod rate limit on 2026-07-25). Own dedicated bucket + IAM identity (backup-velero-dev-id), not OpenBao's own backup-dev-id — sharing it broke OpenBao's snapshot script's retention cleanup. openbao namespace is excluded by omission (includedNamespaces is an allow-list). No volume snapshots — nothing in scope here lives on a PV, only in Secrets. apps/velero-init materializes Velero's S3 credentials from OpenBao via ESO, mirroring apps/external-dns-init's pattern. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CCgyeSfKWH6m5mmJEg6t43
A PreSync hook on gateway-config: waits for a completed Velero backup to sync, restores cert-manager's ACME account key + the gateway's wildcard TLS secret, then lets gateway-config's own sync proceed to create the ClusterIssuers/Gateway that would otherwise trigger a fresh ACME order. This is a hook on gateway-config itself rather than a separate wave-ordered Application, and deliberately so: ArgoCD's sync-wave ordering across sibling Applications is not a hard blocking guarantee (confirmed live across several full cluster rebuilds — a sibling Application stuck retrying didn't stop the next wave from proceeding regardless). A PreSync hook is a real guarantee, scoped to this Application's own sync lifecycle: the Gateway/ClusterIssuers cannot apply until the hook Job exits 0. The hook's wait budget is ~30min, not the ~10min first tried — the timeout exits 0 (success) by design so a genuinely-empty-bucket first boot doesn't block forever, but that same success path means there's no retry-based recovery once cert-manager issues a fresh cert off the back of it. A cold OpenBao -> ESO -> Velero credential chain measured ~15min end to end in testing, so a shorter budget couldn't tell "nothing to restore yet" apart from "not ready yet". existingResourcePolicy is deliberately left at its safe default (skip-if-exists) rather than "update": the ordering above already guarantees the target Secrets don't exist on a genuine fresh rebuild, so "update" would only ever matter for reprocessing against an already-running cluster — silently overwriting a live, possibly- renewed certificate with a stale one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CCgyeSfKWH6m5mmJEg6t43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cert-manager/gatewaynamespace Secrets: this cluster is otherwise 100% declarative from git, so the only thing worth backing up is the handful of Secrets Kubernetes controllers mint at runtime — specifically cert-manager's issued wildcard TLS cert and ACME account key, which get wiped on every full cluster rebuild and force a brand new Let's Encrypt order (what hit the prod rate limit on 2026-07-25).gateway-configwaits for a completed Velero backup to sync, restores the secrets, then letsgateway-config's own sync proceed to create theClusterIssuers/Gatewaythat would otherwise trigger a fresh ACME order. This is a hook ongateway-configitself rather than a separate wave-ordered Application — ArgoCD's sync-wave ordering across sibling Applications turned out not to be a hard blocking guarantee (confirmed live across several full cluster rebuilds), while a PreSync hook is a real guarantee scoped to that Application's own sync lifecycle.existingResourcePolicydeliberately left at its safe default (skip-if-exists), notupdate— confirmed live thatupdatewould silently overwrite a live, possibly-renewed certificate if this object were ever reprocessed against an already-running cluster.Test plan
helm lint/helm templateclean onvelero-initandgateway-configkubectl apply --dry-run=clientclean against the real scaleway-homelab clusterRestore: Completed,letsencrypt-prod-account-key/scalepack-fr-wildcard-tlsboth created within the same second (signature of a restore, not a fresh ACME issuance), zeroCertificateRequest/Order/Challengeobjects created🤖 Generated with Claude Code
https://claude.ai/code/session_01CCgyeSfKWH6m5mmJEg6t43