fix(bootstrap): unify scaleway into one 6-stage RollingSync graph - #24
Draft
nbrieussel wants to merge 4 commits into
Draft
fix(bootstrap): unify scaleway into one 6-stage RollingSync graph#24nbrieussel wants to merge 4 commits into
nbrieussel wants to merge 4 commits into
Conversation
…edential race Velero/external-dns/Dex/Grafana were deployed via the un-waved services-vendor-scaleway recurse Application while the OpenBao-sourced Secrets they need come from services-app-scaleway's wave-respecting ApplicationSet — two independent sync graphs with no ordering guarantee between them, so on a fresh cluster boot these pods raced OpenBao's restore/unseal and started before their credentials existed (self-healing via selfHeal/kubelet retries, but noisy and non-deterministic). Migrate all four into small local wrapper charts (Chart.yaml `dependencies:` on the real chart, same pattern services/products/demo/chart already uses) driven through the app.yaml/ApplicationSet graph, with sync-waves placing them after the app that produces their Secret (wave 4 after the wave-3 -init apps for velero/external-dns; wave 6 after the wave-5 dex-init/ grafana-config). Verified helm template output is byte-identical to the old vendor charts aside from the expected subchart Source: path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZMCuQ5t9hdrmUctLk7hEv
…licationSet The previous commit's per-file serverSideApply conditional broke `helm template` on the bootstrap chart entirely (confirmed live: the bootstrap Application never got past ComparisonError, so nothing downstream ever synced). Two mistakes: a bare if/end directive mid-array isn't valid YAML once Helm's own render pass parses its output, and Helm's template engine scans every line for action delimiters regardless of '#' comments, so the explanatory comment describing the broken syntax also broke rendering. ApplicationSet's goTemplate mode substitutes values inside existing string fields per generated app — it can't conditionally add/remove a syncOptions array entry per file at all, so there's no per-file opt-in to build here. Made ServerSideApply=true unconditional for every app.yaml-driven Application instead (ArgoCD's own recommended default), and dropped the now-dead serverSideApply field from dex/monitoring's chart.app.yaml. Verified with `helm template` for both env=scaleway and env=local. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZMCuQ5t9hdrmUctLk7hEv
…ually orders apps Confirmed live 2026-08-11 that moving Velero/external-dns/Dex/Grafana into the app.yaml/ApplicationSet graph (previous commits) did not by itself fix the OpenBao credential race: sync-wave annotations on Applications generated by an ApplicationSet are inert — each is created and auto-synced independently, with no ordering across them, unlike the classic app-of-apps pattern (a single parent Application managing child Applications as its own resources) that services-vendor-scaleway actually relies on and where sync-wave genuinely works. velero's Application synced before velero-init's own sync (which creates the Secret it needs) had even finished. Requires argocd.argoproj.io/applicationset-controller's Progressive Syncs beta feature (--enable-progressive-syncs, added to the ArgoCD Helm release in the infra repo's 10-cluster/scaleway/argocd.tf). Adds a `wave` label (same value as the existing syncWave field) to every generated Application and a RollingSync strategy with one step per wave 0-6, so group N+1 only starts syncing once every Application in group N is Healthy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZMCuQ5t9hdrmUctLk7hEv
3 tasks
Collapse services-vendor-scaleway (plain recurse Application) and services-app-scaleway (ApplicationSet) into a single services-scaleway ApplicationSet. The two-graph split meant a vendor chart (external Helm repo) could never be ordered against the in-house -init/-config app that fed it OpenBao secrets, no matter what sync-wave value either carried. Every scaleway chart, vendor or not, now goes through the same wrapper-chart + *.app.yaml pattern (extends the previous commits' fix for Velero/external-dns/Dex/Grafana to cert-manager, its Scaleway DNS01 webhook, OpenBao itself, the ESO operator, and Envoy Gateway). Found a second, previously undetected instance of the exact same race in the process: the DNS01 webhook consumes its OpenBao-sourced Secret via `secret.externalSecretName` rather than `existingSecret`/`secretKeyRef`, so it didn't match the grep pattern used to scope the original fix. Ordering is six coarse stages instead of a per-app wave number: openbao -> eso -> velero -> platform -> monitoring -> products, each matching how this platform's own products actually depend on each other (a product's chart and its -init/-config companion share a stage and sync concurrently as one unit; only cross-stage ordering is enforced). Replaces the syncWave field with `stage` throughout; the RollingSync strategy groups on a `stage` label rather than the sync-wave annotation, which does nothing across separate ApplicationSet-generated Applications regardless. CLAUDE.md updated: local keeps the original two-format split, scaleway does not. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZMCuQ5t9hdrmUctLk7hEv
nbrieussel
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause of the original report (Velero and others failing to find credentials on a fresh cluster boot):
services-vendor-scaleway(plain recurse Application, external Helm charts) andservices-app-scaleway(ApplicationSet, in-house charts) were two completely uncoordinated ArgoCD sync graphs — no ordering was possible between a vendor chart and the in-house-init/-configapp that fed it OpenBao secrets, no matter what sync-wave number either carried. Confirmed live multiple times over the course of this PR, including that a plainsync-waveannotation does nothing at all across separate Applications generated by an ApplicationSet (only real fix: ArgoCD's Progressive Syncs /RollingSyncstrategy, label-based grouping, requires--enable-progressive-syncs— see companion infra PR IntegratedDynamic/infrastructure#55).Final design, collapsing everything into one
services-scalewayApplicationSet:Chart.yamldependencies:on the real chart) + a small*.app.yamlparam file, same patternservices/products/demo/chartalready used for a local dependency.stage:field, not a per-app wave number) matching how this platform's own products depend on each other:openbao -> eso -> velero -> platform (wireguard/dex/external-dns/gateway) -> monitoring -> products. A product's chart and its-init/-configcompanion share a stage and sync concurrently as one unit; only cross-stage ordering is enforced (RollingSyncwaits for every Application in stage N to be Healthy before starting stage N+1).secret.externalSecretName(notexistingSecret/secretKeyRef), so it didn't match the grep pattern the original fix was scoped with. Now migrated and correctly placed in theesostage.CLAUDE.mdupdated to document the split:localkeeps the original two-format*.vendor.yaml/*.app.yamlsystem,scalewaydoes not.Verification
# Source:path change. One accepted cosmetic exception: kube-prometheus-stack's bundled Grafana dashboards reformat one embedded PromQL query's whitespace when nested one level deeper (same content, non-functional, confined to dashboard JSON).helm templateon thebootstrapchart itself succeeds for bothenv=scalewayandenv=local.Test plan
terraform applyof10-cluster/scalewaypointed at this branchkubectl get application -n argocd -l stage=<name>shows each stage's Applications only start syncing after the previous stage is fully HealthyCrashLoopBackOff/CreateContainerConfigErrorwhile OpenBao is still restoring🤖 Generated with Claude Code
https://claude.ai/code/session_017ZMCuQ5t9hdrmUctLk7hEv