fix(bootstrap): unify scaleway sync ordering into one sync-wave graph

CLAUDE.md

@@ -19,16 +19,22 @@ The companion infrastructure repo (Terraform + Minikube) bootstraps ArgoCD and s
 
 **`charts/<name>/`** — Shared library charts consumed as a Helm dependency by a service's chart (e.g. `charts/sso-guard`, pulled in by `services/products/demo/chart`). Never deployed as their own Application.
 
-### The two `applications/<env>/` file formats
+### The `applications/<env>/` file formats — env-dependent
 
-- **`*.vendor.yaml`** — a complete, hand-written ArgoCD `Application` manifest that loads an external Helm chart (openbao-helm, cert-manager, dex, envoy-gateway...). Free-form: inline values, `ignoreDifferences`, OCI sources, `ServerSideApply`, whatever that tool needs. Discovered by a plain directory-recurse Application (`services-vendor-<env>`, created by `bootstrap/templates/<env>.yaml`) that applies anything matching `services/*/*/applications/<env>/*.vendor.yaml` as-is — no templating.
-- **`*.app.yaml`** — a small params file (`name`, `namespace`, `chartPath`, `valueFile`, `syncWave`) for an Application that loads one of *our own* charts from this repo. Expanded by the `services-app-<env>` ApplicationSet (git `files` generator, `goTemplate: true`, matching `services/*/*/applications/<env>/*.app.yaml`), whose template injects `repoURL`/`targetRevision` — this is what keeps `gitops_revision` propagating without a hand-maintained per-Application file.
+**`local`** still uses the original two-format split:
 
-Adding or removing an Application for an existing service never requires touching `bootstrap/` — just add/remove a file in that service's `applications/<env>/` folder.
+- **`*.vendor.yaml`** — a complete, hand-written ArgoCD `Application` manifest that loads an external Helm chart. Free-form: inline values, `ignoreDifferences`, OCI sources, `ServerSideApply`, whatever that tool needs. Discovered by a plain directory-recurse Application (`services-vendor-local`, created by `bootstrap/templates/local.yaml`) that applies anything matching `services/*/*/applications/local/*.vendor.yaml` as-is — no templating.
+- **`*.app.yaml`** — a small params file (`name`, `namespace`, `chartPath`, `valueFile`, `syncWave`) for an Application that loads one of *our own* charts from this repo. Expanded by the `services-app-local` ApplicationSet (git `files` generator, `goTemplate: true`, matching `services/*/*/applications/local/*.app.yaml`), whose template injects `repoURL`/`targetRevision`.
+
+**`scaleway` was migrated off that split (2026-08-11)** — the two-format system meant sync-wave ordering could never work across a vendor chart and the in-house `-init`/`-config` app that fed it OpenBao-sourced Secrets (confirmed live: Velero/external-dns/Dex/Grafana/the DNS01 ACME webhook all raced OpenBao's restore on a fresh cluster boot). A first fix tried an ApplicationSet + Progressive Syncs (`RollingSync`) — also confirmed live NOT to work: RollingSync doesn't gate the *first-ever* creation/sync of Applications an ApplicationSet generates, only staged updates to an already-existing fleet, so every "stage" synced simultaneously on a fresh cluster anyway.
+
+What actually works, and what `scaleway` uses now: **no `*.app.yaml`/`*.vendor.yaml` files and no ApplicationSet at all** for this env. Every scaleway Application (22 of them, both external-Helm-chart and in-house) is defined as one entry in `bootstrap/values.yaml`'s `scalewayApps` list (`name`, `namespace`, `chartPath`, `valueFile`, `wave`) and rendered directly by `bootstrap/templates/scaleway.yaml` as a literal `Application` manifest — a managed resource of the `bootstrap` Application's own sync, where plain `argocd.argoproj.io/sync-wave` genuinely is honored (proven mechanism, not Beta, works for both first sync and later updates). External charts still get a thin local wrapper chart instead of a hand-written manifest: a `Chart.yaml` declaring the real chart as a Helm `dependencies:` entry (see any `services/platform/*/chart/Chart.yaml` on scaleway, e.g. `services/platform/openbao/chart`), with its values re-nested one level under the dependency's name in `values-<env>.yaml`. Ordering is six coarse waves (`0`=openbao, `1`=eso, `2`=velero, `3`=platform, `4`=monitoring, `5`=products) — see the comment atop `bootstrap/values.yaml`'s `scalewayApps` list.
+
+Adding or removing a scaleway Application means editing `bootstrap/values.yaml`'s `scalewayApps` list directly (not dropping a file in that service's `applications/scaleway/` folder — that pattern is `local`-only now).
 
 ### Revision propagation (feature-branch testing)
 
-`revision` threads from the provisioner through the whole tree: the infra repo's `gitops_revision` → the `bootstrap` Application's `revision` Helm param → the `services-vendor-<env>` Application's `source.targetRevision` (so it reads `*.vendor.yaml` files from that branch) → the `services-app-<env>` ApplicationSet's generator/template (so both the git generator and every generated Application's `targetRevision` point at that branch). It defaults to `main` everywhere, so setting `gitops_revision` to a feature branch makes the *entire* tree deploy from that branch — letting you validate changes on the local cluster before merging to `main`.
+`revision` threads from the provisioner through the whole tree: the infra repo's `gitops_revision` → the `bootstrap` Application's `revision` Helm param → each env's generator/template. For `local`: `services-vendor-local`'s `source.targetRevision` plus `services-app-local`'s ApplicationSet generator/template. For `scaleway`: every `scalewayApps` entry's rendered `Application.spec.source.targetRevision` in `bootstrap/templates/scaleway.yaml`, templated straight from `.Values.revision`. It defaults to `main` everywhere, so setting `gitops_revision` to a feature branch makes the *entire* tree deploy from that branch — letting you validate changes before merging to `main`.
 
 ## Doctor commands
 
@@ -72,7 +78,9 @@ kubectl --context minikube apply --dry-run=client -f services/platform/openbao/a
 1. Create the Helm chart under `services/<bucket>/<name>/<subfolder>/` (`<bucket>` = `platform` or `products`)
 2. Add `services/<bucket>/<name>/applications/<env>/<subfolder>.app.yaml` with `name`, `namespace`, `chartPath`, `valueFile`, `syncWave` — the `services-app-<env>` ApplicationSet picks it up automatically, no other file to touch. See `services/platform/openbao/applications/local/init.app.yaml` as a template.
 
-**New vendor tool (e.g. cert-manager on a new env):** Add `services/<bucket>/<name>/applications/<env>/chart.vendor.yaml`, a complete Application manifest pointing at the external chart — see `services/platform/openbao/applications/scaleway/chart.vendor.yaml` as a template. The `services-vendor-<env>` Application picks it up automatically.
+**New vendor tool on `local`:** Add `services/<bucket>/<name>/applications/local/chart.vendor.yaml`, a complete Application manifest pointing at the external chart — see `services/platform/openbao/applications/local/chart.vendor.yaml` as a template. The `services-vendor-local` Application picks it up automatically.
+
+**New vendor tool on `scaleway`:** no `*.vendor.yaml` here (see above) — create a thin wrapper chart instead (`services/<bucket>/<name>/chart/Chart.yaml` with a `dependencies:` entry for the real chart, `values-scaleway.yaml` with its values nested under the dependency's name) plus `services/<bucket>/<name>/applications/scaleway/chart.app.yaml` (`name`, `namespace`, `chartPath`, `valueFile`, `stage`) — see `services/platform/openbao/chart` + `bootstrap/values.yaml` as a template. The `services-scaleway` ApplicationSet picks it up automatically; pick whichever of the six stages matches what it depends on (or add its own stage + `RollingSync` step in `bootstrap/templates/scaleway.yaml` if it doesn't fit any of them).
 
 **New cluster environment (e.g. staging):**
 

README.md

@@ -61,7 +61,9 @@ gitops/
 ├── services/
 │   ├── platform/                # infra: openbao, cert-manager, gateway, dex, monitoring, ...
 │   │   └── openbao/
-│   │       ├── init/            # in-house chart (restore/unseal Job, ClusterSecretStore)
+│   │       ├── init/            # in-house chart, restore-at-boot Job only — no ESO
+│   │       │                    #   coupling; the ClusterSecretStore it feeds lives in
+│   │       │                    #   services/platform/secrets-sync/config instead
 │   │       │   ├── Chart.yaml
 │   │       │   ├── values.yaml
 │   │       │   ├── values-local.yaml

bootstrap/templates/scaleway.yaml

@@ -1,77 +1,67 @@
 {{- if eq .Values.env "scaleway" }}
-# Vendor charts: hand-written Application manifests under
-# services/**/applications/scaleway/*.vendor.yaml, applied as-is (no
-# templating, no shared schema) — one recurse Application per env, re-synced
-# fresh from `revision` on every ArgoCD sync (same trick the old `platform`
-# app-of-apps relied on for revision propagation on inline values).
+# Every scaleway Application, rendered directly here as a managed resource
+# of the `bootstrap` Application's own sync — see the long comment atop
+# bootstrap/values.yaml's `scalewayApps` list (the actual data source) for
+# why this replaced both the old services-vendor-scaleway/services-app-scaleway
+# split AND a later ApplicationSet+RollingSync attempt. sync-wave on
+# resources of one parent Application's own sync is the mechanism that
+# actually works for both first-time creation and later updates.
+{{- range .Values.scalewayApps }}
+---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
-  name: services-vendor-scaleway
+  name: {{ .name }}
   namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: {{ .wave | quote }}
 spec:
   project: default
 
   source:
-    repoURL: {{ .Values.repoURL }}
-    targetRevision: {{ .Values.revision }}
-    path: services
-    directory:
-      recurse: true
-      include: "*/*/applications/scaleway/*.vendor.yaml"
+    repoURL: {{ $.Values.repoURL }}
+    targetRevision: {{ $.Values.revision }}
+    path: {{ .chartPath }}
+    helm:
+      valueFiles:
+        - {{ .valueFile }}
+
+  # Any MutatingWebhookConfiguration's caBundle gets injected at runtime by
+  # its owning chart's own cert-generation, which never matches what's in
+  # git — standard ArgoCD pattern for any self-signed webhook cert (only
+  # OpenBao's agent-injector has one today). Applied unconditionally to
+  # every app here: harmless no-op for the rest. See "the case of injected
+  # CA bundles": https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/
+  #
+  # Deliberately NOT using ignoreDifferences for the OpenAPI-schema-default
+  # drift ExternalSecret/HTTPRoute/SecurityPolicy showed live 2026-08-11
+  # (the API server fills in fields like a BackendRef's group/kind/weight
+  # that our manifests omitted) — tried that first, but ignoreDifferences
+  # is for fields genuinely outside GitOps control (like this caBundle),
+  # not for fields we simply left unset. Blanket-ignoring them would also
+  # hide real future drift on those same fields (e.g. someone bumping a
+  # BackendRef's weight for canary routing). Fixed at the source instead:
+  # every affected template now writes those defaults explicitly, so
+  # desired matches live exactly and ArgoCD's diff stays meaningful.
+  ignoreDifferences:
+    - group: admissionregistration.k8s.io
+      kind: MutatingWebhookConfiguration
+      jqPathExpressions:
+        - '.webhooks[]?.clientConfig.caBundle'
 
   destination:
     server: https://kubernetes.default.svc
-    namespace: argocd
+    namespace: {{ .namespace }}
 
   syncPolicy:
     automated:
       prune: true
       selfHeal: true
     syncOptions:
       - CreateNamespace=true
----
-# In-house charts: small param files under
-# services/**/applications/scaleway/*.app.yaml, expanded through this shared
-# template. Needed (instead of the plain recurse above) because these
-# Applications load a chart from THIS repo, so revision/repoURL must be
-# injected — the ApplicationSet template bakes in `revision`/`repoURL` from
-# bootstrap's own Helm values, exactly like the old clusters/scaleway/ chart did.
-apiVersion: argoproj.io/v1alpha1
-kind: ApplicationSet
-metadata:
-  name: services-app-scaleway
-  namespace: argocd
-spec:
-  goTemplate: true
-  generators:
-    - git:
-        repoURL: {{ .Values.repoURL }}
-        revision: {{ .Values.revision }}
-        files:
-          - path: "services/*/*/applications/scaleway/*.app.yaml"
-  template:
-    metadata:
-      name: '{{ `{{.name}}` }}'
-      namespace: argocd
-      annotations:
-        argocd.argoproj.io/sync-wave: '{{ `{{.syncWave}}` }}'
-    spec:
-      project: default
-      source:
-        repoURL: {{ .Values.repoURL }}
-        targetRevision: {{ .Values.revision }}
-        path: '{{ `{{.chartPath}}` }}'
-        helm:
-          valueFiles:
-            - '{{ `{{.valueFile}}` }}'
-      destination:
-        server: https://kubernetes.default.svc
-        namespace: '{{ `{{.namespace}}` }}'
-      syncPolicy:
-        automated:
-          prune: true
-          selfHeal: true
-        syncOptions:
-          - CreateNamespace=true
+      # Unconditional, not opt-in per app: same reasoning as
+      # ignoreDifferences above, and ArgoCD's own recommended default for
+      # reducing client-side 3-way-merge diff issues.
+      - ServerSideApply=true
+{{- end }}
 {{- end }}