Argo Workflows CronWorkflow automation + SSO UI: tracking

[nbrieussel]

Context

What started as "let Argo Workflows run terraform apply for OpenBao/Grafana from inside the cluster" (OpenBao isn't reachable from outside for a while after boot) grew over one long session into: a working CronWorkflow automation, a full Argo Server UI with Dex SSO, and a real ArgoCD OOM incident + HA tuning pass. Tracking it all here since it ended up spanning way more than the original scope.

Two PRs, one per repo, both already live-tested against a real full cluster destroy+rebuild (not just planned):

• infra: feat: argo-workflows SSO client + ArgoCD OOM fix + HA tuning #71
• gitops: feat: Argo Workflows SSO UI + terraform-apply wave fix + OOM alerting gitops#37

Related follow-up already filed separately: #72 (image-pull-caching investigation).

Done (confirmed live, not just written)

Core feature — terraform-apply CronWorkflows

• WorkflowTemplate + one CronWorkflow per root (05-secrets/openbao/managed, 06-monitoring/grafana/{bootstrap,managed})
• Dedicated argo-workflows-state Scaleway identity, project-scoped IAM (not a bucket policy — see the incident note below)
• openbao-managed CronWorkflow confirmed succeeding for real on its own hourly schedule
• terraform-apply moved wave 2 → wave 6: it was firing before grafana (wave 5) existed, failing every run until a scheduled trigger happened to land later by luck

Argo Server UI + Dex SSO

• Native OIDC against the shared Dex, same pattern as ArgoCD/Grafana
• Exposed at argo-workflows.scalepack.fr, RBAC mapped to the GitHub org admin group via SSO rbac-rule
• Dex moved wave 6 → wave 1 (only its own secret is a real dependency; only dex-gateway's HTTPRoute needs the Gateway) — was blocking the entire sync at wave 1 once argo-server's eager OIDC discovery needed it to exist
• server.sso.issuer set to Dex's in-cluster Service (not the public URL) + issuerAlias for the public issuer string, so discovery/token-exchange don't depend on the cluster's own hairpin-NAT-affected public route
• Confirmed live end-to-end: GET / → 200, SSO redirect → Dex → real login flow, correct client_id/redirect_uri

ArgoCD OOM incident + fix

• Root cause: argocd-application-controller OOMKilled repeatedly during a full ~40-Application tree resync, silently killing every sync attempt — this is what actually blocked the sync for the better part of an hour, not an application-level error
• GOMEMLIMIT (proactive GC before the cgroup hard limit) + memory 1000Mi → 1500Mi + priorityClassName: system-cluster-critical (protects it from the kubelet's own node-memory-pressure eviction specifically, a different mechanism than the cgroup OOM)
• Node pool DEV1-M → DEV1-L (more DaemonSets land as ArgoCD advances through waves, compounding pressure on small nodes)
• resource.exclusions (EndpointSlice/Endpoints/cilium.io) to cut cluster-cache memory
• Confirmed live: controller stayed 447–481Mi (well under the 1500Mi limit) through an entire fresh full-tree sync, zero crashes
• Sharding deliberately not used, confirmed via research it's a no-op for a single-cluster ArgoCD instance (splits load by registered cluster, and this instance only ever registers itself)

Observability

• New PrometheusRule: ContainerOOMKilled (cluster-wide) + ArgoCDApplicationControllerMemoryHigh (85% sustained 5m early warning) — confirmed live, both load with health ok, ContainerOOMKilled correctly fired on real data from the incident above
• JSON log format + Prometheus metrics/ServiceMonitor enabled on all 4 ArgoCD components (was scraping nothing before)
• Alloy already ships every pod's logs to Loki from wave 4 — confirmed this predates Grafana's own UI being reachable, so ArgoCD's state is capturable from early boot even before anyone can look at it in Grafana

Incidental fix, unrelated to the above but hit along the way

• Gateway's shared wildcard TLS cert flipped back letsencrypt-staging → letsencrypt-prod — was flipped to staging 2026-07-25 after a rate-limit incident, the retry-after window closed weeks ago (2026-07-26) and it just never got flipped back. Was silently breaking browser trust + every native-OIDC app's server-side token exchange this whole time.

Remaining / follow-up (not blocking merge)

• Strip the two TODO(before merge) lines in the gitops branch (bootstrap/values.yaml revision, terraform-apply's gitRef) back to main before merging — isolated in their own commit already
• Merge order: infra PR feat: argo-workflows SSO client + ArgoCD OOM fix + HA tuning #71 first (companion secret data), then gitops PR feat: Argo Workflows SSO UI + terraform-apply wave fix + OOM alerting gitops#37
• terraform-apply's wave-6 placement is a blunt fix — openbao-managed doesn't actually need to wait that long (only depends on OpenBao itself, wave 0); proper per-root dependency/health gating is real follow-up work, not done here
• vault_kv_secret_v2's data_json_wo/data_json_wo_version write-only pattern: flagged live tonight as a recurring footgun (state can silently diverge from live OpenBao if a version bump is forgotten — this exact thing happened after a backup restore during tonight's testing). Owner wants this pattern removed from 05-secrets/openbao/managed in a dedicated follow-up session, not bundled here.
• Investigate: full cluster rebuild boot time, image pulls suspected major contributor #72 — image-pull caching investigation (separate, already filed)