fix: give every requestless platform chart real resource requests/limits - #23
Merged
Conversation
…mits
Confirmed live 2026-08-11: on a fresh scaleway-homelab boot, one node
carried 31 pods to the other's 12 and hit MemoryPressure, evicting
external-dns, a grafana replica, a wireguard replica, and a velero
replica, plus crashlooping cert-manager and wireguard. Root cause:
every platform chart here except demo and monitoring shipped
resources: {} (their own chart defaults) — with nothing to reserve,
kube-scheduler has no real signal to balance the pool's 2 nodes on,
and Cluster Autoscaler never sees a Pending pod to justify reaching
for the 3rd node, so ArgoCD's burst of ~20 Applications syncing at
once lands wherever the scheduler ties, not where capacity actually
is.
Adds requests/limits to cert-manager (+ its cainjector/webhook/
startupapicheck and the separate scaleway DNS01 webhook chart),
external-dns, external-secrets (+ webhook/certController), openbao
(+ injector and the hourly snapshotAgent CronJob), velero, and the
wireguard chart's own containers (no upstream chart to inherit from,
so these were entirely unset). Sized against each pinned chart
version's actual schema (verified via `helm template`) and kept
modest relative to the pool's DEV1-M nodes (~2.3Gi allocatable each);
components with bursty/startup-heavy memory profiles — cert-manager's
cainjector, openbao's server and snapshotAgent, velero — get a limit
well above their request, mirroring the reasoning already applied to
monitoring's grafana/prometheus after an earlier incident. Companion
fix in the infrastructure repo's 10-cluster/scaleway/argocd.tf for
ArgoCD's own components.
envoy-gateway was checked and left untouched: its controller and the
Envoy proxy data-plane pods it provisions both already carry real
default resource requests from the chart/controller itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mqgrku7T7sfPYHvg9fmBMB
3 tasks
kubectl top pods (2026-08-11, two samples ~15min apart) showed the first pass was anchored on chart-doc suggestions and generic homelab-scale guesses rather than this cluster's reality: several requests sat well below observed usage (cert-manager-webhook-scaleway requested 16Mi against an observed ~45Mi; cert-manager's controller, cainjector, and external-secrets' controller were all undersized too), and CPU requests were oversized fleet-wide relative to the 1-6m idle usage actually observed. Re-derived every value in this diff from the live numbers, with limits kept comfortably above observed usage rather than just barely covering it. Untouched here (already comfortably above observed usage): external-dns, external-secrets' webhook (close but had margin), openbao's injector, wireguard. Companion fix in the infrastructure repo's 10-cluster/scaleway/argocd.tf for ArgoCD's own components — its controller was the most severely undersized of all of these, limit barely above observed usage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mqgrku7T7sfPYHvg9fmBMB
Caught mid-incident on a live relaunch (2026-08-11): dns hit its 32Mi limit and was crashlooping (OOMKilled x3 in under 2 minutes), and proxy-gateway was observed at 31Mi against the same 32Mi limit, one step from the same fate. Root cause: every container here runs `apk add --no-cache <pkg>` fresh at startup (nothing baked into the base alpine image), and that package-fetch/install step spikes well above what the actual running process needs afterwards — the two smallest limits in the original pass didn't leave room for it. proxy-gateway also gets split into its own resources.proxyGateway key instead of sharing resources.proxy: it runs on alpine/kubectl (bundles the kubectl binary) rather than the plain alpine the generic per-target proxy sidecars use, so it has a meaningfully heavier baseline. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mqgrku7T7sfPYHvg9fmBMB
Confirmed live 2026-08-11: node-exporter's DaemonSet pod sat Pending for minutes on a node at ~99% requested memory. DaemonSet pods are pinned to one specific node each via nodeAffinity — unlike a regular pod they can't just land on the other node if this one is full, and Cluster Autoscaler adding a 3rd node doesn't rescue them either, since the pod that needs scheduling is bound to a node that already exists. At default priority (0) it had no way to preempt anything to make room for itself, while every other DaemonSet on this cluster (cilium, csi-node, konnectivity-agent, kube-proxy — all Scaleway-managed) already runs system-node-critical/system-cluster-critical and would preempt a regular pod without hesitation. Adds a new mid-tier `node-daemonset` PriorityClass (value 1000000, not global-default) via kube-prometheus-stack's extraManifests, and sets it on prometheus-node-exporter specifically — it isn't actually node-critical (the node functions fine without metrics, it just goes dark in Grafana), so it doesn't borrow the system-reserved tiers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mqgrku7T7sfPYHvg9fmBMB
Simpler than the previous commit: instead of inventing a dedicated node-daemonset PriorityClass via extraManifests, just point node-exporter at the same system-node-critical tier cilium/csi-node/ kube-proxy already use on this cluster. Zero new cluster-scoped resources, one field. Same fix for the actual symptom (node-exporter couldn't preempt anything to get scheduled) — the eviction-order purity argument for a separate tier is real in principle but not worth the extra machinery at this cluster's scale. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mqgrku7T7sfPYHvg9fmBMB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scaleway-homelabboot: one node carried 31 pods to the other's 12 and hitMemoryPressure, evictingexternal-dns, agrafanareplica, awireguardreplica, and aveleroreplica, plus crashloopingcert-managerandwireguard.demoandmonitoringshippedresources: {}(each chart's own default). With nothing to reserve, kube-scheduler has no real signal to balance the pool's 2 nodes on, and Cluster Autoscaler never sees a Pending pod to justify reaching for the 3rd node — ArgoCD's burst of ~20 Applications syncing at once lands wherever the scheduler ties, not where capacity actually is.cert-manager(+ cainjector/webhook/startupapicheck + the separate scaleway DNS01 webhook chart),external-dns,external-secrets(+ webhook/certController),openbao(+ injector + the hourlysnapshotAgentCronJob),velero, and thewireguardchart's own containers (no upstream chart to inherit from there — added aresourcesmap tovalues.yamland wired it throughtemplates/deployment.yaml).cert-manager's cainjector (CA-bundle injection scales with total CRD count),openbao's server (raft compaction/unseal) andsnapshotAgent(hourly s3cmd export),velero(backup runs) — get a limit well above their request, mirroring the reasoningmonitoring's grafana/prometheus resources block already documents from an earlier incident.envoy-gatewaywas checked and left untouched: its controller and the Envoy proxy data-plane pods it provisions both already carry real default resource requests (verified againstDefaultResourceRequirements()in the controller source and the chart's owndeployment.envoyGateway.resourcesdefault).infrastructure: fix: give ArgoCD real resource requests/limits infrastructure#54 — same fix for ArgoCD's own components (controller/repoServer/server/redis/applicationSet/notifications), which this repo doesn't manage.Test plan
helm.valuesblock's inner YAML, parsed standalone without errorhelm templateagainst each chart at its exact pinned version (cert-manager v1.21.0, scaleway-certmanager-webhook 0.4.2, external-dns 1.21.1, external-secrets 2.6.0, openbao 0.28.6, velero 12.1.0) — all render cleanly, and the resources landed on the expected Deployment/StatefulSet/CronJob in each rendered manifesthelm templateon thewireguardchart itself confirms all 4 containers (initrender-wg-conf,wireguard,proxy-gateway,dns) got requests+limitsscaleway-upand confirm pod distribution across nodes evens out, no more node-pressure evictions during boot🤖 Generated with Claude Code
https://claude.ai/code/session_01Mqgrku7T7sfPYHvg9fmBMB