CAST AI PDB Controller

A custom Kubernetes controller that automatically manages PodDisruptionBudgets (PDBs) for Deployments and StatefulSets using flexible, annotation-driven configuration.
This controller enables safe, automated disruption management with per-workload overrides and cluster-wide defaults.

---

Features

• Automatic PDB Creation/Update:
  Ensures every eligible Deployment/StatefulSet has a PDB, using defaults or per-workload overrides.

• Annotation-Based Customization:
  Users can set minAvailable, maxUnavailable, or opt out of PDB management with annotations.

• Dynamic Default Configuration:
  Cluster-wide default PDB values are set via a ConfigMap and can be changed at runtime without redeploying the controller.

• Automatic Detection and Handling of Poor PDB Configurations:
  The controller detects overly restrictive or ineffective PodDisruptionBudgets (such as minAvailable equal to replicas, minAvailable: 100%, or maxUnavailable: 0/0%) that could block voluntary disruptions or cause operational issues.

  Through the FixPoorPDBs option in the castai-pdb-controller-config ConfigMap, you can choose whether the controller should only warn about these poor configurations (default) or automatically delete and recreate them with safe defaults. This ensures cluster upgrades, node drains, and scaling operations are not blocked by problematic PDBs.

• Live Reconciliation:
  If annotations or ConfigMap values change, existing PDBs are updated automatically to reflect new requirements.

• Bypass Support:
  Workloads can opt out of automatic PDB management at any time by adding a bypass annotation.

• Exclusion Rules:
  Configure regex-based exclusion rules to automatically skip PDB creation for specific workloads based on namespace, name, and label patterns. Useful for system workloads, temporary deployments, or critical services.

• Garbage Collection:
  Orphaned PDBs are cleaned up when workloads are deleted. PDBs are also cleaned up when their workload still exists but has scaled down to fewer than 2 replicas — this runs periodically as a safety net in case a live scale-down event was ever missed (e.g. due to a controller restart or leader-election gap), preventing a stale PDB from blocking node drains/rotations indefinitely.

• Leader Election:
  Supports safe, highly available operation in multi-replica controller deployments.

• Additional PDB Selector Labels:
  Optionally extend PDB matchLabels with extra labels drawn from each workload's pod template. Configure a list of label keys in the ConfigMap and the controller will add them to the PDB selector when they are present on the pod template — silently skipping any keys that are absent.

• Coverage-Based Existing-PDB Detection:
  The controller recognizes a pre-existing PDB as already covering a workload whenever that PDB's selector matches the workload's pod template labels, even if the PDB's selector isn't written identically to the one the controller would generate. This avoids duplicate PDBs for workloads whose Helm chart (or other owner) manages a PDB with a differently-shaped but still-matching selector. When a match is found via a non-identical selector, the controller logs a warn-level message so operators can spot selectors that may need a disambiguating label.

• Controller-Owned PDB Naming Convention:
  The controller only treats a matching PDB as its own (safe to update to match its own config) when the PDB's name both starts with castai- and ends with -pdb — the exact pattern it uses when generating PDBs (castai-<workload>-pdb). Any other matching PDB, including ones from Helm charts whose release name happens to start with castai- (e.g. castai-agent, castai-cluster-controller, castai-pod-mutator), is treated as externally managed: the controller leaves it untouched and skips creating its own.

• CAST Component Leftover Cleanup:
  On reconcile and during the periodic multi-PDB scan, if a CAST Helm-style PDB (castai-* without the -pdb suffix) already covers a workload and a leftover controller-owned castai-*-pdb also covers it, the controller deletes only the leftover controller PDB. Customer castai-*-pdb objects covered solely by unrelated Helm PDBs are left alone so controller-managed customer workloads stay intact.

• New-PDB readiness gate:
  After the multi-replica check (>= 2), the controller only creates a new castai PDB when the workload is fully ready and available (ReadyReplicas and AvailableReplicas both >= desired; StatefulSets use ready only). If pods are not ready yet (cold start, crash-loop, mid-rollout), creation is skipped and a warning is logged. Existing controller PDBs are not deleted for unreadiness.

• Disruption-block warnings:
  The periodic scan warns (rate-limited) when a controller-owned PDB has disruptionsAllowed=0 with populated status (currentHealthy / desiredHealthy / expectedPods and how long it has been stuck). This surfaces drain-blocking budgets without removing the PDB. Operators who want NotReady pods to remain evictable can still set the optional defaultUnhealthyPodEvictionPolicy / annotation overrides (AlwaysAllow); those are not enabled by default.

• Configurable log levels:
  Set logLevel in the castai-pdb-controller-config ConfigMap to debug, info, warn, or error (default info) to control how much the controller writes to stderr.

---

Usage

1. Default Behavior

If no annotation is set, the controller uses the values from the castai-pdb-controller-config ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: castai-pdb-controller-config
  namespace: castai-agent
data:
  minAvailable: "1"
  logLevel: "info"  # debug | info | warn | error — default info suppresses DEBUG trace lines
  # maxUnavailable: "50%"  # Optional, use one or the other
  FixPoorPDBs: "true"  # Set to "true" to auto-fix poor PDBs, "false" to only warn
  exclusions: |
    - namespaceRegex: "^kube-system$"
      nameRegex: ""
      labels: {}
    - namespaceRegex: ""
      nameRegex: ".*-temp$"
      labels:
        app: "^db-.*"
        env: "staging|dev"

---

2. Exclusion Rules

Configure exclusion rules to automatically skip PDB creation for specific workloads:

apiVersion: v1
kind: ConfigMap
metadata:
  name: castai-pdb-controller-config
  namespace: castai-agent
data:
  exclusions: |
    - namespaceRegex: "^kube-system$"  # Exclude all workloads in kube-system
      nameRegex: ""
      labels: {}
    - namespaceRegex: ""               # Exclude workloads with names ending in -temp
      nameRegex: ".*-temp$"
      labels:
        app: "^db-.*"                  # AND app label starts with db-
        env: "staging|dev"             # AND env is staging or dev
    - namespaceRegex: "monitoring"     # Exclude workloads in monitoring namespace
      nameRegex: ""
      labels:
        role: "critical"               # AND role label is critical

Exclusion Rule Logic:

• Each rule is evaluated independently
• If a workload matches ANY rule, no PDB is created
• Within a single rule, all specified criteria must match (AND logic)
• Empty strings for namespaceRegex or nameRegex mean "no filter"
• Empty object {} for labels means "no label filter"
• Regular expressions are supported for flexible matching

---

3. Additional PDB Selector Labels

By default, the controller builds a PDB selector using only the labels from spec.selector.matchLabels on the Deployment or StatefulSet. You can extend this with extra labels from the workload's pod template by listing their keys under additionalSelectorLabels in the ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: castai-pdb-controller-config
  namespace: castai-agent
data:
  minAvailable: "1"
  additionalSelectorLabels: |
    - role
    - component

How it works:

• For each key in additionalSelectorLabels, the controller looks it up in the workload's spec.template.metadata.labels.
• If the label is present, its key/value is added to the PDB's matchLabels.
• If the label is absent, it is silently skipped — the controller never fails due to a missing label.
• Labels already present in the deployment selector are never overwritten.

Example: a Deployment whose pod template carries role: worker and component: api and whose spec.selector.matchLabels is {app: my-app, release: my-app} will produce a PDB with:

selector:
  matchLabels:
    app: my-app
    release: my-app
    role: worker
    component: api

Helm: set config.additionalSelectorLabels in values.yaml:

config:
  additionalSelectorLabels:
    - role
    - component

---

4. Per-Workload Annotations

Add annotations to your Deployment or StatefulSet to override the defaults:

Set minAvailable

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-namespace
  annotations:
    workloads.cast.ai/pdb-minAvailable: "2"
spec:
  replicas: 3
  # ...

Set maxUnavailable

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-db
  namespace: my-namespace
  annotations:
    workloads.cast.ai/pdb-maxUnavailable: "25%"
spec:
  replicas: 4
  # ...

Bypass PDB Creation

apiVersion: apps/v1
kind: Deployment
metadata:
  name: no-pdb-app
  namespace: my-namespace
  annotations:
    workloads.cast.ai/bypass-default-pdb: "true"
spec:
  replicas: 5
  # ...

---

Annotation Reference

Annotation	Description	Example Value
workloads.cast.ai/pdb-minAvailable	Minimum pods that must be available (int or percent, one only)	"2", "50%"
workloads.cast.ai/pdb-maxUnavailable	Maximum pods that can be unavailable (int or percent, one only)	"1", "25%"
workloads.cast.ai/pdb-unhealthyPodEvictionPolicy	Override PDB unhealthyPodEvictionPolicy (Kubernetes 1.26+)	AlwaysAllow, IfHealthyBudget
workloads.cast.ai/bypass-default-pdb	Opt out of automatic PDB management	"true"

---

How It Works

• On workload creation or update:
  The controller checks for annotations and creates/updates a PDB accordingly.
• Detecting an already-covered workload:
  Before creating a PDB, the controller checks whether any existing PDB in the namespace already covers the workload's pods — a PDB "covers" a workload if the PDB's selector matches the workload's pod template labels, not only when the selector is written identically to the one the controller would generate. This means a pre-existing, e.g. Helm-managed, PDB with extra selector labels (such as a shard key added to disambiguate sibling StatefulSets) is correctly recognized instead of triggering a duplicate castai-* PDB.
• On annotation or ConfigMap change:
  The controller reconciles and updates existing PDBs to match new settings.
• On workload deletion or bypass:
  The controller deletes the associated PDB.
• On ConfigMap update:
  All workloads using the default config are updated to the new values.

---

Log levels

Verbosity is controlled by the logLevel key in the castai-pdb-controller-config ConfigMap (no redeploy required). You can also set the CASTAI_PDB_CONTROLLER_LOG_LEVEL environment variable on the controller pod only when logLevel is omitted from the ConfigMap.

Level	Meaning
error	Failures only (API errors, invalid exclusion regexes, failed creates/deletes).
warn	Errors plus warnings (invalid durations in the ConfigMap, invalid selectors, poor-PDB / multi-PDB warnings). Hides routine success lines.
info	Default. Normal operations (PDB create/update/delete, skips, leader messages, config summary). Does not emit DEBUG: trace lines.
debug	Everything including high-volume DEBUG: traces (per-workload exclusion checks, reconciliation steps). Use only while troubleshooting.

Aliases (case-insensitive): d, i, w, e, warning, fatal (same as error). Unknown values fall back to info with a warning.

Helm: set config.logLevel in values.yaml (rendered into the ConfigMap).

---

Requirements

• Kubernetes 1.21+
• Permissions to manage Deployments, StatefulSets, PDBs, and ConfigMaps in your cluster.
• RBAC rules that allow listing namespaces and managing PDBs at the cluster scope.

---

Example: Full Deployment with Controller

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-namespace
  annotations:
    workloads.cast.ai/pdb-minAvailable: "2"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app:latest

---

Troubleshooting

• Duplicate logs:
  Usually caused by log collector configuration, not the controller itself.
• No PDB created:
  Ensure your workload has at least 2 replicas, those replicas are ready/available, and the workload is not opted out with the bypass annotation. Check controller logs for workload not ready skip messages.
• PDB reports disruptionsAllowed=0:
  Look for disruptionsAllowed=0 warnings from the controller. The controller does not delete existing PDBs for this reason; fix the workload health or adjust PDB/unhealthyPodEvictionPolicy overrides intentionally.
• RBAC errors:
  Make sure your controller has permissions to list namespaces and manage PDBs.

---

For advanced usage, deployment via Helm, or troubleshooting, see the controller source code and your cluster’s RBAC configuration.

If you decide to remove the castai-pdb-controller from your cluster, you need to run the following clean-up command if you'd like all custom-created PDBs to also be deleted.

kubectl get poddisruptionbudget --all-namespaces -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name" \
  | awk '$2 ~ /^castai-.*-pdb$/ {print "kubectl delete poddisruptionbudget -n " $1 " " $2}' \
  | sh