[tushagoy]

Summary

This change adds support for draining terminating Pod backends in VCN-native clusters.

During backend reconciliation, when a published Kubernetes Endpoint references the exact Pod whose DeletionTimestamp is set, the controller keeps the existing OCI Load Balancer backend and sets drain=true. Pod termination and Endpoint changes enqueue reconciliation immediately; the existing 10-second periodic reconciliation remains as a fallback.

OCI drain mode stops new TCP connections and new non-sticky HTTP requests from being routed to that backend. Existing connections and persisted sessions can continue while the application performs graceful shutdown, subject to its shutdown behavior and Kubernetes termination grace period.

Changes

• Derive backend drain state from the referenced Pod's DeletionTimestamp.
• Reconcile from Pod termination and Endpoint update/delete events, with periodic fallback.
• Validate the Endpoint TargetRef namespace, kind, and UID before using a cached Pod.
• Fail closed during Endpoint/Pod informer skew by retaining unresolved addresses with drain=true while continuing to reconcile other addresses.
• Apply draining to Ingress path backends and default backends.
• Use targeted OCI UpdateBackend calls when only drain state changes; retain full backend-set updates for membership changes.
• Propagate default-backend resolution and OCI update errors through the work queue.
• Keep Flannel/node backends explicitly undrained because they represent nodes rather than individual Pods.
• Document Service isolation, HTTP readiness health checks, lifecycle timing, limitations, and operational verification.
• Add controller lifecycle, informer-skew, identity, drain-transition, error-propagation, and node-backend regression tests.

How it works

1. Kubernetes assigns a DeletionTimestamp when Pod termination starts.
2. A dedicated Ingress-facing Service keeps the terminating Pod Endpoint published.
3. The Pod update event enqueues the affected Ingress classes. Endpoint update and delete events also trigger reconciliation. The 10-second periodic loop remains a fallback.
4. The controller validates that the Endpoint references the exact Pod identity and updates the OCI backend with drain=true.
5. Kubernetes keeps the Pod alive for its configured termination grace period while the application completes or cancels active work.
6. After Kubernetes removes the Endpoint, a subsequent reconciliation removes the OCI backend.

Backend removal may be delayed while an Endpoint temporarily references a Pod missing from the informer cache. During this skew, the controller retains that address as drained and continues reconciling the remaining addresses.

Configuration requirements

The terminating Endpoint must remain published long enough for the controller to observe and drain it:

spec:
  publishNotReadyAddresses: true

Because this setting also publishes startup-failing and otherwise unready endpoints, use a dedicated Ingress-facing Service and do not reuse it for non-Ingress consumers.

Configure the OCI Load Balancer with an HTTP/HTTPS health check against the application's readiness endpoint instead of relying on the default TCP health check.

Size the Pod lifecycle budget using:

terminationGracePeriodSeconds >= preStop drain wait + application shutdown timeout + safety margin

The drain wait should account for:

1. Controller event processing and the 10-second reconciliation fallback.
2. Measured P99 OCI backend-update duration.
3. Sequential updates to all affected backends and backend sets.
4. A safety margin for transient controller or OCI delays.

The application must handle SIGTERM, stop accepting new work, and finish or bound in-flight, WebSocket, streaming, and other long-lived connections within the remaining grace period.

This change does not delay Pod termination or provide Pod-level acknowledgement that OCI has applied drain mode. A fixed preStop wait is therefore best-effort.

See GettingStarted.md for the complete Service, Ingress, Deployment, timing, and verification guidance.

Validation

• go test -mod=vendor ./...
• go test -race -mod=vendor -count=1 ./pkg/controllers/backend ./pkg/controllers/nodeBackend ./pkg/loadbalancer ./pkg/oci/client ./pkg/server ./pkg/util
• go vet -mod=vendor ./pkg/controllers/backend ./pkg/controllers/nodeBackend ./pkg/loadbalancer ./pkg/oci/client ./pkg/server ./pkg/util

[oleksandr-kostin]

PR #160 addresses the two principal controller defects:

• It checks Pod.DeletionTimestamp and renders the backend with Drain=true.
• It keeps normal backends at Drain=false.
• It applies this to both Ingress path and default backends.
• Backend comparison now includes drain state, so changing only Drain triggers an OCI update.
• It leaves Flannel node backends unchanged because those represent nodes, not individual pods.
• Documentation correctly requires the terminating endpoint to remain published.
  This matches Kubernetes termination semantics: an EndpointSlice exposes terminating=true when deletion begins, while publishNotReadyAddresses keeps endpoints available for consumers that need to manage draining. Kubernetes EndpointSlice documentation.
  OCI Drain=true is the right backend state: it stops new TCP connections and new non-sticky HTTP requests while preserving established/persisted traffic. OCI backend drain documentation.

[AkarshES]

Since this can be considered a breaking change I would suggest you change the design a bit. It's better to make this configurable at Service level. Use a Service annotation such as oci-native-ingress.oraclecloud.com/backend-draining: "true" and only set drain=true when:

• that Service opted in;
• the Endpoint targets the exact terminating Pod; and
• the Endpoint remains published.

All other Ingress backends should preserve today’s behavior.

[tushagoy]

Since this can be considered a breaking change I would suggest you change the design a bit. It's better to make this configurable at Service level. Use a Service annotation such as oci-native-ingress.oraclecloud.com/backend-draining: "true" and only set drain=true when:

• that Service opted in;
• the Endpoint targets the exact terminating Pod; and
• the Endpoint remains published.

All other Ingress backends should preserve today’s behavior.

Updated in f03446f to make draining opt-in using oci-native-ingress.oraclecloud.com/backend-draining: "true". Without it, backends remain drain=false. With it, draining occurs only for a published Endpoint referencing the exact UID-matched terminating Pod. Tests and documentation were updated.

[oleksandr-kostin]

[P1] Complete the Deployment example so it can be applied

The example under Graceful Backend Draining omits the required spec.selector.matchLabels and matching spec.template.metadata.labels. As written, the apps/v1 Deployment is rejected, and the Service selector cannot select its Pods. Please restore the selector and template labels from the earlier suggested example. Since this is intended as a complete operational example, also consider retaining the startup/liveness probes, explicit 60-second application-shutdown configuration, and resource requests/limits.

[oleksandr-kostin]

[P2] Protect drain-only backend updates with the load balancer ETag

UpdateBackends obtains the current ETag, but the targeted UpdateBackendRequest omits IfMatch while resubmitting Weight, Backup, Offline, and MaxConnections from the cached backend. If that state is stale or another reconciler/operator changes the backend between the GET and PUT, the drain update can silently overwrite those fields. Please set IfMatch; for multiple sequential updates, refresh the ETag after each completed work request. Add coverage for a 412 response/retry and for preserving the non-drain fields.

[tushagoy]

[P1] Complete the Deployment example so it can be applied

The example under Graceful Backend Draining omits the required spec.selector.matchLabels and matching spec.template.metadata.labels. As written, the apps/v1 Deployment is rejected, and the Service selector cannot select its Pods. Please restore the selector and template labels from the earlier suggested example. Since this is intended as a complete operational example, also consider retaining the startup/liveness probes, explicit 60-second application-shutdown configuration, and resource requests/limits.

Fixed. The example now includes matching selector/template labels, probes, 60-second shutdown configuration, and resource requests/limits.

[tushagoy]

[P2] Protect drain-only backend updates with the load balancer ETag

UpdateBackends obtains the current ETag, but the targeted UpdateBackendRequest omits IfMatch while resubmitting Weight, Backup, Offline, and MaxConnections from the cached backend. If that state is stale or another reconciler/operator changes the backend between the GET and PUT, the drain update can silently overwrite those fields. Please set IfMatch; for multiple sequential updates, refresh the ETag after each completed work request. Add coverage for a 412 response/retry and for preserving the non-drain fields.

Fixed. Drain updates now use IfMatch, refresh the backend state and ETag between updates, and treat 412 responses as transient for retry. Tests also verify ETag refresh and preservation of non-drain fields.

[AkarshES]

One thing to think through before merging: a backend marked as “draining” is still counted by the Load Balancer as a backend.

OCI Load Balancers allow up to:

• 512 backends in one backend set
• 512 backends total across the whole Load Balancer

So a rollout can get stuck when the LB is already at either limit:

• An old Pod starts shutting down, and we keep it in the LB to drain connections.
• A replacement Pod appears.
• The LB temporarily needs space for both Pods.
• If that makes the backend count exceed 512, OCI rejects the update.
• With zero-downtime rollout settings, Kubernetes may wait for the replacement to become ready before removing the old Pod—while the replacement cannot become ready because it was never added to the LB.

It would be good to decide the intended behavior at capacity:

• Reserve some backend slots for rolling updates.
• Remove a draining backend early when needed to make room for a replacement.
• Detect the condition and surface a clear error so users can adjust replica counts or rollout settings.

My preference is to keep some slots for rolling updates, but it has to be managed across backendsets.