Document multi-node networking for SSL validation

[yalegria]

What?

Adds documentation describing a networking consideration for SSL certificate validation in multi-node Kubernetes clusters.

The new section explains that the HAProxy ingress controller and the temporary certbotbot-http pod may be scheduled on different worker nodes. It documents the requirement for cross-node TCP port 80 connectivity, includes AWS EKS security-group guidance, and provides a command for testing connectivity from HAProxy to the Certbot pod.

Why?

Certbot uses an HTTP-01 challenge served by the temporary certbotbot-http pod on TCP port 80.

In a multi-node cluster, HAProxy and the Certbot pod may run on different worker nodes. If the worker-node firewall or security group does not allow cross-node TCP port 80 traffic, HAProxy cannot reach the Certbot pod. This can cause the ACME challenge endpoint to return 503 Service Unavailable and certificate validation to fail.

This behavior may not appear in a single-node cluster or when both pods happen to be scheduled on the same node, making the failure difficult to diagnose.

Examples

Observed before allowing cross-node TCP port 80 traffic:

HAProxy pod -> Certbot pod:80
Connection timed out

Public ACME challenge endpoint:
HTTP/1.1 503 Service Unavailable

Observed after allowing TCP port 80 between worker nodes:

HAProxy pod -> Certbot pod:80
HTTP/1.1 404 Not Found
Server: nginx

The 404 Not Found response is expected when using a test path that does not contain a valid ACME token. It confirms that HAProxy can reach nginx in the Certbot pod.

After cross-node connectivity was enabled, Certbot completed SSL certificate validation successfully.

How to test

1. In a multi-node test cluster, start the Certbot certificate-validation resources.

2. Confirm that the HAProxy and certbotbot-http pods are running on different nodes:

   kubectl get pods \
     --namespace <namespace> \
     --output wide

3. While the Certbot pod is running, test connectivity from HAProxy to the Cerbot pod:

   NAMESPACE="<namespace>"
   
   CERTBOT_IP=$(kubectl get pod certbotbot-http \
     --namespace "${NAMESPACE}" \
     --output jsonpath='{.status.podIP}')
   
   kubectl exec \
     --namespace "${NAMESPACE}" \
     deployment/haproxy \
     -- curl -sv --connect-timeout 5 \
     "http://${CERTBOT_IP}/.well-known/acme-challenge/test"

4. Verify that nginx returns an HTTP response. A 404 Not Found response is expected for the test path.

5. Run the certificate-validation process and verify that Certbot successfully creates the Kubernetes TLS Secret.

Documentation of functionality

The documentation is included in this pull request in community-edition/readme.md.

It is located under Considerations for Production Environment in the new Multi-node cluster networking and SSL validation section.

Limitations

This pull request only adds documentation. It does not modify Kubernetes manifests, AWS security groups, Terraform, or other infrastructure resources.

The AWS EKS guidance may not apply directly to clusters using another cloud provider, CNI implementation, firewall model, or security-groups-for-pods configuration. Those environments must provide equivalent cross-node connectivity using their platform-specific networking controls.

Alternative implementations considered

Allowing all traffic between worker nodes would also resolve this scenario, but it grants broader access than is required for the Certbot HTTP-01 solver.

The documentation instead recommends allowing TCP port 80 between worker nodes when required. It also recommends managing the rule through the infrastructure-as-code configuration used to build the cluster rather than relying on a manual change.

Open questions

None.

Additional details or related context

The issue was reproduced in a multi-node AWS EKS cluster using the Amazon VPC CNI.

Troubleshooting established the following:

• Nginx was listening on 0.0.0.0:80 in the Certbot pod.
• The Certbot Service and Endpoint correctly targeted the pod on TCP port 80.
• Same-node pod-to-Certbot connectivity succeeded.
• Cross-node pod-to-Certbot connectivity timed out.
• Cross-node access to an application pod using TCP port 4001 succeeded.
• The worker-node security group allowed node-to-node TCP traffic only on ports 1025-65535.
• Adding a self-referencing TCP port 80 rule to the worker-node security group restored cross-node connectivity.
• SSL certificate validation completed successfully after the networking rule was added.

[github-actions]

Thank you for the pull request.

ROADMAP STATUS: This pull request isn't currently on any roadmap. Updates will be conveyed here as its place on/off a roadmap changes.

You can view the roadmaps here: Roadmaps Google Drive folder.

For more information on how the roadmaps work, see our roadmaps policy on GitHub.