Kong fails to proxy HTTPS traffic to MinIO

[mattiamarteddu]

Issue Summary

I am experiencing issues with Kong when trying to route traffic to MinIO through HTTPS. The request results in either Connection reset by peer or Client sent an HTTP request to an HTTPS server.

Kong Version

Kong 3.9.0

Configuration Details

I have configured Kong with the following Gateway, HTTPRoute, and ReferenceGrant definitions:

apiVersion: v1
kind: Namespace
metadata:
  name: kong
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong
  namespace: kong
  annotations:
    konghq.com/gatewayclass-unmanaged: 'true'
spec:
  controllerName: konghq.com/kic-gateway-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong
  namespace: kong
spec:
  gatewayClassName: kong
  listeners:
  - name: proxy
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: wildcard-tls
        namespace: kong
    allowedRoutes:
      namespaces:
        from: All

MinIO routes:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: minio-api-route
  namespace: minio
  annotations:
    konghq.com/strip-path: "false"
    konghq.com/protocols: "https"
    konghq.com/backend-protocol: "https"
spec:
  hostnames:
  - minioapi.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: minio-hl
      kind: Service
      namespace: minio
      port: 9000
  parentRefs:
  - name: kong
    namespace: kong

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: minio-console-route
  namespace: minio
  annotations:
    konghq.com/strip-path: "false"
    konghq.com/protocol: "https"
spec:
  hostnames:
  - minio.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: minio-console
      kind: Service
      port: 9443
  parentRefs:
  - name: kong
    namespace: kong

Error Messages

From Kong logs:

proxy 2025/04/02 13:18:36 [error] 1408#0: *2118 readv() failed (104: Connection reset by peer) while reading upstream, client: 10.42.0.1, server: kong,  ││ request: "GET / HTTP/2.0", upstream: "http://10.42.0.49:9443/", host: "minio.example.com", request_id: "4804f1295d96793db14bfa2540632969"

From curl:

* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
< HTTP/1.1 400 Bad Request
< Server: kong/3.9.0
< X-Kong-Upstream-Latency: 0
< X-Kong-Proxy-Latency: 1
< Via: 1.1 kong/3.9.0
< X-Kong-Request-Id: 703b73644fab3d3018c5d2282238811e
Client sent an HTTP request to an HTTPS server.

Steps to Reproduce

1. Deploy MinIO and Kong in a Kubernetes cluster.
2. Configure MinIO services to expose ports 9000 (API) and 9443 (Console).
3. Configure Kong Gateway to terminate TLS and route traffic to MinIO via HTTPS.
4. Send a request using curl -kv https://minio.example.com
5. Observe the error messages.

Expected Behavior

Kong should correctly forward HTTPS requests to MinIO without resetting the connection or triggering HTTP/HTTPS mismatches.

Actual Behavior

Requests to MinIO fail with either a Connection reset by peer error or Client sent an HTTP request to an HTTPS server.

Additional Information

• MinIO is configured to support HTTPS.
• The Kong Gateway is terminating TLS.
• The error suggests that Kong is attempting to communicate with MinIO over HTTP instead of HTTPS despite the konghq.com/backend-protocol: "https" annotation.

Environment Details

• Kubernetes Version: 1.31.6
• MinIO Version: RELEASE.2025-02-07T23-21-09Z
• Kong Ingress Controller Version: 3.9

[StarlightIbuki]

This is duplicated to #14392