Skip to content

llm-function-enablement.md

Latest commit

 

History

History
1135 lines (941 loc) · 46.4 KB

llm-function-enablement.md

File metadata and controls

1135 lines (941 loc) · 46.4 KB
Raw

LLM Function Enablement

Enable the LLM addon before creating or invoking functions with functionType: "LLM" through the LLM invocation route. The addon deploys the LLM API Gateway and LLM request router, creates the external LLM invocation route, and configures worker pods to use the pylon sidecar for model-aware routing.

For LLM function payload shape and invocation examples, see Function Creation and LLM Gateway. For request-router deployment, trusted headers, and rollout validation, see LLM Request Router Load Balancing.

When to Enable

Enable the LLM addon when NVCF should route OpenAI-compatible requests by function and model through llm.invocation.<domain>. The gateway extracts the function ID from the OpenAI model field, applies LLM-specific validation and rate limits, and sends the request through the LLM request router.

Standard HTTP, gRPC, and LLS functions do not require this addon, even when a container exposes paths such as /v1/chat/completions, /v1/responses, or /v1/embeddings.

When enabled, the stack creates:

  • llm-api-gateway in the nvcf namespace.
  • llm-request-router in the nvcf namespace.
  • The llm.invocation.<domain> HTTPRoute when Gateway API ingress is enabled.
  • LLM worker pods with a pylon sidecar that forwards requests to the function container on the configured inferencePort.

Enable on an Existing Installation

The addon can be enabled on a running control plane after the initial install or after a stack upgrade. Complete and validate the stack upgrade first, then enable the addon as a separate change. This keeps the upgrade diff small and separates the worker network dependencies from the upgrade itself.

Enabling the addon adds releases and does not restart the running core services:

  • New releases: nvcf-pki in cert-manager, llm-request-router and llm-api-gateway in nvcf. cert-manager is already part of the base stack unless certManager.enabled: false.
  • Changed resources: the OpenBao migration Job, which creates the services/all/pki/root mount and the LLM policies, the NVCF API remote configuration, and the ingress release, which adds the llm.invocation.<domain> HTTPRoute and the LLM worker routes when enabled.
  • No existing Deployment or StatefulSet changes. Running non-LLM functions keep serving.

Before applying:

  1. Mirror the addon artifacts when the registry is private. The addon needs the helm-nvcf-pki, helm-nvcf-llm-request-router, and helm-nvcf-llm-api-gateway charts and the stargate, llm-api-gateway, pylon, and nvcf-worker-llm-credentials-oss images. Take versions from the manifest. The pylon and nvcf-worker-llm-credentials-oss images are pulled by worker pods in every compute cluster, not by the control plane. A missing pylon image appears only when the first LLM worker starts, as ImagePullBackOff on the llm-worker container.

  2. For remote compute clusters, create the Gateway listeners and provision DNS, load balancer, and firewall paths for ports 50071 and 50072 before applying, so the LLM worker routes attach on the first apply. See Remote compute clusters and regions.

  3. Review the rendered change before applying:

    cd path/to/nvcf-self-managed-stack
HELMFILE_ENV=<environment-name> helmfile --environment default diff --suppress-secrets

    Expect new nvcf-pki, llm-request-router, and llm-api-gateway releases and changes to the migration Job, the API remote configuration, and ingress. Stop if the diff changes an existing Deployment or StatefulSet you did not intend to change.

Export the control-plane profile only after the apply completes. The migration Job creates the OpenBao PKI mount during the apply. Before that, control-plane profile export fails with an HTTP 403 from the OpenBao PKI endpoint.

The order is:

  1. Add the addon values to the environment file. See Production TLS Configuration and, for separate GPU clusters, Remote compute clusters and regions.
  2. Apply the control-plane stack and wait for the issuer and certificates. See Apply and Verify.
  3. Export a refreshed profile, add requestRouterAddress for remote clusters, register each compute cluster again, and install the refreshed registration values. Cluster identity is preserved; compare clusterID and clusterGroupID in the registration values with the existing NVCFBackend status before installing.
  4. Deploy an LLM function and run the end-to-end check.

Production TLS Configuration

Production deployments secure two independent worker-to-router paths:

Path TLS identity Pylon trust input
gRPC registration and watches The external HTTPS dial hostname, normally an NLB DNS name Enabled system and public roots, augmented by the gRPC override or the reverse-mode QUIC trust bundle
QUIC reverse tunnel The advertised request-router pod hostname --tls-cert-path for the existing QUIC identity or trust input

The current production gRPC design uses a public ACM certificate on the NLB TLS listener. The NLB certificate covers the public gRPC dial hostname. An unmanaged Pylon process, or one using NVCA system trust mode, can leave both trust paths unset so Tonic uses its enabled system and public roots. In reverse mode, Pylon reuses --tls-cert-path as gRPC trust when the gRPC-specific override is unset. NVCA bundle mode points both options at its merged CA file.

--grpc-tls-ca-cert-path or STARGATE_GRPC_TLS_CA_CERT_PATH is the optional gRPC-specific override. A configured custom bundle augments the enabled system and public roots. It does not replace them. In direct mode, --tls-cert-path is the Pylon QUIC server identity and is never treated as a gRPC CA bundle.

A private-CA certificate on the NLB listener is a supported alternative. Give Pylon the CA bundle with --grpc-tls-ca-cert-path <path> or STARGATE_GRPC_TLS_CA_CERT_PATH=<path>. The input is a PEM bundle containing the CA certificates required by the selected certificate chain, including a private root when applicable. It does not disable certificate or hostname verification. Pylon reads the file once during startup. An unreadable path fails startup with the configured path in the error. Invalid PEM, untrusted chains, and hostname mismatches prevent the gRPC watch and registration connections. Replace or rotate the bundle with a rolling restart of the worker pods. The pylonGrpcDialAddress override must be an explicit https:// URI when a custom CA is configured. The separate global.workerEndpoints.llmRequestRouterAddress input remains a scheme-less host:port initial address.

For gRPC, TLS SNI and hostname verification always use the external HTTPS dial hostname. After discovery, Pylon separately sends the concrete request-router pod hostname as the HTTP/2 :authority so stargate-k8s-router can select the pod. The authority override does not become the TLS SNI. A gRPC NLB leaf certificate therefore needs the public NLB DNS SAN, not Kubernetes-internal pod SANs.

The QUIC identity remains separate. The request router presents a certificate issued by cert-manager, or one you issue and supply in a pre-created Secret. For a single-cluster deployment, initial gRPC discovery defaults to llm-request-router-backend-router.nvcf.svc.cluster.local:50071. For remote workers, the backend router preserves the advertised request-router pod hostname as the QUIC SNI while it sends traffic through an external UDP endpoint. The QUIC certificate must cover that advertised hostname. The default wildcard SAN is *.llm-request-router-headless.nvcf.svc.cluster.local. It does not need the external UDP load-balancer hostname. Do not reuse the gRPC NLB leaf certificate as the QUIC leaf certificate.

Managed OpenBao issuer

The default managed configuration installs ClusterIssuer/nvcf-openbao-pki through the helm-nvcf-pki chart. It uses the stack's OpenBao service as the signing backend.

Add the following values to the Helmfile environment:

certManager:
  enabled: true

openbao:
  enabled: true

addons:
  llm:
    enabled: true
    pki:
      enabled: true
      allowedDomains: nvcf.svc.cluster.local
      dnsNames:
        - llm-request-router.nvcf.svc.cluster.local
        - "*.llm-request-router-headless.nvcf.svc.cluster.local"
    gateway:
      replicaCount: 2
    requestRouter:
      replicaCount: 2

The stable Service SAN covers the in-cluster Service name. The wildcard SAN covers the pod-specific headless names advertised through the backend-router path.

To use a custom managed ClusterIssuer, set both the issuer identity and the management override:

addons:
  llm:
    pki:
      issuerKind: ClusterIssuer
      issuerName: custom-openbao-pki
      clusterIssuer:
        enabled: true

Issuer management supports only issuerKind: ClusterIssuer. OpenBao must be enabled when the stack manages the issuer.

External issuer

Set clusterIssuer.enabled: false when the issuer is managed outside this stack. For an external ClusterIssuer, use:

addons:
  llm:
    enabled: true
    pki:
      enabled: true
      issuerKind: ClusterIssuer
      issuerName: external-llm-pki
      clusterIssuer:
        enabled: false
      dnsNames:
        - llm-request-router.nvcf.svc.cluster.local
        - "*.llm-request-router-headless.nvcf.svc.cluster.local"

For a namespaced issuer, create the Issuer in the nvcf namespace and use:

addons:
  llm:
    enabled: true
    pki:
      enabled: true
      issuerKind: Issuer
      issuerName: external-llm-pki
      clusterIssuer:
        enabled: false
      dnsNames:
        - llm-request-router.nvcf.svc.cluster.local
        - "*.llm-request-router-headless.nvcf.svc.cluster.local"

The external issuer must allow the requested DNS names and issue a server certificate from the root CA distributed to the compute planes. You can set openbao.enabled: false when no other stack component requires OpenBao.

addons.llm.pki.allowedDomains constrains the managed OpenBao signing role only. The stack ignores it for an external issuer. Apply the equivalent constraint in the external issuer's own configuration.

When addons.llm.enabled is true, the stack defaults global.workerEndpoints.llmRequestRouterAddress to llm-request-router-backend-router.nvcf.svc.cluster.local:50071 when backend routing is enabled, otherwise llm-request-router.nvcf.svc.cluster.local:50071. For a split control-plane and compute-plane deployment, override this value with a host and port that worker pods can reach.

The stack maps the configured or default address to api.remoteConfig.configData.nvcf.llm-request-router.worker-address. The NVCF API then includes the address in LLM worker configuration. Do not configure the worker address under api.env. When the LLM addon is disabled, the stack does not pass a staged endpoint to the API chart.

Remote compute clusters and regions

One self-managed control plane can serve LLM workers in separate GPU clusters or regions when every worker cluster has routable DNS and network paths to both backend-router endpoints.

Path Worker-facing values Gateway route and backend Terminating component
gRPC registration and watches global.workerEndpoints.llmRequestRouterAddress for the initial connection, then pylonGrpcDialAddress llmGrpc HTTPS listener and GRPCRoute to llm-request-router-backend-router:50071 over h2c The Gateway terminates TLS. The backend router selects a request-router pod from the HTTP/2 authority and proxies the stream.
Reverse inference tunnel pylonReverseTunnelDialAddress llmQuic UDP listener and UDPRoute to llm-request-router-backend-router:50072 The backend router terminates the worker-facing QUIC connection, selects a request-router pod from SNI, and forwards the tunnel.

Configure distinct HTTPS and UDP endpoints when the infrastructure uses separate Gateways or load balancers:

global:
  workerEndpoints:
    llmRequestRouterAddress: https://llm-grpc.example.com:50071

addons:
  llm:
    enabled: true
    pki:
      # Include the external gRPC suffix when using the stack-managed issuer.
      allowedDomains: cluster.local,example.com
    requestRouter:
      grpcTls:
        enabled: true
        mode: certManager
        secretName: llm-grpc-tls
        dnsNames:
          - llm-grpc.example.com
        issuerRef:
          kind: ClusterIssuer
          name: nvcf-openbao-pki
      backendRouter:
        pylonGrpcDialAddress: https://llm-grpc.example.com:50071
        pylonReverseTunnelDialAddress: llm-quic.example.com:50072

ingress:
  gatewayApi:
    routes:
      llmWorker:
        enabled: true
        backend:
          namespace: nvcf
    gateways:
      llmGrpc:
        name: llm-grpc-gateway
        namespace: envoy-gateway
        listenerName: llm-grpc
      llmQuic:
        name: llm-quic-gateway
        namespace: envoy-gateway
        listenerName: llm-quic

Set both pylonGrpcDialAddress and pylonReverseTunnelDialAddress, or omit both to use the in-cluster backend-router Service. Helmfile rendering rejects a partial override. The gRPC worker address normally uses the same HTTPS endpoint as pylonGrpcDialAddress. A secure external route requires the same explicit https:// URI in both values and grpcTls.enabled: true. Development-only plaintext requires http:// in both values and grpcTls.allowInsecureHttp: true; a scheme-less external route is rejected. Port 443 alone does not make a connection secure.

To recursively discover request routers in another region, set explicit remote Watch dial URIs on the self-managed operator surface:

addons:
  llm:
    requestRouter:
      discovery:
        remoteWatchUrls:
          - https://region-b-watch.example.com:50071

Each URI must use https://. Development-only plaintext endpoints require an explicit http:// URI and addons.llm.requestRouter.discovery.allowInsecureRemoteWatchHttp: true. Scheme-less and unsupported endpoints are rejected rather than defaulting to plaintext. For an HTTPS URI, the dial hostname selects TLS SNI. Identities advertised by the remote Watch response remain the HTTP/2 authorities used for registration.

With grpcTls.mode: certManager, the gateway-routes chart creates a dedicated Certificate in the gRPC Gateway namespace. With mode: existingSecret, the operator must create the named TLS Secret in that namespace. In both modes the Gateway HTTPS listener must reference the same Secret. This identity is separate from the QUIC certificate on port 50072. The operator must ensure an existing certificate covers the external dial hostname; dnsNames is required only when the chart requests the certificate.

For an HTTPS dial URI, the gRPC dial hostname is the TLS SNI and must be a SAN on the Gateway listener certificate. It does not replace the advertised request-router pod hostname used as the HTTP/2 authority or as the QUIC SNI. The dedicated HTTPS listener and LLM GRPCRoute therefore do not set hostname or hostnames. The listener serves the dedicated certificate for normal public-SNI verification, while the backend router receives the selected Stargate identity in :authority. Setting the listener hostname to the public dial name would also constrain :authority and reject these streams. Envoy Gateway's route-scoped traffic policy disables the request and maximum stream durations for long-lived Watch and Register streams. Keep the default wildcard SAN on the separate QUIC leaf, or issue a certificate that covers a customized advertised hostname. Every compute cluster must trust the required issuing CAs as described in Compute-plane trust. Server certificate and key updates follow Certificate Renewal. A CA or trust-bundle change requires a worker rollout.

The stack does not provision cross-region networking, firewall rules, load balancers, DNS, or trust distribution. Supply those dependencies and register the trust bundle in every compute cluster. A successful Helm render and accepted Gateway routes do not prove remote TCP or UDP reachability. Test both paths and an LLM invocation from each region before production use.

The request router uses power-of-n when no load-balancer configuration is set, and accepts any supported routingMethod from a function. When a load-balancer configuration is set, a function can only select an algorithm that the configuration enables.

Pre-created request-router Secret

Set mode: existingSecret when you already issue the request-router server certificate yourself and want the stack to mount it without managing issuance:

addons:
  llm:
    enabled: true
    pki:
      enabled: true
      mode: existingSecret
      secretName: stargate-quic-tls

The stack renders no Certificate, installs no ClusterIssuer, and adds no cert-manager or OpenBao dependency for the request router. You can set certManager.enabled: false and openbao.enabled: false when no other stack component needs them.

Create the Secret in the nvcf namespace before installing the stack. It must carry the tls.crt and tls.key entries, as a kubernetes.io/tls Secret does:

kubectl create secret tls stargate-quic-tls \
  --namespace nvcf \
  --cert path/to/tls.crt \
  --key path/to/tls.key

clusterIssuer.enabled, dnsNames, and allowedDomains only steer stack-managed issuance. Rendering fails if any of them is set in this mode, so a configuration that expects the stack to issue a certificate cannot be mistaken for one that expects you to.

The certificate must carry a SAN covering the router's advertised hostname. The stack enables backend routing and advertises per-pod headless names, so include *.llm-request-router-headless.nvcf.svc.cluster.local along with the stable llm-request-router.nvcf.svc.cluster.local name. An external TCP or UDP dial hostname does not need to be a SAN unless the advertised identity is also customized to use that hostname. The stack cannot read your Secret at render time, so it validates neither the SANs nor the expiry. A certificate that does not cover the advertised hostname fails at worker connection time, not at install time.

You own issuance, renewal, rotation, and recovery in this mode:

  • Renewal and rotation: hot reload requires helm-nvcf-llm-request-router 1.10.0 or later and Stargate 0.11.1 or later. With those versions, update the Secret and follow the transport TLS rotation runbook. Older chart releases and Stargate image overrides older than 0.11.1 require a request-router restart after the Secret update.
  • Expiry: track it yourself. Nothing in the stack renews the certificate or alerts on an approaching expiry.
  • Recovery: a running router keeps its last-known-good identity after rejecting a malformed replacement. A new pod cannot start from a missing or malformed Secret. Restore a valid Secret and follow the rotation runbook to verify the active identity.

Compute-plane trust works exactly as described in Compute-plane trust. Author the transportTls block in the control-plane profile by hand with the public root CA that signed your certificate, as you would for an external issuer. The profile exporter sources a bundle only from the managed OpenBao hierarchy, so it leaves the block empty here. The bundle must contain CERTIFICATE blocks only. Profile validation rejects a private key, so the request-router private key never reaches the compute plane.

External cert-manager

Set certManager.enabled: false when cert-manager is installed and managed outside this stack:

certManager:
  enabled: false

Install the cert-manager CRDs and controller before applying the NVCF stack. When the stack manages the OpenBao issuer, the external installation must also provide ServiceAccount/cert-manager in the cert-manager namespace because the managed issuer uses Kubernetes authentication with that identity.

Compute-plane trust

The control-plane profile carries the public transport CA:

transportTls:
  trustMode: bundle
  trustBundleFingerprint: sha256:<64-lowercase-hex-digits>
  trustBundlePem: |
    -----BEGIN CERTIFICATE-----
    <public-root-ca-pem-body>
    -----END CERTIFICATE-----

For the managed OpenBao issuer, export a refreshed profile after the managed PKI hierarchy is available:

nvcf-cli self-hosted \
  --control-plane-stack deploy/stacks/self-managed \
  --env <environment-name> \
  --control-plane-context <control-plane-context> \
  --compute-plane-context <compute-plane-context> \
  control-plane profile export \
  --cluster-name <control-plane-cluster-name> \
  --nca-id <nca-id> \
  --region <region>

The exporter cannot infer the worker-facing request-router endpoint. Add the reachable host:port before validating or registering the profile:

controlPlane:
  addons:
    llm:
      requestRouterAddress: llm-router.example.com:443

Use the gRPC endpoint that compute-plane workers can resolve and reach. This is a dial endpoint. It does not replace the advertised request-router hostname that the certificate covers.

Registration renders this field as agent.llm.requestRouterAddress in the compute-plane values. That is operator configuration, not a runtime fallback for workers. At this release the worker sidecar takes its --stargate-address from the LLM_REQUEST_ROUTER_ADDRESS variable in its launch environment, with STARGATE_ADDRESS accepted as a legacy alias. The NVCF API injects LLM_REQUEST_ROUTER_ADDRESS on the normal launch path, derived from global.workerEndpoints.llmRequestRouterAddress. If neither variable reaches the workload, translation rejects the launch instead of falling back to the registered address.

Keep the default stable and wildcard entries in addons.llm.pki.dnsNames. Add another SAN only when the advertised request-router identity changes, not when only a load-balancer dial endpoint changes. For the managed issuer, any new advertised DNS suffix must also be covered by allowedDomains. The OpenBao role allows subdomains and wildcards but not bare domains, so configure the parent domain rather than the exact leaf name.

The managed export reads the public root CA certificate from services/all/pki/root in the stack's OpenBao service and calculates the canonical NVCF trust-bundle fingerprint. It does not discover the CA for an external issuer. For an external issuer, add the issuer owner's public CA bundle and canonical NVCF fingerprint to transportTls in the existing profile.

Calculate the canonical fingerprint for a public CA bundle with:

set -euo pipefail

bundle_file=path/to/public-ca-bundle.pem
fingerprint_tmp="$(mktemp -d)"
trap 'rm -rf "$fingerprint_tmp"' EXIT

awk -v output_dir="$fingerprint_tmp" '
  /-----BEGIN CERTIFICATE-----/ { certificate++; in_certificate=1 }
  in_certificate {
    print > (output_dir "/certificate-" certificate ".pem")
  }
  /-----END CERTIFICATE-----/ { in_certificate=0 }
' "$bundle_file"

for certificate_file in "$fingerprint_tmp"/certificate-*.pem; do
  openssl x509 -in "$certificate_file" -outform DER \
    | openssl dgst -sha256 -r \
    | awk '{print $1}'
done | sort -u >"$fingerprint_tmp/certificate-hashes"

{
  printf 'nvcf-trust-bundle-v1\n'
  cat "$fingerprint_tmp/certificate-hashes"
} | openssl dgst -sha256 -r \
  | awk '{print "sha256:" $1}'

The procedure hashes each certificate's DER bytes, removes duplicates, sorts the certificate hashes, and hashes the versioned canonical text. Use the result for trustBundleFingerprint.

Validate the profile after updating it:

nvcf-cli self-hosted control-plane profile validate \
  --file deploy/stacks/self-managed/out/control-plane-profile.yaml \
  --require compute-reachable

Compute-plane registration converts transportTls into this exact generated NVCA fragment:

agentConfig:
  mergeConfig: |
    workload:
      transportTLS:
        trustMode: bundle
        trustBundleFingerprint: sha256:<64-lowercase-hex-digits>
        trustBundlePem: |
          -----BEGIN CERTIFICATE-----
          <public-root-ca-pem-body>
          -----END CERTIFICATE-----

Use only public CA certificates in trustBundlePem. Do not add a private key, leaf certificate, or OpenBao token.

In bundle mode, NVCA creates one merged system and private CA file in the llm-worker container and explicitly points both Pylon trust inputs at it:

STARGATE_TLS_CERT_PATH=/etc/ssl/certs/ca-certificates.crt
STARGATE_GRPC_TLS_CA_CERT_PATH=/etc/ssl/certs/ca-certificates.crt

Sharing this CA bundle does not couple the protocols or reuse a leaf certificate. STARGATE_TLS_CERT_PATH retains its QUIC meaning. STARGATE_GRPC_TLS_CA_CERT_PATH applies only to Stargate gRPC HTTPS. The bundle can contain roots for both independent leaf-certificate hierarchies. In system trust mode, NVCA injects neither variable and Pylon keeps Tonic's default HTTPS root behavior.

For an unmanaged Pylon process, set the equivalent --grpc-tls-ca-cert-path <path> option directly. The file is read once at startup and is not reloaded.

When replacing a local plaintext configuration, also set this in the compute-plane Helmfile environment:

agentConfig:
  mergeConfig: |
    workload:
      stargateQUICInsecure: false

The compute-plane Helmfile merges the environment fragment over the generated registration fragment, so the TLS trust configuration is retained and an old plaintext override is disabled.

If you mirror images to a registry that does not use the stack's default global.image.registry and global.image.repository, override the pylon sidecar image passed to generated LLM workers:

api:
  remoteConfig:
    configData:
      nvcf:
        sidecars:
          llm-router-client-image: <registry>/<repository>/pylon:0.14.1

api.env.NVCF_SIDECARS_LLM_ROUTER_CLIENT_IMAGE is deprecated. The stack translates it into remote config for one compatibility window and omits it from the API environment. Do not set both paths to different values.

The LLM API Gateway and request router images are resolved from the same stack artifact registry settings as the other control-plane services.

The self-managed stack passes API environment and remote configuration through separate chart values:

flowchart LR
    E[Helmfile environment] --> T[global.yaml.gotmpl]
    F[Fixed API environment] --> T
    T -->|mergeOverwrite into api.env| V[NVCF API chart values]
    E -->|api.remoteConfig.configData| T
    T -->|merged remote config| V
    V --> C1[nvcf-api-env ConfigMap]
    V --> C2[nvcf-api-remote-config ConfigMap]
    C1 --> A[NVCF API startup]
    C2 --> A
    A --> W[LLM worker launch metadata]
Loading

Configured api.env entries override the fixed stack environment. The remote ConfigMap carries the request-router worker address and Pylon image used when the API creates a new LLM function version.

Apply and Verify

Apply the updated control-plane environment:

cd path/to/nvcf-self-managed-stack
make apply HELMFILE_ENV=<environment-name>

For the managed issuer, wait for the ClusterIssuer:

kubectl wait --for=condition=Ready \
  clusterissuer/nvcf-openbao-pki \
  --timeout=2m
kubectl get clusterissuer nvcf-openbao-pki

For an external issuer, wait for the configured resource instead:

kubectl wait --for=condition=Ready \
  clusterissuer/<issuer-name> \
  --timeout=2m
# For a namespaced Issuer:
kubectl -n nvcf wait --for=condition=Ready \
  issuer/<issuer-name> \
  --timeout=2m

Set REQUEST_ROUTER_TLS_NAME to the value of addons.llm.pki.secretName. Its default is stargate-quic-tls. In certManager mode, verify the request-router Certificate and its issuer reference. The existingSecret mode does not render a Certificate, so skip the first two commands in that mode. The final command applies to both modes and reads only the public certificate:

export REQUEST_ROUTER_TLS_NAME="<configured-request-router-tls-name>"
kubectl -n nvcf wait --for=condition=Ready \
  "certificate/$REQUEST_ROUTER_TLS_NAME" \
  --timeout=2m
kubectl -n nvcf get certificate "$REQUEST_ROUTER_TLS_NAME" \
  -o jsonpath='{.spec.issuerRef.kind}{"/"}{.spec.issuerRef.name}{"\n"}'
kubectl -n nvcf get secret "$REQUEST_ROUTER_TLS_NAME" \
  -o jsonpath='{.data.tls\.crt}' \
  | base64 --decode \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Regenerate and apply the registration values for each compute cluster:

nvcf-cli self-hosted \
  --control-plane-stack deploy/stacks/self-managed \
  --compute-plane-stack deploy/stacks/nvcf-compute-plane \
  --env <environment-name> \
  --control-plane-context <control-plane-context> \
  --compute-plane-context <compute-plane-context> \
  compute-plane register \
  --control-plane-profile deploy/stacks/self-managed/out/control-plane-profile.yaml \
  --cluster-name <compute-plane-cluster-name> \
  --kube-context <compute-plane-context> \
  --region <region> \
  --output deploy/stacks/nvcf-compute-plane/out/<compute-plane-cluster-name>-register-values.yaml

nvcf-cli self-hosted \
  --compute-plane-stack deploy/stacks/nvcf-compute-plane \
  --env <environment-name> \
  compute-plane install \
  --values deploy/stacks/nvcf-compute-plane/out/<compute-plane-cluster-name>-register-values.yaml \
  --kube-context <compute-plane-context> \
  --cluster-name <compute-plane-cluster-name>

Existing LLM function versions retain the worker-sidecar image metadata captured when the version is created. Replacing pods or redeploying the same version does not apply a new Pylon image. After updating the control plane, create and deploy a new function version.

After deploying an LLM function, verify the workload trust bundle. Compare only .data.fingerprint with transportTls.trustBundleFingerprint in the profile. openssl x509 -fingerprint does not calculate the canonical NVCF bundle fingerprint.

kubectl -n nvcf-backend get configmap nvcf-transport-trust-bundle \
  -o jsonpath='{.data.fingerprint}{"\n"}'
kubectl -n nvcf-backend get configmap nvcf-transport-trust-bundle \
  -o jsonpath='{.data.nvcf-ca-bundle\.pem}' \
  | openssl x509 -noout -subject -issuer

Verify the worker sidecar:

kubectl get pods -n nvcf-backend -L FUNCTION_ID
kubectl -n nvcf-backend get pod <function-pod> \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'
kubectl -n nvcf-backend get pod <function-pod> \
  -o jsonpath='{range .spec.containers[?(@.name=="llm-worker")].args[*]}{.}{"\n"}{end}'
kubectl -n nvcf-backend get pod <function-pod> \
  -o jsonpath='{range .spec.containers[?(@.name=="llm-worker")].env[*]}{.name}{"="}{.value}{"\n"}{end}' \
  | grep -E '^STARGATE_(GRPC_TLS_CA_CERT_PATH|TLS_CERT_PATH)='

The worker args must contain --stargate-address=llm-request-router-backend-router.nvcf.svc.cluster.local:50071 with default backend routing, or the configured routable endpoint, and must not contain --quic-insecure. The external address is the initial gRPC dial endpoint. The reverse tunnel verifies the advertised request-router pod hostname instead. The environment must contain:

STARGATE_TLS_CERT_PATH=/etc/ssl/certs/ca-certificates.crt
STARGATE_GRPC_TLS_CA_CERT_PATH=/etc/ssl/certs/ca-certificates.crt

Also verify the control-plane components:

kubectl get deployment -n nvcf llm-api-gateway
kubectl get deployment -n nvcf llm-request-router-backend-router
kubectl get deployment,statefulset -n nvcf llm-request-router
kubectl get service -n nvcf llm-request-router-backend-router
kubectl get pods -n nvcf | grep -E 'llm-api-gateway|llm-request-router'
kubectl get httproute -A | grep llm

For remote workers, also verify the LLM GRPCRoute, UDPRoute, dedicated gRPC Certificate, stream timeout policy, and ReferenceGrant as described in Gateway Routing and DNS.

End-to-end check

Deploy one LLM function and invoke it through the LLM invocation route. For the function payload, see LLM Gateway. Wait for the deployment to become ACTIVE, then confirm the sidecar registered:

kubectl -n nvcf-backend logs <function-pod> -c llm-worker --tail=20

Invoke through the gateway. GATEWAY_ADDR is the control-plane gateway address and DOMAIN is the route suffix:

curl -sSf "http://${GATEWAY_ADDR}/v1/chat/completions" \
  -H "Host: llm.invocation.${DOMAIN}" \
  -H "Authorization: Bearer <api-key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"<function-id>/<model-name>","messages":[{"role":"user","content":"Say hello."}],"max_tokens":32}'

Expect an HTTP 200 with an OpenAI-style completion. For up to about one minute after the deployment becomes ACTIVE, the gateway can return a 404 with no eligible candidates while the sidecar registers with the request routers. For separate GPU clusters, this check is the only proof that the gRPC and UDP worker paths are reachable. Run it from every compute cluster.

Certificate Renewal

cert-manager renews the request-router certificate and updates the Secret named by addons.llm.pki.secretName. The default is Secret/stargate-quic-tls. With mode: existingSecret there is no renewal loop and you update the configured Secret yourself.

Hot reload requires helm-nvcf-llm-request-router 1.10.0 or later and Stargate 0.11.1 or later. The request router polls the mounted server certificate and private key every 30 seconds. A valid replacement becomes active for new handshakes without restarting the request router. Established connections remain open. If a replacement is invalid, the router rejects it and keeps the last-known-good identity. Older chart releases and Stargate image overrides older than 0.11.1 require a request-router restart after the Secret update.

This reload contract covers the server certificate and private key only. Pylon loads both its gRPC CA bundle and QUIC trust bundle at startup. A trust-bundle change, including a root CA rotation, requires a rolling restart of the LLM worker pods. Renewing an intermediate under an already-trusted root does not change the trust bundle and does not require a worker-pod restart. Follow the transport TLS rotation runbook for the atomic Secret update, reload checks, trust-bundle rollout order, and recovery procedure.

Upgrade and Rollback

Use this order for an upgrade from plaintext transport:

  1. Make the helm-nvcf-pki chart available from the configured chart source.

  2. Remove the old plaintext setting or set workload.stargateQUICInsecure: false.

  3. Apply the dependency Helmfile stage to install cert-manager, OpenBao, and the managed issuer, or prepare the external cert-manager and issuer:

    HELMFILE_ENV=<environment-name> helmfile \
  --file deploy/stacks/self-managed/helmfile.d/01-dependencies.yaml.gotmpl \
  --environment default \
  apply

  4. Schedule a maintenance window and undeploy the existing LLM functions. The router does not support a mixed plaintext and TLS transition.

  5. Apply the remaining control-plane stack. The managed router hook prepares the OpenBao signing path before cert-manager reconciles the request-router Certificate.

  6. Wait for the issuer and the Certificate named by addons.llm.pki.secretName to become ready. The default name is stargate-quic-tls.

  7. Export or update the control-plane profile, register each compute plane again, and install the refreshed registration values.

  8. Recreate the LLM functions and verify the certificate SAN, trust-bundle fingerprint, worker address, and worker arguments.

Use this order for a safe rollback:

  1. Create and verify the replacement ClusterIssuer or namespaced Issuer.

  2. Add both the current and replacement public roots to the compute-plane profile, calculate the canonical bundle fingerprint, register each compute plane again, and recreate the LLM workers with the combined trust bundle.

  3. Set addons.llm.pki.issuerKind and issuerName to the replacement, set clusterIssuer.enabled: false, and apply the control-plane stack.

  4. Wait for the Certificate named by addons.llm.pki.secretName to become ready, confirm a successful server_identity reload, and verify the replacement TLS data path with a new connection.

  5. Remove the old root from the compute-plane profile, register each compute plane again, and recreate the workers.

  6. Confirm that no Certificate references the old issuer:

    kubectl --context <control-plane-context> get certificate -A \
  -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,KIND:.spec.issuerRef.kind,ISSUER:.spec.issuerRef.name

  7. Remove the old managed release after it is no longer part of the Helmfile state. Run helm status first and confirm the release is the superseded one, that the previous step listed no Certificate still referencing its issuer, and that the replacement data path is verified. Uninstalling is not reversible from the cluster; only proceed once all three hold.

    helm status nvcf-pki \
  --namespace cert-manager \
  --kube-context <control-plane-context>

    After confirming, remove the release:

    helm uninstall nvcf-pki \
  --namespace cert-manager \
  --kube-context <control-plane-context>

  8. Delete the retained old ClusterIssuer only after all references are gone:

    kubectl --context <control-plane-context> \
  delete clusterissuer <old-issuer-name>

If no replacement issuer is available, undeploy the LLM functions and disable the LLM addon before removing the issuer. Keep LLM traffic stopped until a secure issuer and trust path are available.

The managed ClusterIssuer is retained when its Helm release is removed. Do not delete it before its certificates and consumers have moved to the replacement trust path. Do not use stargateQUICInsecure as a production rollback path.

Disable the addon

To remove the LLM addon from a running control plane:

  1. Undeploy every LLM function. Standard HTTP, gRPC, and LLS functions are not affected.

  2. Set addons.llm.enabled: false in the environment file, or restore the environment file from before enablement, and apply the control-plane stack. This removes the llm.invocation.<domain> route and clears the LLM settings in the NVCF API remote configuration, so new LLM functions can no longer be invoked or launched with the sidecar. No running Deployment or StatefulSet changes.

  3. Remove the addon releases. With the addon disabled, the Helmfile no longer contains these releases, so the apply leaves them installed and running instead of uninstalling them:

    helm uninstall llm-api-gateway --namespace nvcf
helm uninstall llm-request-router --namespace nvcf
helm uninstall nvcf-pki --namespace cert-manager

    Skip nvcf-pki to keep issuing certificates from the managed issuer. The OpenBao PKI mount and the managed ClusterIssuer are retained in both cases, so a later control-plane profile export still succeeds and re-enabling the addon reuses the existing hierarchy.

  4. Registering the compute clusters again is optional. The registered trust bundle and agent.llm.requestRouterAddress are unused without LLM functions and do not affect other function types.

Gateway Authentication and Worker Transport

The gateway authentication hop and the worker transport are independent:

  • The bundled LLM API Gateway authenticates requests through the bundled NVCF API at api.nvcf.svc.cluster.local:9090. The bundled API currently serves plaintext gRPC on that port, so the self-managed stack defaults addons.llm.gateway.auth.grpcInsecure to true.
  • LLM workers use separately managed transport security for gRPC registration, watches, and the QUIC request path to the LLM request router. Keep the production TLS configuration described above for this path.

Typical self-managed deployments do not need to set the gateway authentication transport. Set addons.llm.gateway.auth.grpcInsecure: false only when the NVCF API endpoint is configured to serve TLS-capable gRPC.

For local or isolated test clusters that do not configure worker transport TLS, set only the worker plaintext control:

addons:
  llm:
    enabled: true
    gateway:
      replicaCount: 1
    requestRouter:
      replicaCount: 1

agentConfig:
  mergeConfig: |
    workload:
      stargateQUICInsecure: true

workload.stargateQUICInsecure: true configures generated LLM workers to pass the plaintext QUIC setting to the pylon sidecar.

Do not enable plaintext worker transport in production. This warning applies to the worker-to-router transport, not the bundled in-cluster gateway-to-API authentication hop.

Error Responses

The LLM API Gateway returns errors in the format OpenAI and Anthropic SDKs read. Errors from the gateway and the LLM request router use the OpenAI format on /v1/chat/completions, /v1/responses, /v1/embeddings, and the other OpenAI-compatible endpoints:

{"error":{"message":"no eligible candidates","type":"invalid_request_error","param":null,"code":null}}

On /v1/messages, they use the Anthropic format:

{"type":"error","error":{"type":"not_found_error","message":"no eligible candidates"},"request_id":"<request-id>"}

The type follows the HTTP status. For example, 401 is authentication_error, 429 is rate_limit_error, and 529 is overloaded_error. On OpenAI-compatible endpoints, 529 always has code: "overloaded_error".

Model server errors already in the endpoint's format are returned unchanged, including the server's type and code. Other model server bodies are converted to the endpoint's format, keeping up to 32 KiB of the server's message when the gateway can read it. The gateway reads uncompressed and gzip error bodies up to 1 MiB. For larger bodies or other encodings, the message is the HTTP status text.

To return the request router's rejection reason, such as no_eligible_candidates, in error.code, set llmApiGateway.config.exposeRouterErrorCodes: true in the LLM API Gateway chart. The Anthropic format has no code field, so /v1/messages returns only the message. In either case, the request router logs the reason as rejection_reason and the gateway logs it as router_error_code.

Troubleshooting

A 404 with no eligible candidates from llm.invocation.<domain> means the request reached the LLM Gateway, but the requested function or model was unknown or was not registered on the selected request router. The message is in error.message, as described in Error Responses. Similar 503 candidate errors mean the router knows the target but has no active eligible backend. Check:

  • The deployment became ACTIVE more than about one minute ago. The sidecar registers with the request routers after the pod is Running.
  • The LLM function is deployed and its pod is Running.
  • The request model value uses <function-id>/<model-name>.
  • The function's models[].name matches the model suffix in the request.
  • models[].llmConfig.uris includes the invoked path.
  • When addons.llm.requestRouter.loadBalancer.config is set, it includes the algorithm selected by the function's models[].llmConfig.routingMethod.
  • The llm-worker sidecar connected to llm-request-router.
  • The effective LLM request-router worker address is reachable from the worker cluster.
  • Local clusters without worker transport TLS set workload.stargateQUICInsecure: true. The bundled gateway-to-API authentication hop already defaults to plaintext.

An LLM worker pod with ImagePullBackOff or ErrImagePull on the llm-worker container means the pylon image is not available in the configured registry. Confirm the effective sidecar image reference and mirror it:

kubectl -n nvcf get configmap nvcf-api-remote-config -o yaml \
  | grep llm-router-client-image
kubectl -n nvcf-backend get pod <function-pod> \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'

A 400 from the function container on /v1/chat/completions for a request that works when sent directly to the inference server can come from optional OpenAI fields that the gateway forwards with a null value, such as include_reasoning. Some inference servers reject null for these fields. Send an explicit value, for example "include_reasoning": false.

For transport TLS failures, check:

  • gRPC unknown issuer: for a public ACM NLB certificate, verify the system and public root store. For a private-CA NLB certificate, verify STARGATE_GRPC_TLS_CA_CERT_PATH and the CA chain in that PEM bundle.
  • QUIC unknown issuer: inspect the Certificate Ready condition and verify issuerRef.kind, issuerRef.name, and the issuer namespace. A namespaced Issuer must be in nvcf.
  • gRPC SAN mismatch: compare the external HTTPS dial hostname with the public DNS SANs on the NLB listener certificate. The internal pod authority is not the gRPC TLS identity.
  • QUIC SAN mismatch: compare the request router's --advertised-hostname-template with the SANs in the Secret named by addons.llm.pki.secretName. The default is Secret/stargate-quic-tls. The external UDP dial endpoint does not replace the advertised QUIC identity.
  • Expired or not-yet-valid certificate: inspect the certificate dates and the cluster clock. Renew the certificate and use the transport TLS rotation runbook to verify that the replacement becomes active.
  • System trust mode: confirm STARGATE_TLS_CERT_PATH and STARGATE_GRPC_TLS_CA_CERT_PATH are both unset so Pylon uses its enabled system and public roots.
  • Bundle trust mode: verify ConfigMap/nvcf-transport-trust-bundle, compare its fingerprint with the compute-plane profile, and confirm STARGATE_TLS_CERT_PATH in the llm-worker container. The STARGATE_GRPC_TLS_CA_CERT_PATH override is optional; when unset, the gRPC registration and watch paths reuse the existing Stargate bundle. Restart the worker after changing either bundle.

Useful logs:

kubectl logs -n nvcf deploy/llm-api-gateway --tail=100
kubectl logs -n nvcf statefulset/llm-request-router \
  --all-pods=true --tail=100
kubectl logs -n nvcf-backend <function-pod> -c llm-worker --tail=100

In healthy routing, the request router logs show a reverse tunnel connection from the worker and at least one routing candidate for the requested function.