Skip to content

feat(observability): verify OTLP collectors with a configured CA and client certificate - #5084

Open
krishicks wants to merge 1 commit into
feat/5067-otlp-collector-headers/krishicksfrom
feat/5080-otlp-collector-tls/krishicks
Open

krishicks wants to merge 1 commit into
feat/5067-otlp-collector-headers/krishicksfrom
feat/5080-otlp-collector-tls/krishicks

Conversation

@krishicks

@krishicks krishicks commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Collectors behind a private CA or requiring mutual TLS were out of reach: gRPC export trusted only the platform roots, OTLP/HTTP and the agent relay refused https:// endpoints, and nothing could present a client certificate. Each [openshell.gateway.otlp] table now takes ca_file, cert_file, and key_file, and every exporter verifies against that CA and presents the client certificate.

Related Issue

Closes #5080

Changes

  • ca_file, cert_file, and key_file on every OTLP table, inherited with the rest of the collector. They require an https:// endpoint, are read at startup, are capped at 64 KiB each, and the key must match the certificate.
  • ca_file replaces the platform roots. Without it, https:// collectors are verified against the platform roots, which honor SSL_CERT_FILE and SSL_CERT_DIR.
  • OTLP/HTTP connections are built through openshell-crypto; gRPC uses tonic's TLS configuration, recorded as dependency-owned in the openshell-crypto README.
  • Supervisors receive each distinct set of PEM material once in their launch credentials, with lanes referring to it by position, so nothing is mounted into sandboxes. The gateway refuses to start when that material would exceed 192 KiB, which keeps launch credentials within a shared supervisor's 256 KiB limit; supervisors reject a bundle whose references fall outside its material. The agent trace relay now supports https:// collectors.
  • The VM driver receives the trace collector's file paths. Standalone compute drivers take --otlp-ca-file, --otlp-cert-file, and --otlp-key-file for their own spans.
  • e2e/rust and tests/suites/features lockfiles pick up the TLS dependencies.
  • Gateway configuration docs cover the keys, inheritance, rotation, and mounting the files in Helm with extraVolumes.

Release Notes

OTLP collectors can use a private CA and mutual TLS: set ca_file, and optionally cert_file and key_file, on [openshell.gateway.otlp] or a signal table. The agent trace relay and OTLP/HTTP export now support https:// collectors.

Testing

  • CI lint locally on this commit: tracked lockfile and alignment checks, cargo fmt --check, and every mise run rust:lint clippy invocation with --locked
  • Targeted tests pass in openshell-otel (lib and integration binaries), openshell-core, openshell-server, openshell-gateway, the supervisor binary, and all four compute drivers
  • Unit tests added/updated: generated test PKI with mutual-TLS collectors over gRPC and HTTP, missing or wrong CA, missing client certificate, wrong hostname, platform roots, file errors, redaction of key material, one copy of shared material per bundle, the 192 KiB limit, and rejected out-of-range references
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@krishicks krishicks added area:docs Documentation and examples topic:observability Logging, metrics, and observability work area:supervisor Proxy and routing-path work topic:security Security issues area:gateway Gateway server and control-plane work area:compute labels Oct 30, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@krishicks
krishicks added this pull request to stack #5085 October 30, 2026 22:12
@github-actions

Copy link
Copy Markdown

@krishicks
krishicks marked this pull request as ready for review October 30, 2026 22:16
@krishicks
krishicks force-pushed the feat/5080-otlp-collector-tls/krishicks branch from f24def8 to ffae7cc Compare October 30, 2026 23:02
…client certificate

Collectors behind a private CA or requiring mutual TLS were out of reach:
gRPC export trusted only the platform roots, OTLP/HTTP and the agent relay
refused https endpoints, and nothing could present a client certificate.

Each [openshell.gateway.otlp] table now takes ca_file, cert_file, and
key_file, inherited with the rest of the collector and read at startup.
Every exporter verifies https collectors against that CA and presents the
client certificate, over gRPC or HTTP, with HTTP connections built through
openshell-crypto. Supervisors receive each distinct material once in their
launch bundle, with a 192 KiB total checked at startup so it fits a shared
supervisor's launch material, the VM driver receives the file paths, and
standalone drivers take --otlp-ca-file, --otlp-cert-file, and
--otlp-key-file.

Closes #5080

Signed-off-by: Kris Hicks <khicks@nvidia.com>
@krishicks
krishicks force-pushed the feat/5080-otlp-collector-tls/krishicks branch from ffae7cc to 37315ba Compare November 2, 2026 15:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:compute area:docs Documentation and examples area:gateway Gateway server and control-plane work area:supervisor Proxy and routing-path work topic:observability Logging, metrics, and observability work topic:security Security issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(observability): verify OTLP collectors with a configured CA and client certificate

1 participant