Skip to content

thv llm: localhost reverse proxy for proxy-mode tools #5029

@yrobla

Description

@yrobla

Context

Part of #5016 (RFC: stacklok/toolhive-rfcs#70). Depends on #5028 (token lifecycle). Can be implemented in parallel with #5028 once #5027 is done.

Scope

  • Localhost reverse proxy listening on http://localhost:<port>/v1
  • Strips incoming Authorization header and injects a fresh OIDC bearer token from the token source
  • Forwards requests to upstream gateway preserving original path, query string, headers, and body
  • Passes through upstream responses faithfully: SSE streaming (chat completions), non-streaming JSON (embeddings, completions), error responses (4xx/5xx) — no buffering or body modification
  • Loopback-only guard: reject startup if listen address is not a loopback interface
  • Background process lifecycle management (start, stop, PID tracking)
  • thv llm proxy start — foreground variant with full log output for debugging

Acceptance Criteria

  • AT: Proxy accepts requests on http://localhost:<port>/v1, strips Authorization, injects fresh OIDC bearer token, forwards to upstream gateway — preserving path, query string, headers, and body
  • AT: Proxy passes through SSE streaming, non-streaming JSON, and error responses (4xx/5xx) without buffering or modification
  • Unit: Proxy rejects startup if listen address is not a loopback interface

Dependencies

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    cliChanges that impact CLI functionalityenhancementNew feature or requestgoPull requests that update go codellm gatewayLLM gateway authentication feature

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions