refactor(apigateway): upstream authentication and transport policy move to a shared seam under internal/, so a second HTTP-based connection kind reuses them instead of copying them (#1647)

[cjimti]

What this changes

Reaching an HTTP upstream is one job, and until now only one connection kind knew how to do it. The Authenticator and its six modes, the operator-owned static headers, the timeouts, the response read cap and the TLS material all lived inside pkg/toolkits/apigateway, private to kind: api.

They now live in internal/upstreamauth, built by any HTTP-based connection kind. internal/cfgmap sits beside it with the typed readers over the map[string]any a stored connection arrives as, so the seam can own its own configuration keys without a sixth copy of getString/getDuration appearing in the tree.

Nothing an operator or a model sees changes: same configuration keys, same tool schemas, same admin routes, same error text.

The seam

internal/upstreamauth holds:

• The Authenticator and every mode — none, bearer, api_key (header or query, any header name), basic, oauth (both grants, through pkg/connoauth), mtls (through internal/apigwtls) — with the token-fetch error scrubber that keeps a client secret or a token URL's userinfo out of any message the model sees.
• The header policy — the operator's static_headers, the header names a model may not claim (Authorization, the one this connection's auth mode owns, and every name the operator pinned), and the hop-by-hop and net/http-managed names an operator may not pin.
• The transport policy — connect and call timeouts, the idle-connection pool, the read cap and its capped read, the reservation against the shared in-flight memory budget, and the CheckRedirect that refuses to follow a 3xx so a redirect cannot carry the connection's credential to a host the operator never configured.
• The configuration keys those rules belong to, and their validation: auth_mode, credential, api_key_header, api_key_param, api_key_placement, username, password, the OAuth keys, connect_timeout, call_timeout, max_response_bytes, static_headers, mtls_client_cert_pem, mtls_client_key_pem, tls_ca_bundle_pem, identity_passthrough.

internal/apigwtls and internal/apigwmetrics keep their names and their contents; the TLS package is now reached through the seam rather than from the toolkit.

How the toolkit's surface survives the move

pkg/toolkits/apigateway keeps its exported Config field for field. Authenticator, NewAuthenticator, ErrNeedsReauth and the AuthMode*, CredentialPlacement* and OAuth2AuthStyle* constants are aliased back, so every name an importer could reach is spelled exactly as before.

Config.upstream() and configFromUpstream() in the new pkg/toolkits/apigateway/upstream.go are the single place the toolkit's config and the seam's are joined. They are a mapping rather than an embed because an internal/ type in a public struct would be unnameable to a library consumer and would trip TestPublicSurfacePolicy; keeping the two in one file is what makes a field added to either side have exactly one place to be joined up.

Three details carry the behavior across intact:

Error text stays the caller's. upstreamauth.Config.ErrPrefix names the calling kind in every message the seam produces, and the same field is threaded through apigwtls.Material. A refused connection save still reads apigateway: credential is required when auth_mode is "bearer" and not the name of a seam that appears in no configuration file. A second kind supplies its own prefix and gets its own voice for free.

ErrNeedsReauth keeps its rendered message and its identity. The sentinel carries no prefix of its own; Apply wraps it in the kind's voice, so the text an operator reads is byte for byte what it was and errors.Is still matches through apigateway.ErrNeedsReauth.

The order an operator sees errors in is unchanged. The seam exposes each validator separately — ValidateAuth, ValidateTransport, ValidateStaticHeaders, ValidateIdentityPassthrough, ValidateTLSMaterial — alongside a composite Validate for a kind with no keys of its own. The toolkit calls them individually so its own base_url, trust_level and max_inline_bytes checks stay interleaved exactly where they were.

upstreamauth.SetConnOAuthStore and SetAuthEvents take an Authenticator and report whether it took the value, replacing the three *oauth2AuthorizationCodeAuth type assertions the toolkit made when threading the OAuth token store over its connections.

What deliberately did not move

The memguard result shapes (budgetError.result, bodyTooLargeResult, structuredErrorResult, nonInlineableBodyError) are tool-surface wording, not transport, and stay in the toolkit. So does internal_transport.go: the in-process round tripper behind handler: internal is a routing facility for the platform's own admin API, not a way of reaching an upstream.

internal/cfgmap is used by the seam and by pkg/toolkits/apigateway. The s3, trino, datahub and gateway toolkits still carry their own copies of the same readers; consolidating those is a separate change.

Size

Package	Before	After	Ceiling
pkg/toolkits/apigateway	9,385 lines / 23 files	8,411 lines / 22 files	9,600 / 35
internal/upstreamauth	—	1,408 lines / 4 files	3,418 / 10
internal/cfgmap	—	143 lines / 1 file	3,418 / 10

The ceilings are the ones TestPackageSizeBudget enforces (maxPackageLOC, maxInternalPackageLOC). At 9,385 the toolkit had 215 lines of headroom, so this was a precondition for further work on kind: api generally, not only for a second kind.

TestPackageImportRatchet's allowlist gains the seam's edges (internal/upstreamauth to internal/apigwtls, internal/cfgmap, internal/membudget, pkg/authevents, pkg/connoauth) and the toolkit's two new ones; pkg/toolkits/apigateway -> internal/apigwtls is gone, since the toolkit no longer reaches TLS directly. go list -deps ./internal/upstreamauth names no toolkit package.

Verification

Acceptance through the real surface. test/acceptance/issue_1647_test.go, 23 subtests, run as a real MCP client against the platform on the local stack (make dev). The upstream is the api-test fixture the deployment runs, not a stand-in: its /v1/echo returns the method, path, query and headers of the request as it arrived, so each credential is observed where it lands rather than where it is built.

• none attaches no Authorization and no X-API-Key.
• api_key in a custom header carries the configured name and value; in the query, the configured parameter and value, and no header.
• api_key in the default header authenticates the caller — the fixture's whoami reports apikey / dev-fixture.
• basic attaches exactly one Authorization header.
• bearer attaches the token: the fixture answers 401 invalid credential where the same path with no credential answers 200.
• An operator's pinned header reaches the upstream; a model header the connection does not claim is forwarded.
• The model may not set Authorization, the header its connection's auth mode owns, or a pinned header.
• Six configuration refusals are asserted against the literal text an operator reads, including the two now produced inside packages no operator configures (apigateway: mtls_client_cert_pem and mtls_client_key_pem are required, apigateway: call_timeout must be positive).

Wire forms: api_invoke_endpoint.body is untyped and admits an object and a string of JSON. Both are sent as literal tools/call params and the upstream receives the same document. Transcript, including the Wire forms: line and the binary's build time against the last source edit, is at build/1647/acceptance.md.

Gates. make verify green. Patch coverage 99.5% (594 of 597 changed executable lines); internal/upstreamauth 99.3%, internal/cfgmap 100%, internal/apigwtls up from 87.3% to 93.4%. make lint reports 0 issues. gosec clean on the changed packages. No dead code in the new packages. The API gateway's own suites pass with no assertion changed, with one exception: the test that proved SetConnOAuthStore re-threads an already-materialized authenticator did so by reading a private field that is no longer reachable from the toolkit, and now asserts the same contract by behavior — after the store is threaded, the connection's authenticator reports ErrNeedsReauth rather than "token store not wired". A second test covers the other ordering, a connection registered after the wiring.

Documentation. docs/library/stability.md and docs/llms-full.txt name the seam and record that the toolkit's API, configuration keys and error messages are unchanged; CLAUDE.md's internal/ inventory gains both packages.

Closes #1647

[codecov]

Codecov Report

❌ Patch coverage is 99.39880% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.72%. Comparing base (08e8a35) to head (96ed962).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/apigwtls/apigwtls.go	88.88%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1653      +/-   ##
==========================================
+ Coverage   91.59%   91.72%   +0.12%     
==========================================
  Files         777      781       +4     
  Lines       77877    78003     +126     
==========================================
+ Hits        71332    71545     +213     
+ Misses       4275     4201      -74     
+ Partials     2270     2257      -13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.