feat(registry): overridable remote model catalog URLs (env) - #5291
WaldemarFech wants to merge 2 commits into
Conversation
Adds CPA_REMOTE_MODELS_URL and CPA_REMOTE_CODEX_CLIENT_MODELS_URL. When set, a comma-separated list of custom catalog URLs is tried before the built-in mirrors; defaults are kept as fallback so a broken custom source never disables updates. This lets users ship newer model slots without waiting for upstream catalog releases and without patching the source, and makes catalog PRs testable against a real instance.
|
This pull request targeted The base branch has been automatically changed to |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c49c56997
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| func fetchModelsFromRemote(ctx context.Context) (*staticModelsJSON, string) { | ||
| client := &http.Client{Timeout: modelsFetchTimeout} | ||
| for _, url := range modelsURLs { | ||
| for _, url := range resolveRemoteURLs("CPA_REMOTE_MODELS_URL", modelsURLs) { |
There was a problem hiding this comment.
Redact credentials in custom catalog URLs
When an override uses a private or signed URL, such as https://user:pass@host/... or a URL with a token query parameter, the raw environment value flows into the existing debug, warning, and unconditional successful-refresh log messages, exposing those credentials at startup and every refresh. Keep the raw URL only for the HTTP request and use a redacted form in every log/source label; the Codex client catalog loop has the same issue.
AGENTS.md reference: AGENTS.md:L56-L56
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
Adds an environment-variable override mechanism for the registry’s remote model catalog sources, allowing operators to prepend custom catalog URLs (as a comma-separated list) while retaining the built-in mirrors as fallback. This fits into internal/registry/ by extending the existing background updaters without changing the default behavior when env vars are unset.
Changes:
- Introduces a shared
resolveRemoteURLs(...)helper to prepend trimmed env-provided URLs ahead of default mirrors. - Updates the models and Codex client models updaters to use the new resolver with
CPA_REMOTE_MODELS_URLandCPA_REMOTE_CODEX_CLIENT_MODELS_URL. - Adds unit tests for the resolver and documents the env overrides in
config.example.yaml.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| internal/registry/remote_urls.go | Adds shared env-based URL list resolver (custom URLs prepend defaults). |
| internal/registry/remote_urls_test.go | Adds unit tests validating unset/empty env and trimming/ordering behavior. |
| internal/registry/model_updater.go | Switches remote fetch loop to use resolved URLs from CPA_REMOTE_MODELS_URL. |
| internal/registry/codex_client_models_updater.go | Switches remote fetch loop to use resolved URLs from CPA_REMOTE_CODEX_CLIENT_MODELS_URL. |
| config.example.yaml | Documents the new env vars and provides usage/discovery examples. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // release and makes catalog changes testable against a real instance. | ||
|
|
||
| // Recognized variables: | ||
| // CPA_REMOTE_MODELS_URL overrides/models.json (provider routing) |
Source entries that are not http(s) URLs are now read from disk (plain paths and file:// URIs). This removes any hosting requirement from the override mechanism: a user can point CPA_REMOTE_MODELS_URL at C:catalogsmy-models.json next to their binary instead of publishing JSON somewhere online.
|
Extension pushed to this branch (second commit 5f14797): source entries that are not http(s) URLs are now read from disk - plain paths including Windows ones like C:\catalogs\my-models.json, relative paths, and file:// URIs. This removes the hosting requirement from the override mechanism entirely: $env:CPA_REMOTE_MODELS_URL = 'C:\catalogs\my-models.json'Unit tests cover resolver prepending, scheme detection table cases and local reads via t.TempDir; AGENTS.md build verification ran clean. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5f14797edd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if err != nil { | ||
| return nil, err | ||
| } | ||
| p = u.Path |
There was a problem hiding this comment.
Preserve Windows drive names in file URIs
When a Windows user supplies a file:// URI, including the form constructed by TestReadLocalCatalog (file://C:/...), url.Parse stores C: in u.Host and only /... in u.Path; assigning only u.Path therefore discards the drive and makes the catalog read fail before falling back to the official sources. Decode the URI into an OS-native path while preserving drive letters and UNC hosts.
Useful? React with 👍 / 👎.
| func fetchModelsFromRemote(ctx context.Context) (*staticModelsJSON, string) { | ||
| client := &http.Client{Timeout: modelsFetchTimeout} | ||
| for _, url := range modelsURLs { | ||
| for _, url := range resolveRemoteURLs("CPA_REMOTE_MODELS_URL", modelsURLs) { |
There was a problem hiding this comment.
Cap custom model catalog response sizes
When CPA_REMOTE_MODELS_URL points to a compromised or misconfigured server, this newly configurable source is consumed by the existing unbounded io.ReadAll, so a sufficiently large successful response can exhaust process memory before validation or fallback occurs. Apply a byte limit to this catalog path, as the Codex client catalog updater already does.
Useful? React with 👍 / 👎.
|
Verified the local-file source extension end-to-end against a real Pro account: The tested binary was built from HEAD of this branch with a catalog that deliberately did NOT contain the slot embedded, so the positive result can only come from the new local-file read path. |
What
Lets users override where CLIProxyAPI fetches the remote model catalogs from, via two environment variables:
CPA_REMOTE_MODELS_URL— models.json (provider routing + /v1/models)CPA_REMOTE_CODEX_CLIENT_MODELS_URL— codex_client_models.json (client view)When set, a comma-separated list of custom URLs is tried before the built-in mirrors; built-in defaults stay appended as fallback so a broken custom source never disables catalog updates.
Why
Today, serving a model slot that the bundled/remote catalogs do not know yet (e.g.
gpt-daybreak-blue-latest, visible in the official Codex CLI cache but absent from router-for-me/models) requires either waiting for a catalog release or patching the source and rebuilding. With this change it becomes a one-line env var against any fork/raw URL.It also makes catalog PRs trivially testable: point the env at your branch's raw URL, start the server, verify. We used exactly this method to prove models#44 end-to-end through the real updater (log line + completion probe).
Implementation notes
internal/registry/remote_urls.go; both updaters (model_updater.go,codex_client_models_updater.go) swap their loop source toresolveRemoteURLs(...). Defaults untouched otherwise.~/.codex/models_cache.json,/v1/models) with ready-to-paste PowerShell/bash commands..envfiles continue to work since values are read via os.Getenv at fetch time.