Skip to content

feat(llm): resolve model endpoints from a model catalog - #297

Merged
enkerewpo merged 23 commits into
syswonder:devfrom
agicy:llm-catalog
Nov 15, 2026
Merged

enkerewpo merged 23 commits into
syswonder:devfrom
agicy:llm-catalog

Conversation

@agicy

@agicy agicy commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Robonix's model consumers each duplicate endpoint, model, key and feature assumptions. There is no single source of truth for what a model supports, so changing a model means editing every consumer and feature mismatches surface only at runtime.

User-visible impact

A deployment can set ROBONIX_LLM_CATALOG to one YAML/JSON catalog. Each consumer resolves its endpoint from feature and budget requirements at startup, then connects to the provider directly. Without the variable every consumer keeps its existing VLM_*/manifest path; rollout is per service and rollback is unsetting the variable.

Implementation

  • Python: robonix_api.llm in pylib/robonix-api; Rust: system/llm-catalog.
  • ModelCatalog: validated YAML/JSON ModelSpec list (endpoint, model, api_key_env, features, context, p95 latency, prices, protocol); api_key_env is a variable name, never a key.
  • Resolver: hard feature filter → context/latency/cost filter → rank (latency, price, id) → best-first ModelSpec[]. json_schema falls back to json_object only when no strict candidate fits and marks targets downgraded=true.
  • Same operations in both languages: ModelCatalog.load, ModelCatalog.model, ModelCatalog.resolve, Requirements.default, resolve_from_env, api_key_for.
  • Pilot: catalog selects the VLM endpoint; default features vision,tools,stream; ROBONIX_LLM_* overrides budgets.
  • Opt-in consumers: Scene (vision), Memory (json_object / vision), Memsearch (text), VLM Verifier (vision, covering the new before/after and multi-camera paths through the shared VlmConfig.parse), and CI scripts. All keep the legacy path when the catalog is unset.
  • No service, proxy, port or request hop; consumers connect directly.

Validation

  • PYTHONPATH=pylib/robonix-api python3 -m unittest discover -s pylib/robonix-api/tests -p 'test_llm_*.py' → 18 passed
  • Scene focused CI (pytest/numpy/scipy/httpx/Pillow only): 63 passed, 5 skipped; second group 29 passed
  • Memory: 14 + 8 passed; Memsearch: 2 passed; Scene catalog: 1 passed
  • VLM Verifier: 33 passed after rbnx codegen -p services/verifiers/vlm_verifier --mcp
  • Rust: cargo test -p robonix-llm-catalog 3 passed; cargo test -p robonix-pilot 111 passed; cargo test -p robonix-cli 36 passed
  • Catalog-only boot (VLM_* unset, only ROBONIX_LLM_CATALOG + provider key): rbnx preflight skipped, Pilot resolved vlm=catalog
  • cargo fmt --check, cargo clippy ... -D warnings, cargo metadata --locked, uv lock --check: clean
  • Webots office smoke: all components ACTIVE; quick ask OK; exploration produced 41 objects / 75 relations; Pilot 2 requests, 22,281 input / 1,237 output, 81% cache
  • DeepSeek two-image smoke: red, blue; simulation shut down with no robonix containers and GPUs back to 1 MiB

Configuration and migration

  • ROBONIX_LLM_CATALOG — absolute catalog path; unset means legacy behavior.
  • ROBONIX_LLM_FEATURES — Pilot's required model features; defaults to vision,tools,stream.
  • Keys are read by consumers from each target's api_key_env; Core never stores keys.
  • No new capability contracts, generated code, ports, processes or Python dependencies; uv.lock is unchanged.
  • Rollout is per service; rollback is unsetting the variable.

Related issues

None.

Screenshots / terminal output

No CLI, TUI or web UI changes.

@github-actions github-actions Bot added type:feature New feature (feat:) comp:pilot system/pilot comp:scene system/scene comp:memory services/memsearch comp:docs docs/ and READMEs and removed type:feature New feature (feat:) labels Oct 25, 2026
@github-actions github-actions Bot added the comp:api pylib/robonix-api label Oct 27, 2026
@agicy
agicy marked this pull request as ready for review October 31, 2026 06:51


@dataclass(frozen=True)
class Catalog:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe ModelCatalog? Since we use catalog for capabilities before

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. I'll rename it to ModelCatalog in both Python and Rust to avoid confusion with the capability catalog.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed.

"base_url": "http://127.0.0.1:8000/v1",
"model": "Qwen2.5-VL-7B-Instruct-AWQ",
"api_key_env": "VLLM_API_KEY",
"capabilities": {

@enkerewpo enkerewpo Nov 1, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please don’t use capabilities here and also in other code structures, since we used it as our skill/service/primitive interface. Maybe use feature

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed.

@github-actions github-actions Bot added the comp:rbnx tools/rbnx label Nov 1, 2026
@agicy agicy changed the title feat(llm): select model endpoints from a capability catalog feat(llm): resolve model endpoints from a model catalog Nov 1, 2026

@enkerewpo enkerewpo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally on 5070130 merged onto current dev (a7ec860, no conflicts).

Tests pass: Python test_llm_*, memory 171, memsearch 10, verifier 33, scene 318 passed, cargo test -p robonix-llm-catalog -p robonix-pilot -p robonix-cli 189 (including the new pilot_preflight test), clippy -D warnings and cargo fmt --check clean. The 4 scene failures in test_config_spec.py also fail on plain dev.

Thanks for the rename and the strict boolean check, both verified. Some lines this PR adds still call the model catalog a "capability catalog" or the model features "Capabilities", which reads like Robonix capabilities (Atlas, robonix_api.capability): the three log.warning calls in memory and memsearch, Cargo.toml L19, pylib/robonix-api/README.md L92, system/scene/scripts/start.sh L132, the doc comments on Requirements in resolve.py, lib.rs, tests/contract.rs and config.rs, and the table header in the llm README.

The rbnx change lets Pilot start without a vlm block, but the catalog is still an environment variable that the manifest and rbnx know nothing about:

  • Pilot lets the catalog override explicit --vlm-* flags, which come from the manifest under rbnx deploy (comment on config.rs).
  • With the variable set, rbnx skips the preflight entirely, so a bad catalog shows [ OK ] and then Pilot exits (comment on deploy.rs).
  • The verifier config.spec still requires a vlm block that is ignored.
  • The catalog path and keys only reach consumers whose launcher forwards them; Scene forwards six hard-coded key names.

Proposal, to agree on before implementing:

  1. Keep the model list out of the manifest. Which models exist, their keys, prices and latency belong to the machine or user, not to the robot app, and the same manifest should run on machines with different models. rbnx loads a user-level file (e.g. ~/.robonix/models.yaml); the manifest may override the path with an optional model_catalog: field. Keys stay as api_key_env names.
  2. rbnx resolves and validates the catalog at deploy time and passes the path plus exactly the api_key_env variables the catalog uses to every consumer, containers included.
  3. Explicit config wins and the catalog fills gaps:
    • system.pilot.vlm with upstream + model: used as is, catalog not consulted (today's behaviour);
    • vlm.model set to a catalog id: endpoint and key come from that entry;
    • no vlm: resolved from the catalog by Pilot's requirements.
      The rbnx preflight accepts either a complete vlm block or a catalog that resolves, and the boot line prints the target actually used and where it came from. The verifier's vlm block becomes optional under the same rule.

The other inline comments are problems I could still reproduce on 5070130.

raise RuntimeError(
f"no model in {path} satisfies {sorted(requirements.features)}"
)
for target in targets:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A local endpoint without a key always loses to a keyed cloud endpoint here. With local/vllm (300 ms, VLLM_API_KEY unset) and deepseek/flash (2000 ms, DEEPSEEK_API_KEY set):

resolved: deepseek/flash

So the latency ranking is skipped and a paid model is used for every call, although local vLLM/Ollama usually runs without auth. api_key_env is also required in catalog.py and Pilot bails on an empty key, so the only workaround is exporting a dummy key. Could api_key_env be optional, with a missing field meaning "no key needed"?

raise ValueError(
f"{target.api_key_env} is required for catalog target {target.id!r}"
)
return cls(target.base_url.rstrip("/"), api_key, target.model)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This returns before the URL checks below, so the catalog path accepts URLs the manifest path rejects:

manifest path rejects: ValueError
catalog path accepts: https://user:pw@host/v1?key=secret

Please run the catalog base_url through the same validation.

target = resolve_from_env(
Requirements(features=("json_object",), max_output_tokens=512)
)
except RuntimeError as exc:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only RuntimeError (no matching model) is caught. A wrong path, a YAML error or a validation error raises FileNotFoundError / ValueError and takes the service down:

ROBONIX_LLM_CATALOG=/nonexistent/catalog.yaml -> raises FileNotFoundError

The two failure modes also behave in opposite ways: no match silently falls back to the legacy VLM_* model, while a broken file crashes. When the operator has set ROBONIX_LLM_CATALOG, I'd expect a catalog problem to be an explicit startup error in every consumer, as Pilot and the verifier already do, rather than a silent switch to a different model. The same applies to observe.py, memsearch service.py and the two Scene clients.

Comment thread system/scene/scripts/start.sh Outdated
-e VLM_REASONING_EFFORT="${VLM_REASONING_EFFORT:-}" \
-e SCENE_VLM_MAX_TOKENS="${SCENE_VLM_MAX_TOKENS:-}" \
-e ROBONIX_LLM_CATALOG="${ROBONIX_LLM_CATALOG:-}" \
-e DEEPSEEK_API_KEY="${DEEPSEEK_API_KEY:-}" \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

api_key_env can name any variable, but only these six are forwarded, so a catalog using e.g. MOONSHOT_API_KEY resolves inside the container with an empty key and Scene goes inactive. It also passes every provider key into the container whether or not the catalog uses it. This should be exactly the api_key_env names the catalog uses, ideally passed in by rbnx (point 2 in the review body).

Small one at L315: -e PYTHONPATH=/robonix-api is not needed, since entrypoint.sh already prepends /robonix-api. It also replaces the image's own ENV PYTHONPATH (/opt/dualmap in the DualMap image). Scene adds /opt/dualmap to sys.path itself, so nothing breaks today.

Comment thread system/pilot/src/config.rs Outdated
}

let requirements = env_llm_requirements();
if let Some(target) = resolve_from_env(&requirements)? {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The catalog still overrides explicit --vlm-upstream/--vlm-model/--vlm-api-key without a log line. Under rbnx deploy those flags come from the manifest's system.pilot.vlm, so a deployment that has both silently ignores the manifest. Explicit flags should win, with the catalog filling only what is missing (point 3 in the review body).

required = set(requirements.features)
downgraded = False

if "json_schema" in required and requirements.allow_downgrade:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None of the in-tree consumers asks for json_schema or reads downgraded (Pilot handles schema refusal at request time in vlm.rs), and only Pilot exposes the latency/cost budgets, through env vars. Could the downgrade path and the budget filters be left out until a consumer needs them? That would shrink both implementations and the contract cases quite a bit.

Comment thread tools/rbnx/src/cmd/deploy.rs Outdated
args: &[String],
llm_catalog: bool,
) -> std::result::Result<(), String> {
if name != "pilot" || llm_catalog {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skipping the check when the variable is set brings back the failure this preflight was written for: rbnx prints [ OK ], then Pilot exits during config load. With the PR's Pilot binary and ROBONIX_LLM_CATALOG set:

local model, no key  -> Error: read VLLM_API_KEY for catalog target local/vllm
wrong path           -> Error: read LLM catalog /nonexistent.yaml
no matching model    -> Error: no model in .../c.yaml satisfies vision,json_schema

rbnx already builds in the workspace with robonix-llm-catalog available, so it could call resolve_from_env with Pilot's requirements here: fail fast on these errors, and print the resolved target id and model instead of vlm=catalog on L2181.

@github-actions github-actions Bot added the comp:ci .github/ workflows label Nov 1, 2026

@enkerewpo enkerewpo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reworking this along the proposal. Re-reviewed 2a13428 locally; I checked the earlier points in code: a keyless local endpoint now stays eligible, explicit Pilot vlm config wins and a bare vlm.model selects a catalog id, the verifier validates catalog URLs, memory checks the catalog at startup, Scene forwards only the key names the catalog uses, and rbnx has the model_catalog field with ~/.robonix/models.yaml.

Tests on the PR head: cargo fmt --check and cargo clippy --workspace --tests -D warnings clean, cargo test --workspace --all-targets passes, rbnx 39 + 46 passed, memory 177 and memsearch 13 passed. robonix-api (1 failure), the verifier (10, missing verifier_mcp codegen) and Scene (6 failed, 5 errors) fail identically on the merge base a7ec860, so none comes from this PR.

Two things before merging:

  1. The branch is 36 commits behind dev and conflicts in .github/workflows/ci.yml, system/pilot/src/config.rs and system/scene/README.md. Please rebase. #320 builds on the same files, so this one should land first and #320 rebase on top.
  2. Memory does not follow the precedence the rbnx README states; see the inline comment.


def _llm_config() -> dict:
"""Resolve LLM endpoint from env vars with standard fallbacks."""
target = resolve_from_env(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a catalog set, this resolves from the catalog before looking at the explicit values, so memory ignores them. With ROBONIX_LLM_CATALOG naming one entry and VLM_BASE_URL/VLM_API_KEY/VLM_MODEL (or MEMGRAPH_LLM_*) set, _llm_config() returns the catalog entry in both cases:

explicit: VLM_BASE_URL=https://manifest.example.com/v1 -> memory uses http://catalog.example.com/v1 catalog-model
explicit: MEMGRAPH_LLM_BASE_URL=https://explicit.example.com/v1 -> memory uses http://catalog.example.com/v1 catalog-model

Pilot and the verifier let explicit config win, and tools/rbnx/README.md says explicit values "take precedence even with a catalog". Please check the explicit MEMGRAPH_LLM_* / VLM_* values first and fall back to the catalog only when they are absent.

@agicy

agicy commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

@enkerewpo

Rebased onto current dev (39ab370) and addressed both points from the last review.

  • Rebase conflicts resolved (ci.yml, pilot/config.rs, scene/README.md); Pilot
    compiles again after feat(pilot): forward optional model reasoning effort #321 (added reasoning_effort to the test config).
  • Memory now follows the documented precedence: an explicit
    MEMGRAPH_LLM_BASE_URL / VLM_BASE_URL wins and the catalog only fills what is
    unset. The same rule applies to memsearch and both Scene clients; a key or
    model alone no longer disables the catalog. The model endpoints are also in
    runtime_env.txt so the setting survives the rbnx launcher, with keys
    forwarded only alongside their own endpoint.

Validated: cargo fmt/clippy clean, cargo test --workspace (only the 3 scribe
tests that also fail on dev), and the Python llm/memory/memsearch/scene/verifier
suites pass.

@enkerewpo enkerewpo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Retested 47b17ed locally: fmt/clippy clean, 494 Rust tests and the rbnx tests pass, the memory (182), memsearch (16), Scene catalog (12) and catalog contract (29) Python tests pass, and an explicit VLM_BASE_URL/MEMGRAPH_LLM_BASE_URL now wins over the catalog in memory. Thanks for the quick turnaround.

The secret scan flags 6 test fixture lines ("api_key_env": "...", "latency_p95_ms": N) as generic API keys. They are false positives; I will mark them on dev right after merging.

@enkerewpo
enkerewpo merged commit 8a088c5 into syswonder:dev Nov 15, 2026
15 of 16 checks passed
enkerewpo added a commit that referenced this pull request Nov 15, 2026
gitleaks reads "api_key_env": "...", "latency_p95_ms": N in the model
catalog tests (#297) as a generic API key. Extend the default rules with
a line allowlist for latency_p95_ms values.
@agicy
agicy deleted the llm-catalog branch November 15, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:api pylib/robonix-api comp:ci .github/ workflows comp:docs docs/ and READMEs comp:memory services/memsearch comp:pilot system/pilot comp:rbnx tools/rbnx comp:scene system/scene type:feature New feature (feat:)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants