Repository navigation
feat(llm): resolve model endpoints from a model catalog - #297
Conversation
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class Catalog: |
There was a problem hiding this comment.
Maybe ModelCatalog? Since we use catalog for capabilities before
There was a problem hiding this comment.
Got it. I'll rename it to ModelCatalog in both Python and Rust to avoid confusion with the capability catalog.
| "base_url": "http://127.0.0.1:8000/v1", | ||
| "model": "Qwen2.5-VL-7B-Instruct-AWQ", | ||
| "api_key_env": "VLLM_API_KEY", | ||
| "capabilities": { |
There was a problem hiding this comment.
please don’t use capabilities here and also in other code structures, since we used it as our skill/service/primitive interface. Maybe use feature
enkerewpo
left a comment
There was a problem hiding this comment.
Tested locally on 5070130 merged onto current dev (a7ec860, no conflicts).
Tests pass: Python test_llm_*, memory 171, memsearch 10, verifier 33, scene 318 passed, cargo test -p robonix-llm-catalog -p robonix-pilot -p robonix-cli 189 (including the new pilot_preflight test), clippy -D warnings and cargo fmt --check clean. The 4 scene failures in test_config_spec.py also fail on plain dev.
Thanks for the rename and the strict boolean check, both verified. Some lines this PR adds still call the model catalog a "capability catalog" or the model features "Capabilities", which reads like Robonix capabilities (Atlas, robonix_api.capability): the three log.warning calls in memory and memsearch, Cargo.toml L19, pylib/robonix-api/README.md L92, system/scene/scripts/start.sh L132, the doc comments on Requirements in resolve.py, lib.rs, tests/contract.rs and config.rs, and the table header in the llm README.
The rbnx change lets Pilot start without a vlm block, but the catalog is still an environment variable that the manifest and rbnx know nothing about:
- Pilot lets the catalog override explicit
--vlm-*flags, which come from the manifest underrbnx deploy(comment onconfig.rs). - With the variable set, rbnx skips the preflight entirely, so a bad catalog shows
[ OK ]and then Pilot exits (comment ondeploy.rs). - The verifier
config.specstill requires avlmblock that is ignored. - The catalog path and keys only reach consumers whose launcher forwards them; Scene forwards six hard-coded key names.
Proposal, to agree on before implementing:
- Keep the model list out of the manifest. Which models exist, their keys, prices and latency belong to the machine or user, not to the robot app, and the same manifest should run on machines with different models. rbnx loads a user-level file (e.g.
~/.robonix/models.yaml); the manifest may override the path with an optionalmodel_catalog:field. Keys stay asapi_key_envnames. - rbnx resolves and validates the catalog at deploy time and passes the path plus exactly the
api_key_envvariables the catalog uses to every consumer, containers included. - Explicit config wins and the catalog fills gaps:
system.pilot.vlmwithupstream+model: used as is, catalog not consulted (today's behaviour);vlm.modelset to a catalog id: endpoint and key come from that entry;- no
vlm: resolved from the catalog by Pilot's requirements.
The rbnx preflight accepts either a completevlmblock or a catalog that resolves, and the boot line prints the target actually used and where it came from. The verifier'svlmblock becomes optional under the same rule.
The other inline comments are problems I could still reproduce on 5070130.
| raise RuntimeError( | ||
| f"no model in {path} satisfies {sorted(requirements.features)}" | ||
| ) | ||
| for target in targets: |
There was a problem hiding this comment.
A local endpoint without a key always loses to a keyed cloud endpoint here. With local/vllm (300 ms, VLLM_API_KEY unset) and deepseek/flash (2000 ms, DEEPSEEK_API_KEY set):
resolved: deepseek/flash
So the latency ranking is skipped and a paid model is used for every call, although local vLLM/Ollama usually runs without auth. api_key_env is also required in catalog.py and Pilot bails on an empty key, so the only workaround is exporting a dummy key. Could api_key_env be optional, with a missing field meaning "no key needed"?
| raise ValueError( | ||
| f"{target.api_key_env} is required for catalog target {target.id!r}" | ||
| ) | ||
| return cls(target.base_url.rstrip("/"), api_key, target.model) |
There was a problem hiding this comment.
This returns before the URL checks below, so the catalog path accepts URLs the manifest path rejects:
manifest path rejects: ValueError
catalog path accepts: https://user:pw@host/v1?key=secret
Please run the catalog base_url through the same validation.
| target = resolve_from_env( | ||
| Requirements(features=("json_object",), max_output_tokens=512) | ||
| ) | ||
| except RuntimeError as exc: |
There was a problem hiding this comment.
Only RuntimeError (no matching model) is caught. A wrong path, a YAML error or a validation error raises FileNotFoundError / ValueError and takes the service down:
ROBONIX_LLM_CATALOG=/nonexistent/catalog.yaml -> raises FileNotFoundError
The two failure modes also behave in opposite ways: no match silently falls back to the legacy VLM_* model, while a broken file crashes. When the operator has set ROBONIX_LLM_CATALOG, I'd expect a catalog problem to be an explicit startup error in every consumer, as Pilot and the verifier already do, rather than a silent switch to a different model. The same applies to observe.py, memsearch service.py and the two Scene clients.
| -e VLM_REASONING_EFFORT="${VLM_REASONING_EFFORT:-}" \ | ||
| -e SCENE_VLM_MAX_TOKENS="${SCENE_VLM_MAX_TOKENS:-}" \ | ||
| -e ROBONIX_LLM_CATALOG="${ROBONIX_LLM_CATALOG:-}" \ | ||
| -e DEEPSEEK_API_KEY="${DEEPSEEK_API_KEY:-}" \ |
There was a problem hiding this comment.
api_key_env can name any variable, but only these six are forwarded, so a catalog using e.g. MOONSHOT_API_KEY resolves inside the container with an empty key and Scene goes inactive. It also passes every provider key into the container whether or not the catalog uses it. This should be exactly the api_key_env names the catalog uses, ideally passed in by rbnx (point 2 in the review body).
Small one at L315: -e PYTHONPATH=/robonix-api is not needed, since entrypoint.sh already prepends /robonix-api. It also replaces the image's own ENV PYTHONPATH (/opt/dualmap in the DualMap image). Scene adds /opt/dualmap to sys.path itself, so nothing breaks today.
| } | ||
|
|
||
| let requirements = env_llm_requirements(); | ||
| if let Some(target) = resolve_from_env(&requirements)? { |
There was a problem hiding this comment.
The catalog still overrides explicit --vlm-upstream/--vlm-model/--vlm-api-key without a log line. Under rbnx deploy those flags come from the manifest's system.pilot.vlm, so a deployment that has both silently ignores the manifest. Explicit flags should win, with the catalog filling only what is missing (point 3 in the review body).
| required = set(requirements.features) | ||
| downgraded = False | ||
|
|
||
| if "json_schema" in required and requirements.allow_downgrade: |
There was a problem hiding this comment.
None of the in-tree consumers asks for json_schema or reads downgraded (Pilot handles schema refusal at request time in vlm.rs), and only Pilot exposes the latency/cost budgets, through env vars. Could the downgrade path and the budget filters be left out until a consumer needs them? That would shrink both implementations and the contract cases quite a bit.
| args: &[String], | ||
| llm_catalog: bool, | ||
| ) -> std::result::Result<(), String> { | ||
| if name != "pilot" || llm_catalog { |
There was a problem hiding this comment.
Skipping the check when the variable is set brings back the failure this preflight was written for: rbnx prints [ OK ], then Pilot exits during config load. With the PR's Pilot binary and ROBONIX_LLM_CATALOG set:
local model, no key -> Error: read VLLM_API_KEY for catalog target local/vllm
wrong path -> Error: read LLM catalog /nonexistent.yaml
no matching model -> Error: no model in .../c.yaml satisfies vision,json_schema
rbnx already builds in the workspace with robonix-llm-catalog available, so it could call resolve_from_env with Pilot's requirements here: fail fast on these errors, and print the resolved target id and model instead of vlm=catalog on L2181.
enkerewpo
left a comment
There was a problem hiding this comment.
Thanks for reworking this along the proposal. Re-reviewed 2a13428 locally; I checked the earlier points in code: a keyless local endpoint now stays eligible, explicit Pilot vlm config wins and a bare vlm.model selects a catalog id, the verifier validates catalog URLs, memory checks the catalog at startup, Scene forwards only the key names the catalog uses, and rbnx has the model_catalog field with ~/.robonix/models.yaml.
Tests on the PR head: cargo fmt --check and cargo clippy --workspace --tests -D warnings clean, cargo test --workspace --all-targets passes, rbnx 39 + 46 passed, memory 177 and memsearch 13 passed. robonix-api (1 failure), the verifier (10, missing verifier_mcp codegen) and Scene (6 failed, 5 errors) fail identically on the merge base a7ec860, so none comes from this PR.
Two things before merging:
- The branch is 36 commits behind dev and conflicts in
.github/workflows/ci.yml,system/pilot/src/config.rsandsystem/scene/README.md. Please rebase. #320 builds on the same files, so this one should land first and #320 rebase on top. - Memory does not follow the precedence the rbnx README states; see the inline comment.
|
|
||
| def _llm_config() -> dict: | ||
| """Resolve LLM endpoint from env vars with standard fallbacks.""" | ||
| target = resolve_from_env( |
There was a problem hiding this comment.
With a catalog set, this resolves from the catalog before looking at the explicit values, so memory ignores them. With ROBONIX_LLM_CATALOG naming one entry and VLM_BASE_URL/VLM_API_KEY/VLM_MODEL (or MEMGRAPH_LLM_*) set, _llm_config() returns the catalog entry in both cases:
explicit: VLM_BASE_URL=https://manifest.example.com/v1 -> memory uses http://catalog.example.com/v1 catalog-model
explicit: MEMGRAPH_LLM_BASE_URL=https://explicit.example.com/v1 -> memory uses http://catalog.example.com/v1 catalog-model
Pilot and the verifier let explicit config win, and tools/rbnx/README.md says explicit values "take precedence even with a catalog". Please check the explicit MEMGRAPH_LLM_* / VLM_* values first and fall back to the catalog only when they are absent.
|
Rebased onto current dev (39ab370) and addressed both points from the last review.
Validated: cargo fmt/clippy clean, cargo test --workspace (only the 3 scribe |
enkerewpo
left a comment
There was a problem hiding this comment.
LGTM. Retested 47b17ed locally: fmt/clippy clean, 494 Rust tests and the rbnx tests pass, the memory (182), memsearch (16), Scene catalog (12) and catalog contract (29) Python tests pass, and an explicit VLM_BASE_URL/MEMGRAPH_LLM_BASE_URL now wins over the catalog in memory. Thanks for the quick turnaround.
The secret scan flags 6 test fixture lines ("api_key_env": "...", "latency_p95_ms": N) as generic API keys. They are false positives; I will mark them on dev right after merging.
gitleaks reads "api_key_env": "...", "latency_p95_ms": N in the model catalog tests (#297) as a generic API key. Extend the default rules with a line allowlist for latency_p95_ms values.
Problem
Robonix's model consumers each duplicate endpoint, model, key and feature assumptions. There is no single source of truth for what a model supports, so changing a model means editing every consumer and feature mismatches surface only at runtime.
User-visible impact
A deployment can set
ROBONIX_LLM_CATALOGto one YAML/JSON catalog. Each consumer resolves its endpoint from feature and budget requirements at startup, then connects to the provider directly. Without the variable every consumer keeps its existingVLM_*/manifest path; rollout is per service and rollback is unsetting the variable.Implementation
robonix_api.llminpylib/robonix-api; Rust:system/llm-catalog.ModelCatalog: validated YAML/JSONModelSpeclist (endpoint, model,api_key_env, features, context, p95 latency, prices, protocol);api_key_envis a variable name, never a key.(latency, price, id)→ best-firstModelSpec[].json_schemafalls back tojson_objectonly when no strict candidate fits and marks targetsdowngraded=true.ModelCatalog.load,ModelCatalog.model,ModelCatalog.resolve,Requirements.default,resolve_from_env,api_key_for.vision,tools,stream;ROBONIX_LLM_*overrides budgets.vision), Memory (json_object/vision), Memsearch (text), VLM Verifier (vision, covering the new before/after and multi-camera paths through the sharedVlmConfig.parse), and CI scripts. All keep the legacy path when the catalog is unset.Validation
PYTHONPATH=pylib/robonix-api python3 -m unittest discover -s pylib/robonix-api/tests -p 'test_llm_*.py'→ 18 passedrbnx codegen -p services/verifiers/vlm_verifier --mcpcargo test -p robonix-llm-catalog3 passed;cargo test -p robonix-pilot111 passed;cargo test -p robonix-cli36 passedVLM_*unset, onlyROBONIX_LLM_CATALOG+ provider key): rbnx preflight skipped, Pilot resolvedvlm=catalogcargo fmt --check,cargo clippy ... -D warnings,cargo metadata --locked,uv lock --check: cleanOK; exploration produced 41 objects / 75 relations; Pilot 2 requests, 22,281 input / 1,237 output, 81% cachered, blue; simulation shut down with no robonix containers and GPUs back to 1 MiBConfiguration and migration
ROBONIX_LLM_CATALOG— absolute catalog path; unset means legacy behavior.ROBONIX_LLM_FEATURES— Pilot's required model features; defaults tovision,tools,stream.api_key_env; Core never stores keys.uv.lockis unchanged.Related issues
None.
Screenshots / terminal output
No CLI, TUI or web UI changes.