Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,23 @@ jobs:
system/scene/tests/test_goal_planner.py
system/scene/tests/test_malformed_grid_inputs.py

robonix-api-tests:
name: robonix-api unit tests
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
# Installs the package the way a consumer gets it, so an import the
# tests rely on cannot pass by accident through PYTHONPATH alone.
- name: Install robonix-api
run: python3 -m pip install ./pylib/robonix-api
- name: Install test dependencies
run: python3 -m pip install pytest
- name: Test robonix-api
run: python3 -m pytest -q pylib/robonix-api/tests

chronos-python:
name: Chronos Python bindings
runs-on: ubuntu-22.04
Expand Down
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,9 @@ Third-party skills are not vendored into the repository. These are recommended f
Robonix is an **embodied AI operating system**: a monorepo with Rust system
components, Python packages, capability contracts, IDL libraries, examples, and
documentation. The Rust workspace lives at the repository root in `Cargo.toml`;
current Rust members are `system/{atlas,executor,liaison,pilot,scribe,soma,vitals}`
and `tools/{rbnx,codegen}`. The twelve system component directories live under
current Rust members are
`system/{atlas,chronos,executor,liaison,llm-catalog,llm-metrics,pilot,scribe,soma,vitals}`
and `tools/{rbnx,codegen}`. The system component directories live under
`system/`; several are documentation/planning components today, while
`system/scene` is a Python package managed by uv. Python workspace packages are
listed in the root `pyproject.toml`: shared libraries in `pylib/*`, reference
Expand Down
12 changes: 12 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 8 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
[workspace]
# The 12 Robonix system components live one-per-directory under `system/`.
# Seven currently have a Rust implementation:
# Robonix system components live one-per-directory under `system/`.
# These currently have a Rust implementation:
# - atlas (capability discovery)
# - chronos (device clock library)
# - executor (runs RTDL programs, dispatching each call to its provider)
# - liaison (human-machine interaction)
# - llm-catalog (model catalog and resolver)
# - llm-metrics (per-request model usage records)
# - pilot (turns a task into an RTDL program and validates it)
# - scribe (structured logging library)
# - soma (body description; launches primitive and skill packages)
Expand All @@ -19,6 +22,8 @@ members = [
"system/liaison",
# Shared model catalog for LLM consumers (Pilot and future users).
"system/llm-catalog",
# Per-request model usage records shared by LLM consumers.
"system/llm-metrics",
"system/pilot",
"system/scribe",
"system/soma",
Expand Down Expand Up @@ -86,3 +91,4 @@ crossterm = "0.29.0"
robonix-chronos-clock = { path = "system/chronos/clock" }
robonix-codegen = { path = "tools/codegen" }
robonix-scribe = { path = "system/scribe" }
robonix-llm-metrics = { path = "system/llm-metrics" }
5 changes: 3 additions & 2 deletions pylib/robonix-api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,9 @@ The Atlas wire protocol (atlas_pb2 / atlas_pb2_grpc) is pre-generated and bundle
`robonix_api.llm` is the shared model catalog and resolver used by
Scene, Memory, Memsearch, the VLM Verifier, CI scripts and other consumers. It
answers "which model endpoint should this request use?" from a declarative
catalog and leaves the provider call to the consumer's own client. It does not
proxy traffic, hold keys or track health. See
catalog and leaves the provider call to the consumer's own client. The
`metrics` helper appends one JSON line per model request for usage accounting.
It does not proxy traffic, hold keys or track health. See
[`robonix_api/llm/README.md`](robonix_api/llm/README.md) for the catalog format,
the Python API and the deployment variables.

Expand Down
2 changes: 1 addition & 1 deletion pylib/robonix-api/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,4 @@ include = ["robonix_api*"]

[tool.setuptools.package-data]
robonix_api = ["_generated/*.py"]
"robonix_api.llm" = ["runtime_env.txt"]
"robonix_api.llm" = ["runtime_env.txt", "contract/*.json"]
81 changes: 77 additions & 4 deletions pylib/robonix-api/robonix_api/llm/README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
# robonix_api.llm

Model catalog and model resolver for Robonix consumers, shipped as the
`robonix_api.llm` submodule of `robonix-api`.
Model catalog, model resolver and per-request usage records for Robonix
consumers, shipped as the `robonix_api.llm` submodule of `robonix-api`.

Consumers resolve an endpoint from their requirements and connect directly;
the catalog adds no proxy or service.
the catalog adds no proxy or service. `metrics` records one JSON line per
model call so usage can be counted across consumers.

## ModelCatalog

Expand Down Expand Up @@ -114,7 +115,6 @@ Containerized Scene mounts the catalog at the same absolute path and forwards on
supplies that list; standalone launches derive it with the host's Python and
`robonix_api.llm` (PyYAML is needed for YAML catalogs). Without a catalog,
Scene forwards only the legacy `VLM_API_KEY` and `OPENAI_API_KEY` variables.
The catalog is read once per process; restart a consumer after editing it.

The requirements used by the in-tree consumers:

Expand All @@ -136,12 +136,85 @@ The Rust implementation lives in `system/llm-catalog` and reads the same
catalog with the same rules. The shared cases in
`robonix_api/llm/contract/resolve_cases.json` are run by both test suites.

## Model usage records

Shared helpers for Robonix components that call a model. `metrics` records
one JSON line per model call so usage can be counted, and later billed,
across every component.

### Usage

```python
from robonix_api.llm import metrics

with metrics.Call("scene", "relations", base_url, model) as call:
response = await client.post(f"{base_url}/chat/completions", json=body)
call.finish(response)
```

`finish(response)` reads `status_code` and the token usage in `response.json()`;
call it for any answer, including an error status, so the attempt is counted.
`retry()` counts one more request sent for the same call. Leaving the block
without `finish` records `timeout`, `cancelled` or `transport_error` from
the exception, and the exception still propagates; so call `finish` before
`raise_for_status()`. A write failure is logged
once and never fails the model call. The first call logs whether records are
on, so an unset variable is visible in the component's log.

### Record

Records go to `$ROBONIX_LLM_METRICS_DIR/<consumer>-<pid>.jsonl`; `consumer`
must match `[a-z0-9_-]+`, because it names the file, and `Call` raises
`ValueError` otherwise. Nothing is written when the variable is unset.
`rbnx boot` sets it to `<deploy>/rbnx-boot/llm-metrics/`, which boot and
`rbnx clean -f` keep unless `--llm-metrics` is given. Each record is appended
with one write, so a reader sees whole lines except possibly the one being
written; `rbnx llm-usage` skips such a line and sums the rest. The files grow by
one line per call and are never rotated. Each record is a few hundred bytes,
so the directory grows with call volume; `rbnx llm-usage` reads every record it
sums, so archive or remove the directory with `rbnx clean -f --llm-metrics`
when it reaches a size you are willing to parse in one run.

| Field | Meaning |
|---|---|
| `v` | Layout version, currently 1 |
| `ts_ms`, `latency_ms` | Wall-clock end time and request duration |
| `boot_id`, `pid` | `RBNX_BOOT_ID` of the deployment and the writing process |
| `consumer`, `operation` | Component and what it used the model for |
| `host` | Endpoint host and port; never the path, query or credentials |
| `model`, `provider_model` | Model requested, and the one the response names |
| `stream` | Whether the reply was streamed |
| `status`, `http_status` | `ok` (2xx with a JSON body), `http_error`, `timeout`, `cancelled` or `transport_error` (unreadable body, or a streamed reply cut off before it finished) |
| `attempts` | Requests sent for this call, counting retries |
| `input_tokens`, `output_tokens`, `cached_input_tokens`, `reasoning_tokens` | Reported counts; `null` when not reported |

Prompts, replies and keys are never recorded. Prices are not recorded either:
cost is token counts times the operator's price list, joined on `host` +
`model` against the totals `rbnx llm-usage --json` prints. Whether a consumer
writes records depends on `ROBONIX_LLM_METRICS_DIR`: a launcher that filters the
environment must pass it through, and the startup log says when it is off.
`RBNX_BOOT_ID` only attributes a record to a boot; without it records still
appear with `boot_id: null`.

The Rust crate `system/llm-metrics` writes the same record. Both parse usage
with the cases in [contract/usage_cases.json](contract/usage_cases.json).

| Consumer | Operations |
|---|---|
| `pilot` | `plan`, `summary` |
| `scene` | `relations`, `image_relations`, `caption`, `perception` |
| `memory` | `search`, `observe`, `answer` |
| `memsearch` | `compact` |
| `vlm_verifier` | `judge` |


## Tests

```bash
PYTHONPATH=pylib/robonix-api python3 -m unittest discover \
-s pylib/robonix-api/tests -p 'test_llm_*.py' -v
cargo test -p robonix-llm-catalog
cargo test -p robonix-llm-metrics
```

## Direct-connect smoke test
Expand Down
5 changes: 3 additions & 2 deletions pylib/robonix-api/robonix_api/llm/__init__.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# SPDX-License-Identifier: MulanPSL-2.0
"""Feature catalog and model resolver for Robonix consumers.
"""Feature catalog, model resolver and usage records for Robonix consumers.

This package answers one question: given feature requirements, which model
The catalog answers one question: given feature requirements, which model
endpoint should this consumer connect to? It does not proxy traffic, track
health or manage keys; the caller resolves once and connects directly.
`metrics` records one line per model request; see README.md.
"""
from .catalog import ModelCatalog, ModelSpec
from .resolve import (
Expand Down
94 changes: 94 additions & 0 deletions pylib/robonix-api/robonix_api/llm/contract/usage_cases.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
{
"cases": [
{
"name": "openai body with cache and reasoning",
"body": {
"model": "gpt-5.5-2026-01-01",
"usage": {
"prompt_tokens": 1200,
"completion_tokens": 80,
"total_tokens": 1280,
"prompt_tokens_details": {
"cached_tokens": 900
},
"completion_tokens_details": {
"reasoning_tokens": 32
}
}
},
"expected": {
"input_tokens": 1200,
"output_tokens": 80,
"cached_input_tokens": 900,
"reasoning_tokens": 32,
"provider_model": "gpt-5.5-2026-01-01"
}
},
{
"name": "deepseek cache hit field",
"body": {
"usage": {
"prompt_tokens": 500,
"completion_tokens": 10,
"prompt_cache_hit_tokens": 400,
"prompt_cache_miss_tokens": 100
}
},
"expected": {
"input_tokens": 500,
"output_tokens": 10,
"cached_input_tokens": 400,
"reasoning_tokens": null,
"provider_model": null
}
},
{
"name": "stream chunk without usage",
"body": {
"model": "m",
"choices": [
{
"delta": {
"content": "hi"
}
}
]
},
"expected": {
"input_tokens": null,
"output_tokens": null,
"cached_input_tokens": null,
"reasoning_tokens": null,
"provider_model": "m"
}
},
{
"name": "invalid counts are unknown",
"body": {
"usage": {
"prompt_tokens": -1,
"completion_tokens": "7",
"prompt_tokens_details": null
}
},
"expected": {
"input_tokens": null,
"output_tokens": null,
"cached_input_tokens": null,
"reasoning_tokens": null,
"provider_model": null
}
},
{
"name": "not an object",
"body": null,
"expected": {
"input_tokens": null,
"output_tokens": null,
"cached_input_tokens": null,
"reasoning_tokens": null,
"provider_model": null
}
}
]
}
Loading
Loading