Repository navigation
feat(llm): record and aggregate model usage across consumers - #320
Conversation
enkerewpo
left a comment
There was a problem hiding this comment.
Tested 7e407c5 locally. On the PR head: cargo clippy --workspace --tests -D warnings is clean and cargo test --workspace --all-targets passes (420 tests, including chat_stream_records_each_outcome and the robonix-llm-metrics contract tests); robonix-api metrics 10 OK, memory 20 passed, vlm_verifier 22 OK, memsearch lifecycle 7 OK. Scene's suite gives the same result as on dev (408 passed; the failures and collection errors are missing local modules, identical on both).
I also checked _compact against memsearch 0.4.11: same prompt, chunk joining, memory/YYYY-MM-DD.md path, heading and re-indexing, and MemSearch.store is public there. It also fixes the old call, which passed model=/base_url=/api_key= while 0.4.11's MemSearch.compact takes llm_model=/llm_base_url=/llm_api_key=.
One blocker: merged onto current dev (39ab370), robonix-pilot no longer compiles, see the inline comment. Please rebase onto dev and fix it; the rest looks good to me.
| model: "local-test".into(), | ||
| configured_context_window_tokens: None, | ||
| prompt_cache_key: "local-test".into(), | ||
| metrics_dir: Some(metrics_dir.to_path_buf()), |
There was a problem hiding this comment.
After #321 VlmClient has a reasoning_effort field, so merged with current dev this initializer fails: error[E0063]: missing field reasoning_effort in initializer of vlm::VlmClient (cargo clippy --workspace --tests). Please rebase and add reasoning_effort: None, here.
7e407c5 to
ff25ada
Compare
|
Thanks — rebased onto dev and fixed the blocker ( While rebasing I found the launcher env allowlist dropped CI is green; fmt/clippy clean and the Python suites pass. |
enkerewpo
left a comment
There was a problem hiding this comment.
Thanks for the quick rebase and the extra fix. The reasoning_effort blocker from my last review is resolved, and the ROBONIX_LLM_METRICS_DIR forwarding fix is a good catch: without it, every consumer started through the catalog launcher would silently drop its usage records.
Tested 3cdd09a (the current head, rebased on dev 4368021) locally:
- Rust:
cargo fmt --checkandcargo clippy --workspace --tests -D warningsare clean;cargo test --workspace --all-targetspasses 500 tests with 0 failures, includingllm_usage,usage_matches_the_shared_casesandchat_stream_records_each_outcome. - Python: robonix-api 75 passed / 1 skipped, catalog contract 39 OK, Memsearch lifecycle 16 OK, Memory + Scene 51 passed, verifier 30 OK.
- End to end: wrote records through
robonix_api.llm.metrics.Callfor success, HTTP 429, read timeout, a retried call and a 2xx HTML body, then summed them withrbnx llm-usage. Outcomes, attempts, tokens,provider_modeland the--bootfilter all come out as expected.
Rust: fmt, clippy, tests
$ cargo fmt --all -- --check
(no output, exit 0)
$ cargo clippy --workspace --tests -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 38.31s
$ cargo test --workspace --all-targets
test cmd::llm_usage::tests::sums_by_group_and_boot_and_skips_partial_lines ... ok
test usage_matches_the_shared_cases ... ok
test a_call_writes_one_record_with_the_last_reported_usage ... ok
test vlm::tests::chat_stream_records_each_outcome ... ok
...
total over all test binaries: 500 passed; 0 failed
Python suites (same commands as CI)
$ python -m pytest -q pylib/robonix-api/tests
75 passed, 1 skipped, 65 subtests passed in 0.34s
$ python -m unittest discover -s pylib/robonix-api/tests -p 'test_llm_*.py'
Ran 39 tests in 0.285s
OK
$ python -m unittest discover -s services/memsearch/tests -p 'test_lifecycle.py'
Ran 16 tests in 0.026s
OK
$ python -m pytest -q services/memory/tests/{test_llm_search,test_observe,test_service}.py system/scene/tests/{test_llm_catalog,test_runtime_protobuf_contract}.py
51 passed, 5 subtests passed in 0.56s
$ python -m unittest test_verifier.ParsingTests test_verifier.RequestTests
Ran 30 tests in 0.032s
OK
End to end: records written by robonix-api, summed by rbnx llm-usage
Records: 2× pilot chat 200 with usage, 1× pilot chat 429, 1× scene describe httpx.ReadTimeout, 1× memory embed with one retry, 1× memory embed returning a 2xx HTML body.
$ ls $ROBONIX_LLM_METRICS_DIR
memory-<pid>.jsonl
pilot-<pid>.jsonl
scene-<pid>.jsonl
$ head -n1 pilot-<pid>.jsonl
{"v":1,"consumer":"pilot","operation":"chat","host":"api.example.com","model":"demo-vlm","stream":false,"attempts":1,"status":"ok","http_status":200,"input_tokens":120,"output_tokens":30,"cached_input_tokens":null,"reasoning_tokens":null,"provider_model":"demo-vlm-0612","latency_ms":0,"ts_ms":<ts>,"boot_id":"boot-demo","pid":<pid>}
$ rbnx llm-usage
consumer operation host model calls attempts input output cached reasoning unreported lat_n p50_ms p95_ms max_ms status
memory embed localhost:8000 demo-embed 2 3 40 0 0 0 2 2 0 0 0 ok=1 transport_error=1
pilot chat api.example.com demo-vlm 3 3 240 60 0 0 1 3 0 0 0 http_error=1 ok=2
scene describe api.example.com demo-vlm 1 1 0 0 0 0 1 1 0 0 0 timeout=1
$ rbnx llm-usage --boot other
consumer operation host model calls attempts input output cached reasoning unreported lat_n p50_ms p95_ms max_ms status
Non-blocking, for a later change if you like: rbnx llm-usage --json | head -n1 prints the first line and then panics with failed printing to stdout: Broken pipe. It is cosmetic and only happens when the reader closes the pipe early.
LGTM.
Problem
Robonix's model consumers call OpenAI-compatible endpoints directly, and each one keeps its own logging. There is no shared record of which consumer used which model, how many tokens were billed, how long a call took, or how it failed. Pilot has internal usage logs, but Scene, memory, memsearch and the VLM verifier do not expose usage, so an operator cannot attribute model spend or compare outcomes across components.
User-visible impact
rbnx bootcreates<deploy>/rbnx-boot/llm-metrics/and exportsROBONIX_LLM_METRICS_DIRto every child. Each model call made by Pilot, Scene, memory, memsearch and the VLM verifier appends one JSON line with consumer, operation, endpoint host, model, outcome, attempts, token counts and latency.rbnx llm-usagesums those records by consumer, operation, host and model and reports calls, attempts, outcomes, tokens and p50/p95/max latency;--jsonprints one JSON object per group. Records persist across boots and are removed only withrbnx clean -f --llm-metrics. Standalone consumers that do not set the directory write nothing. The change adds no proxy, service, port or model-catalog dependency.Implementation
pylib/robonix-api/robonix_api/llm/metrics.pyaddsCall,usage_from_response()and the JSONL writer.Callrecords success, HTTP errors, timeouts, cancellation and unreadable 2xx bodies; a metrics write failure is logged once and never fails the model call.system/llm-metricsis the Rust counterpart.Callwrites on drop,Usagenormalizes OpenAI and DeepSeek cache fields, andStatusrecords the real outcome. Both writers userobonix_api/llm/contract/usage_cases.json.Pilot records
planandsummarycalls, including stream completion, abandonment, timeout and truncated replies. Scene recordsrelations,image_relations,captionandperception; memory recordssearch,observeandanswer; the verifier recordsjudge.memsearch now sends its own chat-completions request from
COMPACT_PROMPTso thecompactcall has token counts. The previousMemSearch.compactcall returned no usage. Thememsearchdependency range is pinned andhttpxis added.rbnx creates the metrics directory, exports it to children, adds
rbnx llm-usage, and givesrbnx clean -fa separate--llm-metricsswitch. Scene mounts the directory writable. CI adds a robonix-api unit-test job; the existing workspace job covers Rust.Validation
PYTHONPATH=pylib/robonix-api python3 -m unittest pylib/robonix-api/tests/test_llm_metrics.py: 10 passed.PYTHONPATH=pylib/robonix-api:services/memory python3 -m pytest -q services/memory/tests/test_llm_search.py services/memory/tests/test_observe.py: 20 passed.PYTHONPATH=pylib/robonix-api:services/verifiers/vlm_verifier:services/verifiers/vlm_verifier/tests python3 -m unittest test_verifier.ParsingTests test_verifier.RequestTests: 22 passed.PYTHONPATH=pylib/robonix-api:services/memsearch python3 -m unittest discover -s services/memsearch/tests -p 'test_lifecycle.py': 7 passed.cargo test --workspace --all-targets: passed.cargo fmt --all: clean.cargo clippy --workspace --all-targets -- -D warnings: clean.python3 scripts/check_commit_authorship.py --base origin/dev --head HEAD: 6 commits passed.No Webots run. This change only observes model calls, except memsearch now issues the same chat-completions request itself; its lifecycle tests cover that path.
Configuration and migration
ROBONIX_LLM_METRICS_DIRenables recording.rbnx bootsets it by default; an existing value wins.RBNX_BOOT_IDis copied into each record when present. Setting the directory to an empty value disables recording.rbnx clean -fkeeps the directory unless--llm-metricsis passed. No capability contracts, generated code, wire protocols or model requests change, except the memsearch request described above. When the model-catalog launcher lands, it must passROBONIX_LLM_METRICS_DIRandRBNX_BOOT_IDthrough its environment filter.Related issues
None.
CLI output
Fixture run of
rbnx llm-usage -d <fixture>:rbnx llm-usage --jsonemits the same group withlatency_samples,latency_p50_ms,latency_p95_msand ``latency_max_ms`.Review notes
The memsearch change replaces a library call with an in-tree request to obtain usage; reviewers should check prompt and indexing parity against the pinned memsearch version. The Pilot change is larger because a stream needs a real completion/abandonment/timeout outcome for the record; the new tests cover those paths.