Repository navigation
fix(executor): save long outputs for retrieval - #319
Merged
Merged
Conversation
Capability manuals and command results lost information at several independent truncation points. Save long results before presenting a bounded preview, and let Pilot search the original output using the existing Executor shell with rg or POSIX grep and sed. Preserve verification inputs and report storage failure explicitly. Use a configurable 32 KiB inline threshold based on measured fixtures. Document the sampled workload and real DeepSeek retrieval checks. Refs #306
Closed
2 of 20 tasks
Keep one retention path: the leaf event view saves results over ROBONIX_OUTPUT_INLINE_BYTES (now 16 KiB) and returns one JSON reference with path, size, line count, call ID, completion time and preview. - Remove the streaming collector; run_command keeps at most 16 MiB per stream, then stops the command and marks the output incomplete. - Delete the oldest saved files when the storage budget is full instead of refusing every later save. - Bound read_file at 16 MiB; reuse the builtin workspace root. - Shorten the Pilot retrieval prompt; drop the install script, its CI step, the research note and the model-test runner from the repo.
Capping run_command at 16 MiB per stream dropped data and asked the model to rerun the command with a redirect. Connect stdout and stderr directly to files instead, so output is neither held in memory nor lost and the command is never rerun. Short output is read back inline. Exceeding the whole storage budget stops a command like the timeout, and what it wrote is kept. read_file now returns a reference to a large file instead of copying it. References share one shape: file, bytes, lines and preview.
Output is temporary data, so Executor no longer sets a quota or deletes earlier results. Files go to a private directory under TMPDIR. A failed write is logged and reported to the model as storage_error, and the command still runs to completion. Default the inline limit to 32 KiB, in line with common coding agents, and read it from the environment Executor starts with.
enkerewpo
added a commit
that referenced
this pull request
Nov 16, 2026
Since #319, builtin run_command appends "exit code: N" after the command's output, so the scenario's regex no longer matched and every Webots run on dev failed at builtin/run_command. Match the marker followed by "exit code: 0". Use \s rather than \n so the offline simulator, which turns \s into a space, still synthesizes a matching output. Signed-off-by: Yulong Han <wheatfox17@icloud.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #306.
Problem
Long capability results were cut before the model could see their end, and nothing kept the rest.
read_capability_doccut a manual to 12,000 bytes, then Pilot replaced any result over 4,096 characters with a 1,365-character preview.read_filewas cut to 8,000 bytes, andrun_commandto 4,000 bytes of stdout and 2,000 bytes of stderr. For the 18,314-byte Minecraft manual reported in #306, the model saw 1,365 characters and the whole Target identity provenance section was lost.Design
All of those cuts are removed. One place, the leaf event Executor sends to Pilot, decides what the model sees.
ROBONIX_OUTPUT_INLINE_BYTES(UTF-8 bytes, default 32 KiB) reaches the model unchanged.TMPDIR, usually/tmp). The model receives a JSON reference with the call ID, contract,observed_unix, andoutput: {file, bytes, lines, preview}with a 512-character preview.run_commandcapability:rg -n, orgrep -n -Fwherergis not installed, thensed -nto read the matching lines. The Docker image now installs ripgrep.run_commandcopies stdout and stderr to files while the command runs, so Executor keeps almost none of the output in memory. Output that fits the limit is read back inline and its files are removed. The result always carries the exit code.read_fileon a large file returns a reference to that file instead of a copy.Reading a saved file never re-runs the original script or robot action. The prompt tells the model that a file is the call's result at
observed_unix, not current state, and that the call must not be repeated to recover text.Storage
Executor sets no size limit and never deletes saved files. Available space is whatever the temporary directory has, and the operating system clears it. If a write fails, for example because the directory is full, the command still runs to completion, Executor logs a warning, and the result carries
storage_errorwith a preview, so the model tells the user what is missing. A failed save does not change the call's success, so nothing is retried automatically.Why 32 KiB
The default follows what common coding agents use for one tool result. It is a choice in the same range, not a measured optimum, and it can be changed through
ROBONIX_OUTPUT_INLINE_BYTESwhen Executor starts.Token counts use rates measured on DeepSeek Flash: 4.19 bytes per token for an English manual, and 1.61 characters (4.84 bytes) per token for Chinese text. A limit in characters lets Chinese text use more than twice the tokens of English, while a limit in bytes stays close for both, so Robonix counts bytes.
OpenCode and DeepSeek Harness also save the full output to a file and name it in the truncation notice. Robonix does the same, because a robot action cannot simply be run again to see what was cut.
Relation to the context budget
The inline limit applies to one result. Pilot's history budget comes from the model's context window, and compaction cuts any single kept message to 20% of the history room (
MESSAGE_SHARE_PCT). An inline result above that cut loses its tail with no file to fall back on. Pilot estimates 4 bytes per token, so 32 KiB is about 8,192 tokens and needs a history room of at least 40,960 tokens. Lower the limit for models with less room. Without a declared context window Pilot does not compact.Results
Real Atlas, Executor and Pilot processes with
deepseek-flashon the official DeepSeek API. No robot was connected and the tasks are synthetic. Each cell is 3 runs, 54 in total: correct answers, then median total input tokens across all rounds of the task.The command task runs a script once, finds a code on the last line of its output, then makes three dependent
./step.shcalls. The manual task reads a test manual withread_capability_docand reports a value and a rule from its last lines.All three old runs of the 48 KB manual hit the client timeout, so that token count is partial. At 18 KB the output is under the limit, so new and full behave the same. Saving is cheaper in every larger cell except the 48 KB manual, where the model reads the text only once and the search rounds cost more than the text itself. Saving also adds rounds and time: in the example below, 8 to 12 seconds against about 5.
One run of the 106 KB manual (1,564 lines, both answers in the last lines). The model received this reference:
{ "output": { "file": "/tmp/robonix-output-982511-1794188593/output-0.txt", "bytes": 106144, "lines": 1564, "preview": "# Fixture skill manual ..." } }It then searched the file and read the lines around the matches:
Tests
cargo fmt --check,cargo clippy --workspace --all-targets -D warnings, andcargo test -p robonix-executor -p robonix-pilotpass locally. New or changed tests cover a long result saved byte for byte with the original unchanged, short results and images not saved, a failed save keeping success and reportingstorage_error, saved files never overwritten or deleted, short command output inline with its exit code, 20 MB of command output written whole while the command runs once, a full disk reported while the stream is still drained, and the overlap verifier reusing the saved reference while verifying the original.Not in this PR
There is no interface yet for clearing one conversation's saved outputs. It needs its own design. This has not been run on a physical robot; testing in TIAGo simulation and on a real robot with real manuals and command output comes after review.