Skip to content

fix(executor): save long outputs for retrieval - #319

Merged
enkerewpo merged 7 commits into
devfrom
wheatfox/pilot-output-retrieval-20261108
Nov 14, 2026
Merged

enkerewpo merged 7 commits into
devfrom
wheatfox/pilot-output-retrieval-20261108

Conversation

@enkerewpo

@enkerewpo enkerewpo commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Fixes #306.

Problem

Long capability results were cut before the model could see their end, and nothing kept the rest. read_capability_doc cut a manual to 12,000 bytes, then Pilot replaced any result over 4,096 characters with a 1,365-character preview. read_file was cut to 8,000 bytes, and run_command to 4,000 bytes of stdout and 2,000 bytes of stderr. For the 18,314-byte Minecraft manual reported in #306, the model saw 1,365 characters and the whole Target identity provenance section was lost.

Design

All of those cuts are removed. One place, the leaf event Executor sends to Pilot, decides what the model sees.

  • A text result of at most ROBONIX_OUTPUT_INLINE_BYTES (UTF-8 bytes, default 32 KiB) reaches the model unchanged.
  • A longer result is written whole to a private directory under the system temporary directory (TMPDIR, usually /tmp). The model receives a JSON reference with the call ID, contract, observed_unix, and output: {file, bytes, lines, preview} with a 512-character preview.
  • The standing prompt tells the model to search that file with the existing run_command capability: rg -n, or grep -n -F where rg is not installed, then sed -n to read the matching lines. The Docker image now installs ripgrep.
  • run_command copies stdout and stderr to files while the command runs, so Executor keeps almost none of the output in memory. Output that fits the limit is read back inline and its files are removed. The result always carries the exit code.
  • read_file on a large file returns a reference to that file instead of a copy.
  • Verification and logs keep using the original result. Images keep their multimodal path and are never saved.

Reading a saved file never re-runs the original script or robot action. The prompt tells the model that a file is the call's result at observed_unix, not current state, and that the call must not be repeated to recover text.

Storage

Executor sets no size limit and never deletes saved files. Available space is whatever the temporary directory has, and the operating system clears it. If a write fails, for example because the directory is full, the command still runs to completion, Executor logs a warning, and the result carries storage_error with a preview, so the model tells the user what is missing. A failed save does not change the call's success, so nothing is retried automatically.

Why 32 KiB

The default follows what common coding agents use for one tool result. It is a choice in the same range, not a measured optimum, and it can be changed through ROBONIX_OUTPUT_INLINE_BYTES when Executor starts.

Agent Default limit for one output Unit About, English tokens About, Chinese tokens
OpenCode 50 KiB or 2,000 lines bytes 12,200 10,600
Codex 10,000 tokens, estimated at 4 bytes each tokens 9,500 8,300
Gemini CLI 40,000 characters 9,500 24,800
Claude Code 30,000 for Bash output characters 7,200 18,600
Robonix 32 KiB bytes 7,800 6,800

Token counts use rates measured on DeepSeek Flash: 4.19 bytes per token for an English manual, and 1.61 characters (4.84 bytes) per token for Chinese text. A limit in characters lets Chinese text use more than twice the tokens of English, while a limit in bytes stays close for both, so Robonix counts bytes.

OpenCode and DeepSeek Harness also save the full output to a file and name it in the truncation notice. Robonix does the same, because a robot action cannot simply be run again to see what was cut.

Relation to the context budget

The inline limit applies to one result. Pilot's history budget comes from the model's context window, and compaction cuts any single kept message to 20% of the history room (MESSAGE_SHARE_PCT). An inline result above that cut loses its tail with no file to fall back on. Pilot estimates 4 bytes per token, so 32 KiB is about 8,192 tokens and needs a history room of at least 40,960 tokens. Lower the limit for models with less room. Without a declared context window Pilot does not compact.

Results

Real Atlas, Executor and Pilot processes with deepseek-flash on the official DeepSeek API. No robot was connected and the tasks are synthetic. Each cell is 3 runs, 54 in total: correct answers, then median total input tokens across all rounds of the task.

  • Old: the base commit of this branch.
  • New: this branch with the default limit.
  • Full in context: this branch with the limit raised so that nothing is saved.

The command task runs a script once, finds a code on the last line of its output, then makes three dependent ./step.sh calls. The manual task reads a test manual with read_capability_doc and reports a value and a rule from its last lines.

Task Old New Full in context
Command, 18 KB 0/3, 39,213 3/3, 38,533 3/3, 39,056
Command, 48 KB 0/3, 61,985 3/3, 32,031 3/3, 63,278
Command, 106 KB 0/3, 67,542 3/3, 40,861 3/3, 111,667
Manual, 18 KB 0/3, 17,803 3/3, 10,538 3/3, 10,571
Manual, 48 KB 0/3, 12,176 3/3, 23,421 3/3, 16,731
Manual, 106 KB 0/3, 22,047 3/3, 18,420 3/3, 28,661

All three old runs of the 48 KB manual hit the client timeout, so that token count is partial. At 18 KB the output is under the limit, so new and full behave the same. Saving is cheaper in every larger cell except the 48 KB manual, where the model reads the text only once and the search rounds cost more than the text itself. Saving also adds rounds and time: in the example below, 8 to 12 seconds against about 5.

One run of the 106 KB manual (1,564 lines, both answers in the last lines). The model received this reference:

{
  "output": {
    "file": "/tmp/robonix-output-982511-1794188593/output-0.txt",
    "bytes": 106144,
    "lines": 1564,
    "preview": "# Fixture skill manual ..."
  }
}

It then searched the file and read the lines around the matches:

$ grep -n -F -e 'ZX-491' -e 'Target identity provenance' /tmp/robonix-output-982511-1794188593/output-0.txt
1561:Calibration target ZX-491: violet-cabinet-72946
1563:## Target identity provenance
$ sed -n '1550,1564p' /tmp/robonix-output-982511-1794188593/output-0.txt
Mode Input tokens per round Total Correct
New 3,014, 4,389, 5,270, 6,329 19,002 yes
Full in context 3,014, 25,699 28,713 yes

Tests

cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, and cargo test -p robonix-executor -p robonix-pilot pass locally. New or changed tests cover a long result saved byte for byte with the original unchanged, short results and images not saved, a failed save keeping success and reporting storage_error, saved files never overwritten or deleted, short command output inline with its exit code, 20 MB of command output written whole while the command runs once, a full disk reported while the stream is still drained, and the overlap verifier reusing the saved reference while verifying the original.

Not in this PR

There is no interface yet for clearing one conversation's saved outputs. It needs its own design. This has not been run on a physical robot; testing in TIAGo simulation and on a real robot with real manuals and command output comes after review.

Capability manuals and command results lost information at several
independent truncation points. Save long results before presenting a
bounded preview, and let Pilot search the original output using the
existing Executor shell with rg or POSIX grep and sed.

Preserve verification inputs and report storage failure explicitly.
Use a configurable 32 KiB inline threshold based on measured fixtures.
Document the sampled workload and real DeepSeek retrieval checks.

Refs #306
@github-actions github-actions Bot added type:bug Bug fix (fix:) comp:executor system/executor comp:pilot system/pilot comp:docs docs/ and READMEs and removed type:bug Bug fix (fix:) labels Nov 8, 2026
@github-actions github-actions Bot added the type:bug Bug fix (fix:) label Nov 8, 2026
@github-actions github-actions Bot added the comp:ci .github/ workflows label Nov 8, 2026
Keep one retention path: the leaf event view saves results over
ROBONIX_OUTPUT_INLINE_BYTES (now 16 KiB) and returns one JSON reference
with path, size, line count, call ID, completion time and preview.

- Remove the streaming collector; run_command keeps at most 16 MiB per
  stream, then stops the command and marks the output incomplete.
- Delete the oldest saved files when the storage budget is full instead
  of refusing every later save.
- Bound read_file at 16 MiB; reuse the builtin workspace root.
- Shorten the Pilot retrieval prompt; drop the install script, its CI
  step, the research note and the model-test runner from the repo.
Capping run_command at 16 MiB per stream dropped data and asked the
model to rerun the command with a redirect. Connect stdout and stderr
directly to files instead, so output is neither held in memory nor lost
and the command is never rerun. Short output is read back inline.
Exceeding the whole storage budget stops a command like the timeout,
and what it wrote is kept. read_file now returns a reference to a large
file instead of copying it. References share one shape: file, bytes,
lines and preview.
@enkerewpo enkerewpo changed the title fix(pilot): retain long capability outputs for shell retrieval fix(executor): save long outputs for retrieval Nov 8, 2026
Output is temporary data, so Executor no longer sets a quota or deletes
earlier results. Files go to a private directory under TMPDIR. A failed
write is logged and reported to the model as storage_error, and the
command still runs to completion. Default the inline limit to 32 KiB,
in line with common coding agents, and read it from the environment
Executor starts with.
@enkerewpo
enkerewpo merged commit 2f2fa1a into dev Nov 14, 2026
15 checks passed
@enkerewpo
enkerewpo deleted the wheatfox/pilot-output-retrieval-20261108 branch November 14, 2026 08:13
enkerewpo added a commit that referenced this pull request Nov 16, 2026
Since #319, builtin run_command appends "exit code: N" after the
command's output, so the scenario's regex no longer matched and every
Webots run on dev failed at builtin/run_command. Match the marker
followed by "exit code: 0". Use \s rather than \n so the offline
simulator, which turns \s into a space, still synthesizes a matching
output.

Signed-off-by: Yulong Han <wheatfox17@icloud.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:ci .github/ workflows comp:docs docs/ and READMEs comp:executor system/executor comp:pilot system/pilot type:bug Bug fix (fix:)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] read_capability_doc output is truncated twice: executor 12,000 B, then pilot 4,096 chars

1 participant