fix(stream): bound and coalesce live tool output previews

[salim4n]

Summary

Fixes #946.

Bound and coalesce live tool-output previews in the direct-session event pump. Previously, every ToolOutputDelta cloned the entire cumulative string into a 512-event broadcast queue. A slow consumer could retain hundreds of large cumulative copies.

Behavior

• Keep a UTF-8-safe 64 KiB tail per tool, including an explicit truncation marker. Small outputs remain cumulative, not raw deltas.
• Emit at most one preview per session per 100 ms, with FIFO fairness among dirty tools and no catch-up burst after a stall.
• Stop admitting previews when the existing broadcast queue contains 16 events. Continue consuming backend events and updating bounded tails without blocking. This limits queued live-preview output payload to 1 MiB per session.
• Forward authoritative ToolResult output through the existing path, without adding truncation. Discard pending preview state before forwarding a result.
• Preserve the bounded latest output in cancellation/stream-close tool frames. Keep detached-command previews across clean turn ends; discard non-detached previews across generation changes.
• Log the first truncation per tool using metadata only; no output or command bodies are logged.

No wire-format, API, DB schema, frontend, or deployment-file changes in this PR.

Regression evidence

The new tool_output_preview integration test runs the real SessionAgentTask pump over a controlled synthetic SessionBackend, with deterministic event acknowledgements and a paused Tokio clock. It is not a mock of the accumulation branch and does not require a live CLI or private data.

The test was added and run before changing the pump on main 5a447e15bc32c4295f437d0ffd530b9992911836:

1,024 fragments, 256 KiB output, stalled subscriber	Before	After
Retained preview payload	100,728,832 bytes	34,816 bytes
Retained preview frames	512	16
Lagged events	512	0

The original assertion failed on 100728832 > 1048576. With the fix, resuming the subscriber delivers the latest bounded tail, not an old backlog.

The eight-session test sends 5 MiB per session, with readers consuming only every 20 producer ticks. Maximum retained preview payload was 8,388,608 bytes across all eight sessions. On a local macOS arm64 run of the test binary, /usr/bin/time -l reported 31,588,352 bytes maximum RSS. This RSS is an observation of the synthetic test process, not a CI threshold or a production/VPS measurement.

Real-agent validation (2026-08-30 UTC)

To cover the outward/reference requirement in AGENTS.md, added the opt-in live test crates/aionui-ai-agent/tests/live_tool_output_preview.rs. It uses the real Codex CLI 0.147.0, through production RealSpawner -> CodexConnection -> SessionAgentTask. Its observation wrapper forwards commands and events unchanged; it does not generate backend events.

The real agent runs a synthetic UTF-8, no-newline output generator in a disposable workspace. After 512 KiB, the generator waits for a reader checkpoint. Only the outer subscriber is suspended; the backend and pump continue consuming. After measuring the retained preview payload, the subscriber resumes at one event per 200 ms while the command completes its full 1 MiB output.

Same live test / paused-reader checkpoint	Old main 5a447e1	Patched pump d92b4d5
Output streamed at checkpoint	524,288 bytes	524,288 bytes
Retained preview payload	17,039,360 bytes	393,216 bytes
Queued events, including metadata	72	16
Assertion: previews <= 1 MiB	Fails	Passes

The patched run then received 1,048,576 bytes in 128 real output deltas and one final result of 1,048,576 bytes. It compares the final result's length and SHA-256 before/after the pump, requires terminal success, bounds each delivered preview, detects subscriber lag, and checks for delayed previews after completion. All passed. The old run intentionally stops at the failing checkpoint assertion; it is not a full-turn success claim for old main.

These are separate real-agent runs with the same generator, not a replay of identical wire timing. Counts below the fixed bound can vary with timing. The changed crate was explicitly rebuilt after switching reference worktrees: an intermediate shared-cache run was discarded, not counted as patched evidence. CLI behavior evidence is this executed live test and its aggregate output, not a guessed wire format. The binary's own --version, app-server --help, and generated JSON schema were also inspected.

To reproduce (requires an existing Codex login; spends model tokens):

export AIONUI_LIVE_CODEX_BIN=/path/to/codex
export AIONUI_LIVE_CODEX_AUTH_FILE=/path/to/auth.json
cargo test -p aionui-ai-agent --test live_tool_output_preview -- --ignored --nocapture

Use separate Cargo target directories for the two worktrees, or rebuild aionui-ai-agent when switching. The test is ignored by default: normal CI does not receive credentials, and a skipped live test is not a live validation. No credentials, private hostnames, real conversation content, or raw CLI session identifiers are included here or in the test.

This exercises the real CLI and the pump, not an authenticated browser or network WebSocket under load. The separate deterministic relay/SQLite test below proves final persistence.

Tests

• Three buffer unit tests: allocation cap, exact-size boundary, fragmented Unicode, fair coalescing, and pending-state removal.
• Twelve integration tests: stalled/slow readers, cadence and missed ticks, hot-tool fairness, multi-megabyte single-line UTF-8, immediate/final/error results, cancellation, EOF, detached commands, generation changes, and eight concurrent session pumps.
• The persistence integration test sends a bounded preview and full result through the real StreamRelay, WebSocket event bus, and in-memory SQLite; it verifies full final output, terminal status, name, arguments, and parent attribution.
• The pre-existing cumulative-output assertion was updated for intentional coalescing; deterministic cadence/backpressure assertions are in the new integration suite.

Initial validation at d92b4d5 on macOS arm64 / Rust 1.95.0:

• cargo test -p aionui-ai-agent -p aionui-conversation — passed (pre-existing ignored tests remain ignored).
• Targeted cargo clippy --all-targets and cargo clippy --workspace -- -D warnings — passed.
• cargo fmt --all -- --check and migration immutability check — passed.
• Mandatory just push — passed its full gate: 9,338 tests passed, zero failures, 52 skipped; branch pushed successfully.
• Additional cargo test --workspace --quiet — completed successfully: 9,338 passed, zero failures, 53 ignored, including doctests in this count.

Follow-up validation for the test-only commit 1af08fd3e087a1ef1337f0e17943b46a84234d08:

• Explicit real-Codex run — 1 passed, including the full successful turn after the slow-reader checkpoint. The same test fails on old main as shown above.
• cargo test -p aionui-ai-agent -p aionui-conversation --quiet — 1,624 passed, zero failures, 13 ignored. The new live test's separate execution is not hidden in this aggregate.
• cargo clippy -p aionui-ai-agent -p aionui-conversation --all-targets -- -D warnings and format/diff checks — passed.
• Additional cargo test --workspace --quiet — 9,338 passed, zero failures, 54 ignored, across 249 groups including doctests.
• Mandatory just push — migration immutability, full-workspace code checks, formatting, and 9,338 tests passed / 53 skipped; branch pushed successfully. The gate did not change the commit.

The upstream Linux CI is separate from these local results. The current run for 1af08fd is action_required, awaiting maintainer approval, with zero jobs executed, as checked on 2026-08-30 UTC after the push. The earlier run for d92b4d5 was gated the same way. Local checks do not substitute for upstream CI. No skipped test is counted as verified.

Reproduce the focused checks with:

cargo test -p aionui-conversation --test tool_output_preview -- --nocapture
cargo test -p aionui-ai-agent session_agent::tool_output
cargo clippy -p aionui-ai-agent -p aionui-conversation --all-targets -- -D warnings

Limits

The bound covers queued live-preview output payload, excluding metadata, receiver-held clones, terminal results, and other event classes. Active tools also retain at most 64 KiB each. This is not a global session/process memory cap, a durable output archive, or a general WebSocket backpressure redesign.

The final result remains whatever the backend supplies; this PR cannot restore output already omitted upstream. Existing terminal-event delivery policy is unchanged. The tests prove the targeted amplification and lifecycle behavior, not the exact heap attribution of the reported production incident.

Deployment evidence is separate

The initial PR validation did not include a deployment. Later, on 2026-08-30 at 21:18 UTC, the same preview fix was backported onto AionCore v0.1.72 and installed on one VPS with AionUi 2.1.61 unchanged. This was a separate backport build, not the main-based PR HEAD, and no deployment files are part of this PR.

That deployment was checked for running-binary identity, startup, unchanged database counts/identity fingerprints and SQLite integrity, successful authenticated HTTP requests, and a short observation with agent teams active. Those checks do not prove a massive-output/slow-reader browser scenario on the VPS. The controlled real-agent test above ran locally, not against those production teams. No authenticated browser canary or comprehensive production-memory guarantee is claimed; nothing was merged automatically.

[salim4n]

Added real-agent/reference evidence to address the verification requirements in AGENTS.md.

Commit 1af08fd3e087a1ef1337f0e17943b46a84234d08 adds only an opt-in live test; it does not change the production fix. Verified source: crates/aionui-ai-agent/tests/live_tool_output_preview.rs, executed against real Codex CLI 0.147.0 through RealSpawner -> CodexConnection -> SessionAgentTask.

At the same 512 KiB synthetic-output checkpoint, old main retains 17,039,360 bytes of preview payload and fails the <=1 MiB assertion; the rebuilt patched pump retains 393,216 bytes and passes. After the reader resumes slowly, the patched run receives all 1,048,576 bytes / 128 deltas, preserves the final result's exact length and SHA-256, completes successfully, and emits no delayed preview after completion. The PR body includes reproduction details and evidence boundaries.

The description now also separates the initial PR validation from the later v0.1.72 backport deployment. It does not claim the main-based PR HEAD was deployed, or that local live testing proves the production/browser load case.

Local validation for this head is complete: affected-crate tests/Clippy and full cargo test --workspace passed; mandatory just push passed its full gate (9,338 passed, 53 skipped) and pushed the branch.

Could a maintainer please approve the upstream CI run for this updated head? It is action_required with zero jobs executed, and the submitting account has no permission to approve it. Local validation is not a substitute for upstream CI. No merge or further deployment is requested.