Skip to content

fix(automations): deliver Hermes prompts via native query#7862

Merged
Jinwoo-H merged 9 commits into
stablyai:mainfrom
branben:fix/hermes-automation-prompt-delivery
Jul 11, 2026
Merged

fix(automations): deliver Hermes prompts via native query#7862
Jinwoo-H merged 9 commits into
stablyai:mainfrom
branben:fix/hermes-automation-prompt-delivery

Conversation

@branben

@branben branben commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Hermes automations could open a terminal but lose the initial task, leaving Hermes idle.
  • Orca now uses Hermes' native startup contract: hermes chat --query=<prompt> --tui.
  • The prompt is transported outside shell history, normalized across custom command wrappers, and delivered once without the old PTY-readiness/paste timing race.

Implementation

  • Adds a dedicated hermes-query startup mode in the shared agent planner.
  • Builds shell-safe native query commands for POSIX shells, PowerShell, cmd, SSH, and WSL.
  • Preserves multiline, Unicode, flag-shaped, quoted, and spaced arguments.
  • Removes conflicting configured query/TUI flags so Orca owns initial prompt delivery.
  • Removes the transport environment variable before Hermes or its tools inherit it.
  • For Windows PowerShell 5, applies native argv quote escaping before launching Hermes.
  • Forwards the prompt transport variable through WSLENV in both direct and daemon PTY providers.

Test plan

  • 281 focused tests pass on the latest-main merge result.
  • Full typecheck passes.
  • Touched-file lint, switch exhaustiveness, reliability manifest, max-lines ratchet, and diff checks pass.
  • Cross-shell probes pass under sh, zsh, dash, ksh, macOS pwsh, Windows PowerShell, and cmd command shapes.
  • A Windows-only test launches a copied native node.exe as hermes.exe through Windows PowerShell 5 and asserts the exact native argv for a multiline quoted query and spaced custom arguments.
  • Live Orca + Hermes v0.18.2 validation submitted both prompt lines exactly once, produced the requested response, and remained in the interactive TUI.
  • Two fresh independent reviewers returned No issues found after the Windows fix.

Provider and platform coverage

  • Local and daemon PTY paths: covered by deterministic contract tests.
  • macOS/Linux POSIX shells and macOS pwsh: covered by execution probes.
  • Windows PowerShell/cmd: covered by native argv validation and deterministic command tests.
  • WSL and SSH: covered by provider/transport contracts; no live post-fix SSH host was required because the Windows change affects only the encoded PowerShell wrapper.

Performance

Startup planning performs one bounded O(command + environment) transformation per Hermes launch. It adds no polling, listeners, provider fanout, or retained runtime state. Command plus environment size is capped conservatively at 24,000 characters.

Security audit

The literal prompt is absent from the shell command/history. The neutral transport variable remains in the parent terminal environment for compatibility, but the child wrapper removes it before starting Hermes, so Hermes tools do not inherit it. Hermes' native API necessarily places the prompt in the Hermes process argv. No new network endpoint, IPC surface, credential handling, or dependency is introduced.

Screenshots

No Orca layout or copy changes. Reviewer evidence is the real hosted Hermes terminal showing the two-line automation prompt, exact response, and the still-running TUI.

AI review report

CodeRabbit checks passed. Maintainer review replaced the original timing-based paste approach with the native Hermes query path, then completed repeated two-reviewer fix loops and a final Windows-specific review round.

Hermes's prompt_toolkit TUI never emits the DECSET 2004 bracketed-paste
handshake that the default draft-paste readiness waiter gates on, so the
automation prompt was silently dropped and the agent sat idle (terminal
opens, nothing entered). Add a 'process-ready' DraftPasteReadySignal that
arms the quiet-window on first PTY output and pastes once the TUI settles,
and assign it to the hermes agent config.

Adds a Hermes unit test covering the no-handshake path.

Design for the follow-up PR/issue review loop lives in
docs/design/pr-issue-review-loop.md (not committed; gitignored).
@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR adds a process-ready value to DraftPasteReadySignal and configures Hermes to use it. The scanner now arms quiet-window handling on the first PTY output without requiring DECSET 2004 or marker detection. Draft paste flows skip process-inspection fallbacks for this mode and can defer Enter submission until terminal readiness, content echo, or a timeout. New tests cover renderer and PTY-bound Hermes paste timing, deferred submission, echo handling, and timeout behavior.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly matches the main change: Hermes prompt delivery via the native query path and related readiness fixes.
Description check ✅ Passed The description covers Summary, Implementation, Testing, screenshots, AI review, security, and platform coverage; only Notes is missing.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/renderer/src/lib/agent-paste-draft.test.ts (1)

146-170: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add toHaveBeenCalledTimes(1) and keep the quiet-window boundary aligned.
This path only checks the payload; the extra count assertion guards against duplicate paste, and the 1499/1 split should stay in sync with the quiet-window timeout.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 8e029139-8172-49b9-a759-8dc941775331

📥 Commits

Reviewing files that changed from the base of the PR and between 29cbe2d and bffdfc3.

📒 Files selected for processing (3)
  • src/renderer/src/lib/agent-paste-draft.test.ts
  • src/shared/draft-paste-ready-scanner.ts
  • src/shared/tui-agent-config.ts

Comment thread src/shared/tui-agent-config.ts Outdated
Brandon Bennett added 2 commits July 8, 2026 19:14
CodeRabbit: the process-ready fallback consulted waitForAgentReady, which
compares the foreground process basename via isExpectedAgentProcess. Wrapped
interpreter launches surface as 'python3 .../hermes', so the check can never
confirm readiness and only drops the paste. process-ready readiness is the
PTY-quiet window (handled by waitForAgentDraftInputReady); the process-name
fallback is dead for this signal. Skip it and return false on timeout instead.
Guard against a duplicate paste in the no-handshake Hermes path by
asserting sendRuntimePtyInputVerified is called exactly once. Addresses
the CodeRabbit nitpick on PR stablyai#7862.
…e path

pasteDraftToAgentPtyWhenReady (quick-create/work-item route) still ran the
python3-vs-hermes process-name fallback that can never match for
process-ready agents, burning ~1s and dropping the paste. Mirror the
process-ready guard from pasteDraftWhenAgentReady and surface onTimeout.

Adds regression tests for both the quiet-window happy path and the
fallback-skip on the pty-bound path.

Co-authored-by: Orca <help@stably.ai>
Live-testing against Hermes v0.18.2 showed the fixed 50ms post-paste Enter
is swallowed: the node ui-tui takes 15s+ to boot, the paste fires ~1.5s in
(process-ready quiet window), and the cooked-mode line discipline turns the
early \r into \n, which the editor treats as newline-insert. The prompt
parked in the input box and the automation never executed.

For process-ready agents, defer the Enter until the TUI signals
interactivity (DECSET 2004 enable) or echoes the pasted content marker-free
(legacy prompt_toolkit), with a 120s best-effort cap. PTY input is FIFO, so
the deferred Enter always lands after the buffered paste text. The paste
echo of the content itself (raw or caret-notation markers adjacent) is
rejected so a cooked-mode echo can't release the Enter early.

Verified end-to-end in the dev app: automation prompt pasted, deferred
Enter released on tui-ready, Hermes submitted and ran the turn.

Co-authored-by: Orca <help@stably.ai>
@Jinwoo-H

Copy link
Copy Markdown
Contributor

Pushed two commits after an independent review + live end-to-end testing of this PR against the real Hermes CLI (v0.18.2, installed via the official installer):

1. 898d0141e — sibling paste path was left unpatched. pasteDraftToAgentPtyWhenReady (quick-create / work-item direct route, via new-workspace.ts) still ran the python3-vs-hermes process-name fallback this PR removes from the primary path — that route kept dropping the prompt (~1s slower than before, same net loss). Mirrored the process-ready guard and surfaced onTimeout; added regression tests for the quiet-window happy path and the fallback-skip (including a mutation check that the guard is load-bearing).

2. ec6971457 — the submit Enter was swallowed by the TUI boot, so automations still never executed. Live automation runs in the dev app showed the prompt landing in the Hermes input box but never submitting: Hermes v0.18.2's node ui-tui takes 15s+ to boot, the process-ready paste fires ~1.5s in, and submit: true's fixed 50ms Enter arrives while the PTY is still in cooked mode — the line discipline converts \r to \n, which the editor treats as newline-insert. Result: prompt parked in the input box forever, run "completes" with zero agent work (the exact symptom this PR set out to fix, one layer deeper).

Fix: for process-ready agents, defer the Enter until the TUI is interactive — first of: DECSET 2004 enable (modern Hermes), marker-free echo of the pasted content (legacy prompt_toolkit; the cooked-mode echo of the paste itself is rejected via paste-marker adjacency), or a 120s best-effort cap (matches the old exposure). PTY input is FIFO, so the deferred Enter always lands after the buffered paste text.

Live verification (automation → new_per_run workspace → hermes --tui):

  • Before commit 2: prompt visible in Hermes input box, status ready, transcript empty, never submitted (reproduced 3×; a manually injected \r after TUI-ready submitted instantly, isolating the timing as the cause).
  • After commit 2: renderer traces show paste-submit-ready: tui-ready at ~43s (TUI boot) then CR sent; Hermes transcript shows the prompt submitted and a turn executed (reaching the model call). Input box empty, run completed.
  • Also validated the raw PTY mechanics against standalone hermes --tui: early paste survives the 15s+ boot via kernel PTY buffering, no bracketed-paste marker leakage into the input box.

Tests: 49 pass across agent-paste-draft + draft-paste-ready-scanner (3 new deferred-submit cases incl. false-echo rejection and budget expiry); pnpm typecheck clean; oxlint clean. No behavior change for any agent other than process-ready ones (currently Hermes only).

@branben

branben commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @Jinwoo-H this is exactly the layering the unit tests couldn't surface, and the live repro is the part that matters. Confirmed on my end:

One question: keep these as 5 separate commits, or squash before merge? Either works for me; leaving as-is unless a maintainer prefers cleaner history.

@Jinwoo-H Jinwoo-H changed the title fix(automations): deliver prompt to Hermes TUI via process-ready signal fix(automations): deliver Hermes prompts via native query Jul 11, 2026

@Jinwoo-H Jinwoo-H left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @branben — approved after the Windows follow-up. The final implementation uses Hermes’ native startup query, preserves multiline/quoted arguments across POSIX, PowerShell, cmd, SSH, and WSL, removes the transport env before Hermes/tools inherit it, and includes a real Windows PowerShell 5 native-argv regression test. 281 focused tests, typecheck, touched lint/reliability guards, live Hermes validation, and two fresh reviewers are clean.

Co-authored-by: Orca <help@stably.ai>
@Jinwoo-H Jinwoo-H merged commit dcfa919 into stablyai:main Jul 11, 2026
7 checks passed
@branben

branben commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @branben — approved after the Windows follow-up. The final implementation uses Hermes’ native startup query, preserves multiline/quoted arguments across POSIX, PowerShell, cmd, SSH, and WSL, removes the transport env before Hermes/tools inherit it, and includes a real Windows PowerShell 5 native-argv regression test. 281 focused tests, typecheck, touched lint/reliability guards, live Hermes validation, and two fresh reviewers are clean.

I appreciate the quick work on this @Jinwoo-H!

AmethystLiang added a commit that referenced this pull request Jul 13, 2026
Host-owned agent-launch subsystem, orchestration U6 identity/forget,
resolver #7862 Windows-shell + Hermes native-query parity, shared launch
contracts, preload/relay/CLI surfaces. One rebase landing split for review;
only the branch tip is expected to build/test green.

Co-authored-by: Orca <help@stably.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants