Gate hidden terminal renderer skip evidence

[nwparker]

Summary

This PR tightens the terminal perf report budget checker so hidden OpenCode scenarios must prove the behavior they are meant to protect: hidden/background terminal output is skipped at the renderer boundary instead of being written into xterm while the user is typing elsewhere.

Before this, the saved report checker enforced latency, queue bytes, restore latency, and dropped backlogs, but a regression could remove hidden renderer skipping and still pass if CI latency happened to remain under budget. The E2E tests asserted this during live execution, but the first-class report gate did not preserve that invariant when validating saved JSON artifacts.

What Changed

• Added scenario-specific minimum evidence checks for:
  • opencode-cross-workspace-typing
  • opencode-scale-cross-workspace-*
  • opencode-hidden-real-pty-pressure-typing*
  • opencode-hidden-real-pty-restore*
• Hidden typing scenarios now require hiddenSkips >= 1 and hiddenSkippedChars >= 1.
• Hidden restore scenarios now require hiddenSkippedChars >= 1, because restore latency is only meaningful if hidden output existed to restore.
• Added unit tests for failing missing/zero hidden-skip evidence and passing valid hidden-skip evidence.

Benchmark / Verification Evidence

Targeted real Electron perf report run:

SKIP_BUILD=1 ORCA_E2E_OPENCODE_FRAME_COUNT=20 ORCA_E2E_OPENCODE_FRAME_INTERVAL_MS=3 \
  pnpm run test:e2e:terminal-perf:scale:report -- \
  --grep "another workspace streams OpenCode-style output" \
  --report test-results/hidden-render-budget-report.json

Resulting report row:

Scenario	Panes	Frames	Median	Worst	Max Drift	Hidden Skips	Hidden Chars	Renderer Peak Chars	Drops
opencode-cross-workspace-typing	4	20	3.1ms	6.5ms	10.0ms	63	22917	0	0

The strengthened budget checker accepted the generated report:

node config/scripts/check-terminal-perf-report-budgets.mjs test-results/hidden-render-budget-report.json
# Terminal perf budget check passed for 1 annotation row(s).

Focused unit coverage:

pnpm vitest run config/scripts/check-terminal-perf-report-budgets.test.mjs
# Test Files  1 passed (1)
# Tests       8 passed (8)

Touched-file lint:

pnpm exec oxlint config/scripts/check-terminal-perf-report-budgets.mjs config/scripts/check-terminal-perf-report-budgets.test.mjs

Release Safety

This does not change terminal runtime behavior. It only makes CI/report validation fail if hidden-terminal perf scenarios stop proving hidden renderer writes are being skipped. That keeps the next perf iterations isolated from release stabilization while adding a guardrail for the 100-agent direction.

Summary by CodeRabbit

• New Features

  • Enhanced terminal performance report validation to enforce minimum requirements for hidden performance indicators across multiple scenarios.

• Tests

  • Added comprehensive test coverage for hidden metric validation in performance reports.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0f37cd17-c0af-48c0-9d60-251c1aa1e428

📥 Commits

Reviewing files that changed from the base of the PR and between 3d013e4 and 81e1428.

📒 Files selected for processing (2)

• config/scripts/check-terminal-perf-report-budgets.mjs
• config/scripts/check-terminal-perf-report-budgets.test.mjs

---

Walkthrough

This PR adds hidden-scenario classification and minimum-threshold validation to the terminal perf budget checker. It introduces scenario regex lists, matching helpers, and minimum-failure generation to enforce that required hidden evidence metrics are present when hidden scenarios are detected.

Changes

Hidden scenario validation for terminal perf budgets

Layer / File(s)	Summary
Hidden scenario classification and matching helpers config/scripts/check-terminal-perf-report-budgets.mjs	Introduced HIDDEN_RENDERER_SKIP_SCENARIOS and HIDDEN_RESTORE_SCENARIOS regex lists, plus matchesAny() to test scenario patterns and addMinFailure() to generate "below required" violations.
Hidden evidence validation in validateRow() config/scripts/check-terminal-perf-report-budgets.mjs	Extended validateRow() to parse hiddenSkips and hiddenSkippedChars, check them against the hidden scenario lists, and emit minimum-threshold failures when required evidence is missing or insufficient.
Test coverage for hidden evidence validation config/scripts/check-terminal-perf-report-budgets.test.mjs	Added three test cases: one asserting failure when hidden skip evidence is missing, one asserting success when evidence is provided, and one asserting failure for restore scenarios with missing hidden skipped chars evidence.

Possibly related PRs

• stablyai/orca#4813: Directly modifies the same terminal perf budget checker by extending validation logic and related tests for hidden evidence scenarios.
• stablyai/orca#4767: Updates an e2e terminal-load test to assert that hidden renderer skip counters are generated, validating the same hidden-evidence contract that this PR enforces.
• stablyai/orca#4805: Introduces the new hidden PTY pressure e2e scenario that produces and asserts the same hidden renderer skip/restore metrics this PR validates.

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[pullfrog]

✅ No new issues found.

Reviewed changes — adds minimum-evidence checks to the terminal perf budget validator so CI catches regressions where hidden renderer skipping silently drops to zero while latency stays coincidentally under budget.

• Add addMinFailure and matchesAny helpers — complements the existing max-budget checks with inverse minimum-budget validation and regex-based scenario matching.
• Add HIDDEN_RENDERER_SKIP_SCENARIOS and HIDDEN_RESTORE_SCENARIOS — define which hidden-perf scenarios must prove that renderer writes were actually skipped during the test run.
• Enforce minimum hiddenSkips and hiddenSkippedChars — hidden typing scenarios require both ≥ 1; restore scenarios require ≥ 1 hidden chars (restore latency is meaningless without hidden output to restore).

  ^{｜ View workflow run ｜ Using DeepSeek Pro (free via Pullfrog for OSS) ｜ 𝕏}

[nwparker]

Closing as already landed: exact head 81e1428 is an ancestor of merged #7214/current main, which carries and evolves the hidden-renderer evidence gates.