Repository navigation
Conversation
Two-track evaluation spec (deterministic conformance E2E + real-model agent quality) ported from workers PR 503, with session-tree metric attribution, trace-verified triggered work, and multi-prompt conversation scripts folded in from review. Includes the generated roadmap deck at /roadmap/2026-07-15-harness-evaluation/.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds two harness-evaluation specifications and a hash-routed roadmap application with typed content, interactive diagrams, deep-dive pages, verdict displays, and resilient Mermaid loading. ChangesHarness evaluation specifications
Roadmap application
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant CI
participant integrationRunner
participant harness
participant scriptedRouter
participant durableEvidence
CI->>integrationRunner: start isolated scenario
integrationRunner->>scriptedRouter: arm scripted contract
integrationRunner->>harness: invoke harness::send
harness->>scriptedRouter: consume router::chat frames
harness->>durableEvidence: persist transcript and terminal state
integrationRunner->>durableEvidence: collect and grade evidence
sequenceDiagram
participant vitest
participant harnessTest
participant harness
participant provider
participant evidence
vitest->>harnessTest: configure fixture and pinned subject
vitest->>harness: invoke harness::send
harness->>provider: execute production router/provider path
harness-->>harnessTest: emit completion signal
harnessTest->>evidence: read session-tree, metrics, and triggered-work
vitest->>evidence: assert durable results
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
website/roadmap/2026-07-15-harness-evaluation/src/sections/QualityLoopSection.tsx (1)
7-12: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider extracting
BUDGETSto a content dataset file.To maintain a strict separation of concerns and keep this component consistent with how
TRUST_RULES,QUAL_LANES, andQUAL_STEPSare consumed, consider moving the staticBUDGETSarray into../content/protocols.tsalongsideTRUST_RULES.♻️ Proposed refactor
Move the array into
website/roadmap/2026-07-15-harness-evaluation/src/content/protocols.ts:export const BUDGETS = [ { name: 'max_cycles · attempt_timeout_ms', type: 'hard', desc: 'the evaluator refuses work beyond them and calls harness::stop at the deadline.' }, { name: 'max_total_tokens · max_cost_usd', type: 'soft ceiling', desc: 'visible only after a model turn; may overshoot by one bounded turn, recorded, never mislabeled as hard.' }, { name: 'max_browser_actions · max_network_requests', type: 'soft ceiling', desc: 'checked before the next unit; a fixture can pin network activity to zero.' }, { name: 'attempts', type: '1–10 per leg', desc: 'comparison legs run in a persisted, seed-derived interleaved schedule, reproducible pair by pair.' }, ] as constThen, update the imports in this file:
-import { TRUST_RULES } from '../content/protocols' +import { BUDGETS, TRUST_RULES } from '../content/protocols' -const BUDGETS = [ - { name: 'max_cycles · attempt_timeout_ms', type: 'hard', desc: 'the evaluator refuses work beyond them and calls harness::stop at the deadline.' }, - { name: 'max_total_tokens · max_cost_usd', type: 'soft ceiling', desc: 'visible only after a model turn; may overshoot by one bounded turn, recorded, never mislabeled as hard.' }, - { name: 'max_browser_actions · max_network_requests', type: 'soft ceiling', desc: 'checked before the next unit; a fixture can pin network activity to zero.' }, - { name: 'attempts', type: '1–10 per leg', desc: 'comparison legs run in a persisted, seed-derived interleaved schedule, reproducible pair by pair.' }, -] as const🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@website/roadmap/2026-07-15-harness-evaluation/src/sections/QualityLoopSection.tsx` around lines 7 - 12, Move the static BUDGETS array from QualityLoopSection.tsx into the shared content module protocols.ts, exporting it alongside TRUST_RULES, QUAL_LANES, and QUAL_STEPS. Remove the local declaration and update QualityLoopSection.tsx to import BUDGETS from that content module while preserving its existing usage and readonly typing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@website/roadmap/2026-07-15-harness-evaluation/src/sections/QualityLoopSection.tsx`:
- Around line 7-12: Move the static BUDGETS array from QualityLoopSection.tsx
into the shared content module protocols.ts, exporting it alongside TRUST_RULES,
QUAL_LANES, and QUAL_STEPS. Remove the local declaration and update
QualityLoopSection.tsx to import BUDGETS from that content module while
preserving its existing usage and readonly typing.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: e1ecb825-6325-4f6d-8af0-5a7c12f106a4
📒 Files selected for processing (31)
tech-specs/2026-07-15-harness-evaluation/README.mdtech-specs/2026-07-15-harness-evaluation/agent-quality.mdtech-specs/2026-07-15-harness-evaluation/conformance-e2e.mdwebsite/roadmap/2026-07-15-harness-evaluation/src/App.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/content/architecture.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/contract.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/deck.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/flows.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/hero.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/lifecycle.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/oracles.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/payoff.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/protocols.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/tracks.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/verdicts.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/why.tswebsite/roadmap/2026-07-15-harness-evaluation/src/diagrams/.gitkeepwebsite/roadmap/2026-07-15-harness-evaluation/src/pages/AgentQualityProtocolPage.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/pages/ConformanceContractsPage.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/ConformanceRunSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/ContractSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/Hero.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/LifecycleSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/MapSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/OraclesSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/PayoffSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/QualityLoopSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/TracksSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/VerdictsSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/WhySection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/spec-docs.ts
…enerated validator rules. Adjusted last reviewed date and refined completion criteria. Enhanced documentation for generated validators in the roadmap.
Track failed function calls in MetricSetV1 (subject and triggered work), with a per-session breakdown in SessionUsageV1. The counter is diagnostic: nonzero errors only fail an attempt when a validator or declared budget says so.
Rename the harness-evaluation spec's deterministic test track from "conformance" to "integration" across the tech spec markdown and its roadmap deck: file rename to integration-e2e.md, proposed identifiers (make target, harness-integration CI job, integration-recorder::*, IntegrationScenarioV1/ResultV1), I-E2E-* scenario ids, and the deck's components, route slug, and content. The old contrast sentence now distinguishes internal white-box tests from public-path integration.
Spec: mark the integration track's core as implemented (compiler, live-contract readiness, typed teardown), defer cassette tooling, and gate agent quality on the durable session-tree dependency. Deck: refresh every stale claim to match — split boot order in the run sequence, twelve-stage lifecycle with dual sha-256-linked reports, in-process recorder control plane, harness-core-v1 profile, C-E2E-* scenario ids, authoring/compilation deep dive, and per-track status rows. Adds a "break the run" interactive: eight sabotages, the oracle that catches each, and all six classifications reachable.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
website/roadmap/2026-07-15-harness-evaluation/src/pages/IntegrationContractsPage.tsx (1)
131-132: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueInclude the
[--retain-success]flag for consistency.The supervisor CLI code block omits the
[--retain-success]flag that is documented in theintegration-e2e.mdspecification. Consider adding it so the UI closely mirrors the documented CLI parameters.♻️ Proposed refactor
{'\n '}[--repeat <S><n></S>] <M>\</M> - {'\n '}--artifacts-dir <S><path></S> + {'\n '}--artifacts-dir <S><path></S> <M>\</M> + {'\n '}[--retain-success]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@website/roadmap/2026-07-15-harness-evaluation/src/pages/IntegrationContractsPage.tsx` around lines 131 - 132, Update the supervisor CLI usage block in IntegrationContractsPage to include the optional [--retain-success] flag, matching the parameter documented in integration-e2e.md while preserving the existing argument formatting and ordering.website/roadmap/2026-07-15-harness-evaluation/src/pages/AgentQualityProtocolPage.tsx (1)
57-60: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAlign the
generatorstructure with the specification.In the abridged manifest example,
generatoris shown as a string (pinned-generator-model), but the specification strictly defines it as an object requiring{ model: string }. Updating the snippet keeps the UI consistent with the manifest schema.♻️ Proposed refactor
- {'\n '}- source: <M>{'{ '}</M>kind: <S>generated</S>, goal: <S>"…"</S>, generator:{' '} - <S>pinned-generator-model</S> <M>{'}'}</M> + {'\n '}- source: <M>{'{ '}</M>kind: <S>generated</S>, goal: <S>"…"</S>, generator:{' '} + <M>{'{ '}</M>model: <S>pinned-generator-model</S> <M>{' }'}</M> <M>{'}'}</M>🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@website/roadmap/2026-07-15-harness-evaluation/src/pages/AgentQualityProtocolPage.tsx` around lines 57 - 60, Update the abridged manifest example in AgentQualityProtocolPage so the generator field uses the specification’s object shape with a model string, rather than a bare string value. Preserve the surrounding kind, goal, and comment example unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@website/roadmap/2026-07-15-harness-evaluation/src/diagrams/BreakTheRun.tsx`:
- Around line 31-37: Update the select handler’s setVisited call to use a
functional state updater, evaluating the latest visited state when checking and
appending next.classification. Preserve the existing no-duplicate behavior and
only update when next exists and the classification has not been visited.
---
Nitpick comments:
In
`@website/roadmap/2026-07-15-harness-evaluation/src/pages/AgentQualityProtocolPage.tsx`:
- Around line 57-60: Update the abridged manifest example in
AgentQualityProtocolPage so the generator field uses the specification’s object
shape with a model string, rather than a bare string value. Preserve the
surrounding kind, goal, and comment example unchanged.
In
`@website/roadmap/2026-07-15-harness-evaluation/src/pages/IntegrationContractsPage.tsx`:
- Around line 131-132: Update the supervisor CLI usage block in
IntegrationContractsPage to include the optional [--retain-success] flag,
matching the parameter documented in integration-e2e.md while preserving the
existing argument formatting and ordering.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 352157ed-dc11-4f61-874a-2b8584b931e8
📒 Files selected for processing (29)
tech-specs/2026-07-15-harness-evaluation/README.mdtech-specs/2026-07-15-harness-evaluation/agent-quality.mdtech-specs/2026-07-15-harness-evaluation/integration-e2e.mdwebsite/roadmap/2026-07-15-harness-evaluation/src/App.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/content/architecture.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/contract.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/deck.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/flows.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/hero.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/lifecycle.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/oracles.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/payoff.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/protocols.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/sabotage.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/tracks.tswebsite/roadmap/2026-07-15-harness-evaluation/src/diagrams/BreakTheRun.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/pages/AgentQualityProtocolPage.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/pages/IntegrationContractsPage.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/BreakSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/ContractSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/Hero.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/IntegrationRunSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/LifecycleSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/MapSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/OraclesSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/PayoffSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/QualityLoopSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/TracksSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/VerdictsSection.tsx
🚧 Files skipped from review as they are similar to previous changes (18)
- website/roadmap/2026-07-15-harness-evaluation/src/content/hero.ts
- website/roadmap/2026-07-15-harness-evaluation/src/content/deck.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/OraclesSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/oracles.ts
- website/roadmap/2026-07-15-harness-evaluation/src/content/payoff.ts
- website/roadmap/2026-07-15-harness-evaluation/src/content/contract.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/LifecycleSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/sections/TracksSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/sections/MapSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/sections/QualityLoopSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/sections/PayoffSection.tsx
- tech-specs/2026-07-15-harness-evaluation/README.md
- website/roadmap/2026-07-15-harness-evaluation/src/sections/VerdictsSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/App.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/sections/Hero.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/architecture.ts
- website/roadmap/2026-07-15-harness-evaluation/src/content/flows.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/ContractSection.tsx
Comment on lines
+31
to
+37
| const select = (id: string) => { | ||
| setModeId(id) | ||
| const next = SABOTAGE_MODES.find((m) => m.id === id) | ||
| if (next && !visited.includes(next.classification)) { | ||
| setVisited([...visited, next.classification]) | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Prevent stale state when rapidly clicking.
Using the closure value of visited inside setVisited can cause dropped updates if a user clicks multiple buttons rapidly before the component re-renders. A functional state update ensures the latest state is always evaluated.
💚 Proposed fix
const select = (id: string) => {
setModeId(id)
const next = SABOTAGE_MODES.find((m) => m.id === id)
- if (next && !visited.includes(next.classification)) {
- setVisited([...visited, next.classification])
- }
+ if (next) {
+ setVisited((prev) =>
+ prev.includes(next.classification) ? prev : [...prev, next.classification]
+ )
+ }
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const select = (id: string) => { | |
| setModeId(id) | |
| const next = SABOTAGE_MODES.find((m) => m.id === id) | |
| if (next && !visited.includes(next.classification)) { | |
| setVisited([...visited, next.classification]) | |
| } | |
| } | |
| const select = (id: string) => { | |
| setModeId(id) | |
| const next = SABOTAGE_MODES.find((m) => m.id === id) | |
| if (next) { | |
| setVisited((prev) => | |
| prev.includes(next.classification) ? prev : [...prev, next.classification] | |
| ) | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@website/roadmap/2026-07-15-harness-evaluation/src/diagrams/BreakTheRun.tsx`
around lines 31 - 37, Update the select handler’s setVisited call to use a
functional state updater, evaluating the latest visited state when checking and
appending next.classification. Preserve the existing no-duplicate behavior and
only update when next exists and the classification has not been visited.
Per the design review discussion on 2026-07-20: - agent-quality: vitest test files over public iii/harness functions replace the YAML scenario manifest and the harness-eval worker surface; evidence (session-tree, metrics, trace propagation) becomes default harness API consumed via minimal harness-test helpers; traces are first-class evidence; comparison, held-out/generated validators, and production eval move to future work. - integration: scenarios are authored as Rust builder modules instead of YAML (migration before Phase 1); the later console profile uses Playwright with the shared harness-test worker, no LLM calls.
Agent-quality slides now show the vitest suite (trigger() + expect(), harness-test helpers, default evidence contracts) instead of the harness-eval worker, manifest, and validator protocol; integration slides show Rust builder authoring instead of scenario.yaml. Also fixes diagram geometry: funnel copy sized to its boxes, and both sequence players authored at width 1030 so the last lane header is not clipped by the default viewBox.
The Mermaid component lazy-imports mermaid, so the Vite dev server only discovered the dep on first use and the dynamic import could 404 a not-yet-optimized chunk, silently falling back to the diagram source. Pre-bundle it via optimizeDeps.include, and move the import inside the component's try so a failed module fetch degrades to the fallback instead of an unhandled rejection.
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
website/roadmap/2026-07-15-harness-evaluation/src/pages/AgentQualityProtocolPage.tsx (1)
110-130: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winRender the
TRUST_RULEScontent.The
TRUST_RULESarray is defined inprotocols.tsand explicitly outlined in the PR objectives as a key part of the agent-quality protocol, but it is not imported or rendered on this page.As per path instructions, ensure the communications on the website are consistent with the documentation. Please add a
SpecSheetfor the trust rules so this protocol content is visible.💡 Proposed fix to render the rules
Apply this change to render the rules:
- <SpecSheet title="future work — deferred, not redesigned" meta="enabled by the default assets" defaultOpen> + <SpecSheet title="trust rules" meta="isolated and explicit" defaultOpen> + <div className="flex flex-col"> + {TRUST_RULES.map((row) => ( + <SpecRow key={row.name} name={row.name}> + {row.desc} + </SpecRow> + ))} + </div> + </SpecSheet> + + <SpecSheet title="future work — deferred, not redesigned" meta="enabled by the default assets" defaultOpen>Update the imports at the top of the file to include
TRUST_RULES:import { AUTHORING_RULES, EVIDENCE_CONTRACTS, FUTURE_WORK, QUALITY_HELPERS, SCENARIO_CORPUS, TRUST_RULES, } from '../content/protocols'🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@website/roadmap/2026-07-15-harness-evaluation/src/pages/AgentQualityProtocolPage.tsx` around lines 110 - 130, Import TRUST_RULES from the protocols content module and add a SpecSheet in AgentQualityProtocolPage alongside the existing AUTHORING_RULES and FUTURE_WORK sheets. Render TRUST_RULES with the same SpecRow mapping pattern so the trust-rule protocol content is visible.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In
`@website/roadmap/2026-07-15-harness-evaluation/src/pages/AgentQualityProtocolPage.tsx`:
- Around line 110-130: Import TRUST_RULES from the protocols content module and
add a SpecSheet in AgentQualityProtocolPage alongside the existing
AUTHORING_RULES and FUTURE_WORK sheets. Render TRUST_RULES with the same SpecRow
mapping pattern so the trust-rule protocol content is visible.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 4361a861-236c-469a-a43e-ffb134174c80
📒 Files selected for processing (21)
tech-specs/2026-07-15-harness-evaluation/README.mdtech-specs/2026-07-15-harness-evaluation/agent-quality.mdtech-specs/2026-07-15-harness-evaluation/integration-e2e.mdwebsite/astro.config.mjswebsite/roadmap/2026-07-15-harness-evaluation/src/content/architecture.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/flows.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/hero.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/payoff.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/protocols.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/sabotage.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/tracks.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/verdicts.tswebsite/roadmap/2026-07-15-harness-evaluation/src/pages/AgentQualityProtocolPage.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/pages/IntegrationContractsPage.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/ContractSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/Hero.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/IntegrationRunSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/MapSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/QualityLoopSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/VerdictsSection.tsxwebsite/roadmap/src/content/mermaid.tsx
🚧 Files skipped from review as they are similar to previous changes (12)
- website/roadmap/2026-07-15-harness-evaluation/src/sections/VerdictsSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/sections/QualityLoopSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/hero.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/ContractSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/pages/IntegrationContractsPage.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/sabotage.ts
- website/roadmap/2026-07-15-harness-evaluation/src/content/verdicts.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/MapSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/architecture.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/IntegrationRunSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/tracks.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/Hero.tsx
…to integration/e2e tests
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tech-specs/2026-07-15-harness-evaluation/integration-e2e.md (1)
1045-1048: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winPin
actions/checkoutto a commit SHA. Theharness-integrationjob still usesactions/checkout@v4; replace it with a full commit SHA and note the corresponding release tag.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tech-specs/2026-07-15-harness-evaluation/integration-e2e.md` around lines 1045 - 1048, Update the actions/checkout invocation in the harness-integration job to use a full commit SHA instead of the mutable v4 tag, and add an adjacent comment documenting the release tag associated with that SHA.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tech-specs/2026-07-15-harness-evaluation/integration-e2e.md`:
- Around line 559-562: Update the teardown contract around the SIGTERM/SIGKILL
sequence to enforce the configured teardown_ms deadline rather than an
unconditional five-second wait, and target each process group so descendants are
terminated along with direct children. Preserve the existing escalation order
while ensuring teardown completes within the deadline.
---
Nitpick comments:
In `@tech-specs/2026-07-15-harness-evaluation/integration-e2e.md`:
- Around line 1045-1048: Update the actions/checkout invocation in the
harness-integration job to use a full commit SHA instead of the mutable v4 tag,
and add an adjacent comment documenting the release tag associated with that
SHA.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 6627022c-5ddc-4619-9ebf-10d0c6accc73
📒 Files selected for processing (28)
tech-specs/2026-07-15-harness-evaluation/README.mdtech-specs/2026-07-15-harness-evaluation/agent-quality.mdtech-specs/2026-07-15-harness-evaluation/integration-e2e.mdwebsite/roadmap/2026-07-15-harness-evaluation/src/App.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/content/architecture.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/contract.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/deck.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/flows.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/hero.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/oracles.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/payoff.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/protocols.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/tracks.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/verdicts.tswebsite/roadmap/2026-07-15-harness-evaluation/src/content/why.tswebsite/roadmap/2026-07-15-harness-evaluation/src/pages/E2EProtocolPage.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/pages/IntegrationContractsPage.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/BreakSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/ContractSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/E2ESection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/Hero.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/IntegrationRunSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/LifecycleSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/MapSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/OraclesSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/PayoffSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/TracksSection.tsxwebsite/roadmap/2026-07-15-harness-evaluation/src/sections/VerdictsSection.tsx
🚧 Files skipped from review as they are similar to previous changes (21)
- website/roadmap/2026-07-15-harness-evaluation/src/sections/BreakSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/sections/LifecycleSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/deck.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/OraclesSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/hero.ts
- website/roadmap/2026-07-15-harness-evaluation/src/content/architecture.ts
- website/roadmap/2026-07-15-harness-evaluation/src/content/payoff.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/Hero.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/contract.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/PayoffSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/App.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/sections/ContractSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/pages/IntegrationContractsPage.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/sections/IntegrationRunSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/why.ts
- website/roadmap/2026-07-15-harness-evaluation/src/content/oracles.ts
- website/roadmap/2026-07-15-harness-evaluation/src/sections/MapSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/sections/TracksSection.tsx
- website/roadmap/2026-07-15-harness-evaluation/src/content/flows.ts
- tech-specs/2026-07-15-harness-evaluation/README.md
- tech-specs/2026-07-15-harness-evaluation/agent-quality.md
ytallo
changed the title
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates the harness evaluation documentation and its interactive presentation to match the current worker implementation, with integration and quality/E2E covered as two views of one public harness path.
What changed
Integration tests (
integration-e2e.md)integration-probecovers the direct and playground paths.E2E-001,E2E-002,UI-001,UI-002), and report/gate behavior.Quality/E2E tests (
agent-quality.md)Interactive presentation (presentation source)
Validation
pnpm exec biome check website/roadmap/2026-07-15-harness-evaluation/srcpnpm --filter iii-website type-checkpnpm --filter iii-website buildpnpm --filter iii-website test:dist(19/19)