Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions docs/designs/pre-fanout-e2e-actionability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Pre-fan-out E2E actionability

## Problem

INV-46 treats every non-zero E2E lane result as a code defect. Some project
adapters use one non-zero exit code for both code failures and operator/runtime
failures, so the exit code cannot safely decide whether DEV should run. Because
the gate exits before review fan-out, it also lacks the Reviewed HEAD evidence
that INV-98 uses to avoid re-reviewing the same commit.

## Smallest native extension

The review wrapper creates a private per-run E2E lane directory and exports
`E2E_FAILURE_CLASSIFICATION_FILE`, whose value is exactly
`$lane_dir/e2e-failure-classification`. Both command and browser integrations
inherit it. On a non-zero lane result an integration may atomically rename a
regular file into that path containing exactly one of:

```text
dev-actionable=true
dev-actionable=false
```

Absence preserves the historical fail-open `true`. Presence is accepted only
for a non-symlink regular file no larger than 1 KiB with the exact one-line
grammar. Every unsafe or invalid present object resolves fail-closed to
`dev-actionable=false`. The wrapper never derives actionability from exit codes,
logs, evidence prose, or agent judgment.

Before moving `reviewing` to `pending-dev`, an E2E failure must durably write the
strict issue/full-HEAD/self-authored pre-fan-out disposition
`result=e2e-failed`, followed by INV-92's `failed-substantive` trailer carrying
the classification. The disposition deliberately has no actionability field:
it only proves that this HEAD completed a terminal pre-fan-out decision, while
INV-92 remains the single durable actionability authority. Any required write
failure leaves the issue out of `pending-dev`.

The dispatcher then uses its existing same-HEAD verdict-aware recovery:
`dev-actionable=true` reaches the bounded correction route and
`dev-actionable=false` reaches the INV-92 operator/stall route. A changed HEAD
remains review-owned. Conflict, convergence, request-changes, retry, and strict
linkage machinery remain unchanged.
37 changes: 37 additions & 0 deletions docs/pipeline/invariants.md
Original file line number Diff line number Diff line change
Expand Up @@ -339,6 +339,43 @@ In environments without Layer 3 (server-side), Layers 1 and 2 must both be insta

---

## INV-150: a pre-fan-out E2E failure may be classified only by an optional wrapper-owned private-lane sidecar, while INV-92's verdict trailer remains the single durable actionability authority

The review wrapper exports a path inside its private E2E lane directory to both
command and browser integrations. For a non-zero lane result, absence preserves
the legacy `dev-actionable=true` behavior. A present sidecar is trusted only
when it is a non-symlink regular file of at most 1 KiB containing exactly one
newline-terminated `dev-actionable=true|false` line; every malformed or unsafe
present object resolves to `false`. Producers should write a sibling temporary
regular file and atomically rename it into place. No exit-code, log, report, or
LLM inference participates.

Before `reviewing -> pending-dev`, the wrapper must durably produce a strict
current-issue/full-HEAD/self-authored `result=e2e-failed` disposition and then a
`failed-substantive dev-actionable=<classification> head=<HEAD>` verdict. The
disposition is deliberately neutral: it proves pre-fan-out review completion,
while INV-92 alone owns durable actionability. Required-write failure cannot
make pending-dev observable. Same-HEAD dispatcher routing therefore reaches the
existing bounded correction authority for `true` and INV-92's operator/stall
route for `false`; a new HEAD remains pending-review-owned.

**Producer**: review wrapper and the configured E2E integration's optional
atomic sidecar write.

**Consumer**: `lib-review-e2e.sh` strict parser,
`lib-review-disposition.sh`, and `handle_pending_dev_pr_exists`'s existing
verdict-aware route.

**Status**: **ENFORCED**.

**Tests**: `tests/unit/test-e2e-failure-actionability.sh`,
`tests/unit/test-review-disposition.sh`, and
`tests/unit/test-dispatcher-review-disposition-routing.sh`.

**Design**: [`docs/designs/pre-fanout-e2e-actionability.md`](../designs/pre-fanout-e2e-actionability.md).

---

## INV-141: token budgets use strict accounting for post-run wrapper gates and pre-dispatch admission, with durable terminal routing

_Triage (issue #236): [machine-checked: tests/unit/test-lib-token-budget.sh, tests/unit/test-token-budget-wiring.sh, tests/unit/test-token-budget-e2e.sh, tests/e2e/run-token-budget-gates-e2e.sh]_
Expand Down
9 changes: 7 additions & 2 deletions docs/pipeline/review-agent-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ before either INV-46 E2E lane can start.
The pre-fan-out machine evidence is an exact whole-comment marker:

```text
<!-- review-disposition: issue=<N> head=<full-lowercase-head> phase=pre-fanout result=<conflict-rebase|mergeable-unknown> -->
<!-- review-disposition: issue=<N> head=<full-lowercase-head> phase=pre-fanout result=<conflict-rebase|mergeable-unknown|e2e-failed> -->
```

`lib-review-disposition.sh` owns rendering and parsing. It requires a strict
Expand Down Expand Up @@ -304,7 +304,11 @@ E2E_ACTIVE == true:
GATE_FAIL_STALL_THRESHOLD (default 2) AND not
already stalled (NO may_stall_now — see below) →
−reviewing +stalled ; ONE report ; exit 0
else → [BLOCKING] E2E finding + failed-substantive
else → read optional strict lane-local
E2E_FAILURE_CLASSIFICATION_FILE (INV-150),
persist current-HEAD result=e2e-failed then
failed-substantive dev-actionable=<value>,
then [BLOCKING] E2E finding
+ submit_request_changes (INV-52, best-effort)
+ −reviewing +pending-dev ; exit 0 (NO fan-out)
gate == block-nonsubstantive → "Review held" + failed-non-substantive
Expand All @@ -318,6 +322,7 @@ E2E_ACTIVE == false → no lane, no gate (E2E_GATE=inactive); straight to fan-ou
- **command-mode lane is shell, not an LLM.** `_run_command_e2e_lane` runs entirely in the wrapper shell — no `run_agent`, no tokens. The verify command runs under `setsid` + `timeout --kill-after=30s --signal=TERM ${E2E_COMMAND_TIMEOUT_SECONDS}` so its subtree is reapable ([INV-23](invariants.md#inv-23-pid_file-points-at-a-process-whose-death-reaps-the-entire-agent-subtree)); the lane's PGID is added to `_reap_fanout_processes`'s arg list alongside the fan-out agents'.
- **browser-mode is ONE LLM lane**, not replicated across review agents. **Report broker ([INV-79]):** the LLM WRITES the report to `$E2E_REPORT_FILE` and the wrapper posts it (`_post_brokered_e2e_report`) — matching the verdict-artifact broker direction so report DELIVERY does not depend on the agent's own write capability under the scoped token (the agent's direct post is a retained fallback via `issues:write`). Before posting, the broker DEDUPS against the review window: its `_existing` in-window `## E2E Verification Report` count now routes through `itp_list_comments "$PR_NUMBER"` ([INV-87](invariants.md#inv-87-provider-dispatch-is-spec-defined--callers-route-every-issuecode-host-op-through-itp_chp_-never-a-raw-gh-in-the-caller-layer)/[INV-90](invariants.md#inv-90-the-normalized-issue-comment-shape-is-id-author-body-createdat-sorted-ascending-by-createdat-with-author-a-machine-handle-for-exact-equality), #333) rather than a raw `gh api .../issues/<PR>/comments`; the caller-side `select((.createdAt >= "$WRAPPER_START_TS") and (.body | contains("## E2E Verification Report")))|length` is a literal `contains`/`>=` test (no `test()`/regex, so no RE2→Oniguruma divergence — the #319/#321 lesson) and the retained `| tail -n1` is a zero-cost defensive collapse against a future re-paginating provider. The wrapper then stamps `<!-- e2e-evidence: complete sha="${PR_HEAD_SHA}" -->` **onto the `## E2E Verification Report` comment** (`_stamp_browser_evidence_marker`, REST `PATCH`, idempotent) so the gate anchor is deterministic without the LLM transcribing the SHA — and so the gate's evidence-present signal + the review agents' evidence-read both resolve to the REAL report, not a marker-only comment. A clean exit with NO stampable report comment fails the gate closed (rc forced non-zero) — a marker-only comment can never satisfy the gate. **Since #345** (#296 deferred), `_stamp_browser_evidence_marker`'s id-lookup + body-fetch also route through `itp_list_comments "$PR_NUMBER"` — ONE call replaces the former raw GET-comment-id (`gh api …/issues/<PR>/comments --paginate --jq … | last | .id`) plus a second raw GET-body-by-id (`gh api …/issues/comments/<id> --jq .body`). The caller-side select re-expresses the author/time/marker predicates over the normalized fields (`.author`, `.createdAt`) and picks the newest match via `sort_by(.createdAt // "", .id // 0) | last` (the #321 verdict-poll tie-break); the selected element's `.body` serves both the id and the body, so the second GET is redundant and dropped. This retires the former [INV-46] carve-out that kept these two reads raw ("out of #333's scope").
- **The gate is a mechanical dual-signal**, fail-closed: a crash between parser-ok and comment-post (`rc=0` but no SHA-matching evidence) routes `block-nonsubstantive` (transient re-queue), not a dev bounce; a real verify failure (`rc≠0`) routes `fail` (substantive). A `rc≠0`-with-stale-present-evidence does NOT pass.
- **Failure actionability is explicit and lane-local ([INV-150](invariants.md#inv-150-a-pre-fan-out-e2e-failure-may-be-classified-only-by-an-optional-wrapper-owned-private-lane-sidecar-while-inv-92s-verdict-trailer-remains-the-single-durable-actionability-authority)).** The wrapper exports `E2E_FAILURE_CLASSIFICATION_FILE=$lane_dir/e2e-failure-classification` to both command and browser lanes. A non-zero lane may atomically install exactly `dev-actionable=true` or `dev-actionable=false`. Absence preserves legacy `true`; any present non-regular, symlink, oversized, unreadable, or malformed object fails closed to `false`. Exit codes, logs, report prose, and LLM output are never classifiers. Before `pending-dev`, the wrapper requires the neutral `result=e2e-failed` disposition and the matching INV-92 trailer; the disposition intentionally carries no duplicate actionability.
- **Evidence-freshness pre-check on a new HEAD (issue #449, R3).** Same-HEAD reuse (`_fetch_sha_evidence` + the lane's own reuse block) already works for a repeat check against an UNCHANGED head. The gap this closes: after a NEW head, the lane runs clean (`rc==0`) but a SHA-matching evidence comment is not yet visible on re-fetch — purely a GitHub propagation lag on the comment the lane itself JUST posted. `lib-review-e2e.sh::_e2e_ci_green_precheck <pr_num>` is consulted ONLY in that exact case (`rc==0` AND `evidence_present==0`, right before `_classify_e2e_gate` is called): when the PR's overall CI status (`chp_ci_status`, independent of this wrapper's own dedicated E2E lane) is already `green` for the current HEAD, the wrapper treats that as satisfying the evidence requirement (`evidence_present` is set to `1`) rather than routing to `block-nonsubstantive` and waiting for the comment to propagate. A `rc≠0` lane failure is UNTOUCHED — it always routes to `fail` regardless of CI status, so a red/pending CI never changes existing fail semantics. Design note: the dispatcher-side `ci_is_green` (`lib-dispatch.sh`) is NOT reused — `autonomous-review.sh` does not source `lib-dispatch.sh` (no precedent for cross-sourcing dispatcher logic into the review wrapper, mirroring [INV-122]'s own deliberate non-reuse of `may_stall_now`); `_e2e_ci_green_precheck` is a review-wrapper-local equivalent calling the SAME already-sourced `chp_ci_status` primitive directly.
- **Same-HEAD circuit breaker on repeated `gate == fail` ([INV-122](invariants.md#inv-122-a-same-head-repeated-e2e-gate-failure-an-inv-46-fail-verdict-against-an-unchanged-head_sha-e2e_lane_rc-fingerprint-gate_fail_stall_threshold-consecutive-rounds-is-detected-and-halted--the-breaker-transitions-reviewing--stalled-then-posts-one-structured-reasonsame-head-gate-failure-report-gated-on-an-already-stalled-skip-deliberately-not-the-dispatcher-side-may_stall_now-live-pid-pre-gate--see-rationale-below), #453).** Runs INSIDE the `gate == fail` branch, BEFORE the existing `pending-dev` routing. A REPEATED `fail` against an UNCHANGED `(head_sha, e2e_lane_rc)` fingerprint (≥`GATE_FAIL_STALL_THRESHOLD`, default 2) transitions `reviewing → stalled` instead of re-queuing — the fixed-point-repetition sibling of [INV-105]'s divergent-findings convergence breaker, for the case where the E2E gate itself never lets a review agent run at all. State is a `dispatcher-gate-fail-breaker` HTML-comment marker (unbounded full-history scan); the fingerprint resets on either a new commit OR a different `rc` on the same head, so an unrelated transient failure followed by a genuinely new bug never misclassifies as a stuck loop. **Deliberately does NOT call `may_stall_now`** (codex review round 2 [P1]): that shared INV-105 predicate's dispatch-marker-freshness check exists for the DISPATCHER to ask whether some EXTERNAL process might still be alive for this issue before mutating labels from outside; this breaker instead runs synchronously inside the very review wrapper the dispatcher just launched, so it would always see its own fresh `review`-mode dispatch marker and defer for the marker's full TTL (default 600s) — silently defeating the breaker for any E2E failure completing within that window, the common case. The `reviewing`-label single-writer invariant already provides the liveness guarantee `may_stall_now` exists to add. **Mention-target (issue #495):** the trip report calls `resolve_escalation_mention "$ISSUE_NUMBER" "$PR_NUMBER"` ([INV-138] — issue author first, PR author second, three-state operator fallback), not a bare `@${REPO_OWNER}` — per [dispatcher-flow.md's escalation-comment mention-target policy](dispatcher-flow.md#escalation-comment-mention-target-policy-issue-495).
- **Review agents are PURE code reviewers.** `build_review_prompt` no longer contains any E2E execution block; the prompt tells each agent to READ the wrapper-posted evidence comment as input and cross-check it against the acceptance criteria. They do not run E2E.
Expand Down
13 changes: 13 additions & 0 deletions docs/test-cases/pre-fanout-e2e-actionability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Pre-fan-out E2E actionability test cases

| ID | Scenario | Expected result |
|---|---|---|
| TC-E2E-ACT-001 | Browser and command lanes start | Both inherit the wrapper-private classification path. |
| TC-E2E-ACT-002 | Non-zero lane rc, sidecar absent | Classification remains `dev-actionable=true`. |
| TC-E2E-ACT-003 | Valid `dev-actionable=false` | Current-HEAD `e2e-failed` disposition and `failed-substantive dev-actionable=false` are required before pending-dev; dispatcher stalls with no DEV dispatch or pending-review bounce. |
| TC-E2E-ACT-004 | Valid `dev-actionable=true` | Existing bounded same-HEAD correction routing dispatches DEV. |
| TC-E2E-ACT-005 | Malformed, oversized, symlink, directory, or other unsafe sidecar | Classification fails closed to `false`. |
| TC-E2E-ACT-006 | Human-spoofed, stale, abbreviated, or wrong-issue disposition | Strict routing parser ignores it. |
| TC-E2E-ACT-007 | PR HEAD changes after disposition | New HEAD transitions to pending-review and no DEV is dispatched against stale evidence. |
| TC-E2E-ACT-008 | Required disposition or verdict write fails | No `reviewing -> pending-dev` transition occurs. |
| TC-E2E-ACT-009 | Existing INV-92, INV-98, #351, #453, and #540 suites | All remain green. |
34 changes: 26 additions & 8 deletions skills/autonomous-dispatcher/scripts/autonomous-review.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2039,6 +2039,10 @@ _AGENT_PGIDS_E2E=""
if [[ "${E2E_ACTIVE:-false}" == "true" ]]; then
_E2E_LANE_DIR=$(mktemp -d "/tmp/agent-review-e2e-${ISSUE_NUMBER}-XXXXXX")
_E2E_RC_FILE="${_E2E_LANE_DIR}/e2e.rc"
# Optional integration-owned classification for a non-zero lane result.
# Both command and browser lanes inherit this wrapper-private path. Producers
# should atomically rename a complete regular file into place.
export E2E_FAILURE_CLASSIFICATION_FILE="${_E2E_LANE_DIR}/e2e-failure-classification"
log "INV-46: running the E2E lane ONCE before the review fan-out (mode=${E2E_MODE})."
case "${E2E_MODE:-none}" in
command)
Expand Down Expand Up @@ -2135,6 +2139,11 @@ if [[ "${E2E_ACTIVE:-false}" == "true" ]]; then
fi
fi
E2E_GATE=$(_classify_e2e_gate "$_e2e_lane_rc" "$_e2e_evidence_present")
_e2e_dev_actionable="true"
if [[ "$_e2e_lane_rc" -ne 0 ]]; then
_e2e_dev_actionable=$(_e2e_failure_actionability \
"$_E2E_LANE_DIR" "$E2E_FAILURE_CLASSIFICATION_FILE")
fi
log "INV-46: E2E hard gate: lane_rc=${_e2e_lane_rc}, evidence_present=${_e2e_evidence_present} → gate=${E2E_GATE}"

# Capture the lane PGID for the reaper / SIGTERM trap (alongside fan-out PGIDs).
Expand Down Expand Up @@ -2299,19 +2308,28 @@ Findings->Decision Gate: 1 blocking finding(s) -- FAIL.
1. **[BLOCKING] E2E verification failed** — the wrapper ran the project E2E once before review (INV-46) and it did NOT pass (lane exit code ${_e2e_lane_rc}). See the E2E failure comment on PR #${PR_NUMBER}. The review agents were NOT run because a failing E2E is a hard gate. Fix the failure and push; the next review round re-runs E2E.$(declare -F run_footer >/dev/null 2>&1 && run_footer || true)

${_gf_marker}" 2>/dev/null || true
# INV-92 (#298): a failing E2E is a dev-actionable code defect (fail-open).
emit_verdict_trailer "$ISSUE_NUMBER" "$REPO" "failed-substantive" "" "true" 2>/dev/null || true

# INV-52: a failed E2E hard gate is a dev-actionable blocking FAIL — assert
# Persist strict same-HEAD routing evidence and the authoritative INV-92
# actionability trailer before pending-dev becomes observable. Either write
# failing aborts this route closed; cleanup must not fabricate a dispatchable
# pending-dev state without the evidence its consumer requires.
# INV-52: a failed E2E hard gate is a substantive blocking FAIL — assert
# it on the PR's GitHub-native state too (reviewDecision → CHANGES_REQUESTED),
# symmetric with the agent-findings and CONFLICTING substantive routes.
# Best-effort; the E2E `block-nonsubstantive` (evidence-missing) re-queue
# below deliberately does NOT request changes (transient, not a code defect).
submit_request_changes "$PR_NUMBER" \
"E2E verification failed (lane exit code ${_e2e_lane_rc}): the wrapper ran the project E2E once before review (INV-46) and it did NOT pass. See the E2E failure comment on PR #${PR_NUMBER}, fix the failure, and push — reviewDecision is set to CHANGES_REQUESTED until a new review with a passing E2E (INV-52)." \
|| log "WARNING: submit_request_changes returned non-zero (unexpected — helper is best-effort); continuing the FAIL route."

itp_transition_state "$ISSUE_NUMBER" "reviewing" "pending-dev" 2>/dev/null || true
"E2E verification failed (lane exit code ${_e2e_lane_rc}): the wrapper ran the project E2E once before review (INV-46) and it did NOT pass. See the E2E failure comment on PR #${PR_NUMBER}; reviewDecision is set to CHANGES_REQUESTED until a new review with a passing E2E (INV-52)." \
|| log "WARNING: submit_request_changes returned non-zero (best-effort); continuing the E2E FAIL route."

_e2e_route_rc=0
_review_route_e2e_failure \
"$ISSUE_NUMBER" "$PR_HEAD_SHA" "$_e2e_dev_actionable" \
|| _e2e_route_rc=$?
if [[ "$_e2e_route_rc" -ne 0 ]]; then
log "ERROR: required E2E failure route failed (rc=${_e2e_route_rc}); refusing to report a pending-dev transition."
RESULT_PARSED=true
exit 1
fi
log "Issue #${ISSUE_NUMBER} moved to pending-dev (E2E hard gate fail — no fan-out)."
RESULT_PARSED=true
exit 0
Expand Down
Loading