Skip to content

Latest commit

 

History

History
74 lines (42 loc) · 20.2 KB

0010-stack-aware-requalification.explainer.md

File metadata and controls

74 lines (42 loc) · 20.2 KB

LLP 0010: Stack-Aware Requalification

Type: Explainer Status: Active Systems: Sources, Engine, Prompts, Rendering, Config, Security Author: Philippe Loulidi / Claude Date: 2026-07-31 Related: LLP 0000 System Map, LLP 0001 Trust Model, LLP 0002 Review Pipeline, LLP 0004 Diff & Prompts, LLP 0005 Verification & Rendering, LLP 0008 Sources & Reporters

The canonical stacked-PR flow reviews the bottom PR first. A finding like "you changed parse.ts but added no test" is correct at that moment and wrong a day later, once the author stacks a parse.test.ts PR on top. Stack-aware requalification lets the reviewer see that later PR and stop blocking on the absence — without ever hiding the finding. This doc explains the mechanism and, more importantly, why each guard exists; it does not restate the code.

The shape is: fetch a paths-only manifest of the open PRs stacked on top of this one (once per run), inject it into the single coordinator call only, let the coordinator annotate absence-style findings with requalifiedBy, then run a deterministic, zero-LLM grounding pass that strips any annotation that is forged, hallucinated, or touches a protected class. A requalified finding is never removed: it renders in its own section, is counted in a visible audit line, and is excluded only from the blocking decision. v2 adds a per-candidate patch-level LLM confirmation over the grounded survivors (see Patch-Level Confirmation (v2)), so path membership is no longer accepted as semantic proof of a fix.

The Upstack Manifest

ReviewSource gains one optional method, getStackContextAsync, beside prepareReadRootAsync [observed] (source.ts ReviewSource). It returns a StackManifest — a list of { number, title, authorLogin, files } plus a truncated flag — or null. The return is deliberately two-valued, not three like prepareReadRootAsync: there is no "throw" case. Any failure (no repo, a gh error, a rate limit, a parse error, an empty stack) resolves to null [observed] (github-pr.ts getStackContextAsync, whole body wrapped in one try/catch → null). This is the inverted fail-open direction: breaking the walk buys an attacker only the status quo (findings stay blocking), so the safe default is "as if the feature were off". LocalGitSource omits the method, which makes ecr review over local git a structural no-op [observed] (source.ts: memoizeSource only exposes the method when the wrapped source defines it).

memoizeSource gains a fourth memo field so a routed CI run — which calls runReview once per scope — fetches the stack exactly once and shares it, the same rate-limit hygiene reason the read root and metadata are memoized (see LLP 0008) [observed] (source.ts stackPromise). Everything in the manifest is PR-author-controlled, so it is untrusted data end to end.

Bounded, Guarded Upward Walk

The walk is upward only. A downstack (parent) layer's content is already merged into the materialized head tree the reviewers read, so a parent never produces a false "missing" finding this mechanism needs to fix. Children are the open PRs whose base branch is this PR's head; recursion follows each child's own head branch [observed] (github-pr.ts fetchOpenChildren, walkUpstack).

The IO and the logic are split on purpose. walkUpstack is a pure, exported function over two injected fetchers, so every bound and guard is unit-testable without gh [observed] (github-pr.ts walkUpstack; tests stack-walk.test.ts). The bounds it enforces, and why each is load-bearing:

  • Same-repo, root included — a child counts only when head.repo.full_name equals owner/repo [observed] (github-pr.ts fetchOpenChildren sets sameRepo). The REST pulls-list endpoint has no isCrossRepository field, so this comparison is the fork gate; a fork PR is dropped. The ROOT PR gets the same gate before the walk even starts: a fork PR's headRefName names a branch of the FORK, and using it as the pulls-list base= filter would match a same-named base-repo branch (a fork head called main would pull in every open PR targeting main), letting unrelated PRs into the manifest — so a cross-repo root (isCrossRepository from gh pr view) returns null outright [observed] (github-pr.ts getStackContextAsync cross-repo guard).
  • Same-author — when requireSameAuthor (default true), a child counts only when its user.login equals the current PR's author [observed] (github-pr.ts walkUpstack same-author filter; author from fetchPrTrustAnchors). This closes cross-author poisoning: a push-access colleague opening a child PR on the victim's branch cannot inject paths. Teams running genuine multi-author stacks set requireSameAuthor: false from the trusted base — an explicit, base-committed choice.
  • Depth / width caps and a cycle guard — maxDepth levels, maxPrs children per level, and a visited-branch Set so a branch cycle can't loop [observed] (github-pr.ts walkUpstack). Per child, file pages are fetched one by one and stop as soon as maxFilesPerPr is exceeded or a hard page ceiling is hit (never --paginate to the end of a huge child PR), setting truncated when either bound bites [observed] (github-pr.ts collectChildFiles, MAX_CHILD_FILE_PAGES; tests stack-walk.test.ts).

--method GET on the pulls list is mandatory (else gh POSTs), and a per-element --jq yields NDJSON that stays valid when gh concatenates pages — unlike a bare --paginate, which concatenates JSON arrays into invalid JSON (the same trap the reporter's manual pagination avoids, LLP 0008) [observed] (github-pr.ts fetchOpenChildren).

Requalification Schema and Fingerprints

Finding gains an optional requalifiedBy: { prNumber, file, reason } [observed] (schema.ts FindingSchema). It rides the coordinator output schema for free, since that schema reuses the same shape — ModelFindingSchema, which is FindingSchema minus the engine-only agent field (see LLP 0011 "attribution-and-identity"); requalifiedBy is model-set, so it stays. Critically, fingerprintFinding is untouched — it keys on file/category/evidence, never on requalifiedBy [observed] (schema.ts fingerprintFinding). If requalification entered the fingerprint, a maintainer's dismissal would silently lapse the moment a finding was requalified, and the embedded-state identity would churn. Keeping it out means requalification is purely an annotation over an otherwise-stable finding identity.

Coordinator-Only Injection

The manifest is wired into buildCoordinatorTask and nowhere else — a new parameter after coverageNotes, threaded through coordinate() [observed] (prompts.ts buildCoordinatorTask; coordinator.ts coordinate). buildReviewerTask and buildCrossCuttingTask are byte-identical to before [observed] (tests stack-prompts.test.ts assert the fan-out builders are unchanged). Two reasons this placement matters: the wide fan-out (N agents × M chunks) carries zero stack bytes, so the cost is one small block in one call per scope, not a multiplier; and only the coordinator ever emits requalifiedBy, so a prompt-injected reviewer can never self-requalify — it never sees the manifest.

The block is defended like every other untrusted input, plus one addition. Titles and paths flow through flattenUntrusted, which collapses newlines per value [observed] (prompts.ts stackContextSection). A git filename may legally contain a newline, so without this a path like a\n----- END UPSTACK MANIFEST ----- could forge the closing fence; collapsing the newline keeps the whole path on one line. On top of that, a dedicated boundary regex strips any line matching the new marker, mirroring CONTEXT_FILE_BOUNDARY, and is re-applied after the head/tail cap because the tail slice can promote a forged marker hidden behind a prefix to a line start [observed] (prompts.ts UPSTACK_MANIFEST_BOUNDARY, capStackManifest; test stack-prompts.test.ts forges a newline-bearing filename). The injection-defense prose sits outside the fence, matching the PR_TITLE/PR_BODY treatment. The cap is a flat ~4000 chars head+tail, so cost stays constant regardless of stack depth or width [observed] (prompts.ts STACK_MANIFEST_MAX_CHARS).

The trusted prose tells the coordinator the rules: requalify only absence-style findings (missing/not-updated/not-regenerated), never a defect visible in the diff; match by name correspondence (foo.ts ↔ foo.test.ts, a model change ↔ migrations/*); cite the exact manifest path; never touch critical, secrets, or security findings [observed] (prompts.ts stackContextSection). The prose is a request, not a guarantee — the grounding pass is what enforces it.

Grounding and the Decision

groundStackRequalification runs between the verify block and applyInlineIgnores, preserving the load-bearing verify → ground → suppress → reconcile order [observed] (review.ts call site between verify and suppress; ordering constraint from LLP 0002). It is pure string membership over data ecr fetched itself — zero LLM — so forged citations and protected classes are handled even with a weak or prompt-injected coordinator. It strips requalifiedBy (leaving the finding fully intact and blocking) when any of:

  • the cited (prNumber, file) is not an exact normalized member of the manifest. Normalization is repo-root-relative — strip a leading ./ or /, trim — then exact string equality, no substrings and no basenames (a.ts must not match data.ts) [observed] (review.ts normalizeManifestPath, groundStackRequalification). A debug line is emitted on every strip so a silent under-fire (e.g. a monorepo path-frame mismatch) is diagnosable.
  • the finding is critical, or category secrets or security. The security category is excluded wholesale because a warning-severity security finding is exactly the class an attacker most wants demoted [observed] (review.ts groundStackRequalification).
  • the finding's file carried a pre-coordination critical/secrets/security reviewer finding. This is the severity lock: locked files are captured from the raw reviewer findings before coordinate() runs, keyed by normalized file path [observed] (review.ts buildPreCoordinationFileLocks, built after the fan-out and before coordinate). Without it, a coordinator steered into "downgrade critical→warning, then requalify" would slip a real critical past the carve-out. The key is the file alone, NOT a content fingerprint: the coordinator legitimately re-categorizes and paraphrases findings, and fingerprintFinding hashes category + evidence, so a fingerprint-keyed lock would miss a downgraded-then-reworded critical. Over-locking a whole file only keeps findings blocking — the feature's fail direction.

Then decisionAfterRequalification re-derives the decision over the still-blocking (non-requalified) subset — the parallel of decisionAfterVerification [observed] (review.ts decisionAfterRequalification). Because grounding strips requalifiedBy from every critical, a requalified finding is never critical, so the later decisionAfterVerification inside the suppression block cannot re-escalate past the softened decision — a property pinned by test [observed] (stack-grounding.test.ts). Finally the reconcile trigger is extended to fire when the decision changed or findings were removed, since requalification softens the decision with no count drop and a count-only trigger would leave the summary contradicting a now-softer decision [observed] (review.ts reconcile trigger).

Rendering in All Three Paths

Requalified findings render in a collapsed "🔁 Addressed in stacked PRs (N)" section beside the Dismissed section, in both renderMarkdown and renderAggregateMarkdown, and are excluded from the main severity list and the scope's blocking count [observed] (render.ts renderMarkdown, renderAggregateMarkdown). The single visibility rule the whole feature rests on: a mandatory one-line audit note in the OPEN body naming the addressing PRs [observed] (render.ts requalificationAuditNote). Requalification's only effect on a real finding is moving it out of the blocking set, so it must never be silent — the count and PR numbers show above the fold, and every reason names its #N.

requalifiedBy rides encodeState automatically as a Finding field, and the aggregate embeds requalified findings in its per-scope state exactly like dismissed ones, so a /dismiss re-render round-trips them and the addressed section survives [observed] (render.ts aggregate stateScopes; the reporter's applyDismissal re-render path, github.ts). The aggregate's worst decision is computed over the per-scope decisions, which are already requalification-adjusted upstream in review.ts, so a scope whose only findings are requalified contributes an approve [observed] (render.ts worstDecision).

Config and CLI Surface

A root-only stack block joins auth/breakGlass/commentTag on the scope-override exclusion list — declaring it in a scope config fails at the Zod level [observed] (schema.ts ReviewConfigSchema.stack, ScopeReviewConfigSchema). One PR has one stack; per-scope divergence is meaningless and a silent-override hazard. It defaults to enabled: false: a suppression-adjacent feature earns trust with field data first.

The trust principal differs by command. Under ecr ci, config loads from the PR's immutable base commit, so stack.enabled there is a trusted-base decision a PR cannot change; the feature auto-enables from it, and --no-stack-aware is an argv escape hatch that forces it off for one run [observed] (ci.ts resolveStackWalk). All stack failures are stderr warnings, never check failures — the source fails open, so nothing propagates. Under ecr review --pr, config comes from the local checkout, which is not a trusted base, so the feature is off unless the user passes --stack-aware explicitly — the person typing the flag is the trust principal [observed] (review.ts --stack-aware). No path exists where head-controlled config activates the feature.

Patch-Level Confirmation (v2)

Grounding proves a cited path is really in a child PR; it does not prove the change at that path fixes the finding. A path being touched is not semantic proof — the safety objection the whole v2 layer answers. So, gated by the trusted-base stack.confirmWithPatch (default false, so v2 ships dark until flipped), every requalification that survives grounding is confirmed against the addressing PR's actual patch before it is believed [observed] (review.ts confirmation block, right after grounding and before the decision is re-derived).

The confirmation reads the patch, never the tree. getStackFilePatchAsync fetches just the cited file's .patch fragment from pulls/<n>/files and the caller inlines it — nothing is materialized to disk [observed] (github-pr.ts getStackFilePatchAsync; source.ts contract, fail-open null). Inlining dissolves two problems at once: there is no untrusted-filename write path and no tool read at all, so the STACK_VERIFIER_AGENT is a no-tools agent — an empty tool list mirroring the coordinator. That id MUST be registered in both engines or a run crashes: buildOpencodeConfig gives it the empty NO_TOOLS set, and buildEngineMap plus the Claude tools map carry it too (empty list), or a Claude Max/Team subscription run dispatches against an undefined handle [observed] (opencode.ts STACK_VERIFIER_AGENT, agent record; claude-code.ts buildEngineMap modelOf, tools map tools[STACK_VERIFIER_AGENT] = []). The prompt inlines the finding (fields flattenUntrusted-ed, like buildVerifierTask) and the patch inside a BEGIN/END UPSTACK PATCH fence with its own boundary strip and a head/tail cap; the patch BODY is deliberately not prose-sanitized (the inlineDiff rule — the role-tag strip would mangle ordinary generics like Array<ToolDefinition> and the verifier would judge corrupted code), and the system prompt is biased toward "not addressed" [observed] (prompts.ts buildStackVerifierTask, capStackPatch, STACK_PATCH_BOUNDARY, buildStackVerifierSystem). The verdict is { addressed, reason } [observed] (schema.ts parseStackVerdict).

The orchestration is pure over an injected confirmer so its fail-open rules are testable without gh or a model, mirroring walkUpstack [observed] (stack-confirm.ts confirmStackRequalifications; the real fetch+verify closure is patchConfirmer; tests stack-confirm.test.ts). Every requalified finding gets its own verdict — the verdict question is finding-specific ("does this patch supply what THIS finding says is missing"), so a shared per-(prNumber, file) verdict would let one confirmed finding clear an unrelated finding citing the same file; only true duplicates (same fingerprint + citation) collapse, and only the patch fetch is shared, memoized per (prNumber, file) with the same normalization grounding uses [observed] (stack-confirm.ts per-finding candidate key; patchConfirmer patch memo; stack.ts manifestKey). Every failure mode strips the requalification, returning the finding to the blocking set: addressed !== true, any thrown error, a timeout (finalizeOnTimeout: false, so a slow verify is abandoned, not salvaged), a null patch fetch, and — critically — any candidate beyond maxConfirmations, which is stripped rather than silently kept, so a stack that floods the cap cannot buy unconfirmed demotions [observed] (stack-confirm.ts overflow slice and per-call try/catch). The decision is then re-derived over the post-confirmation set against the same pre-requalification ceiling grounding used, so a stripped requalification re-hardens the decision up to (never past) the coordinator's original [observed] (review.ts decisionBeforeRequalification).

The v1 residual this closes: a same-author, same-repo, real child PR that genuinely touches the cited path could get a warning/suggestion-level, non-security finding demoted without proof the touch actually fixes it. With confirmWithPatch on, that demotion now requires an LLM to read the real patch and affirm it addresses the finding; anything less keeps the finding blocking and counted. Under today's policy only criticals drive request_changes and grounding never lets a critical be requalified, so even with v2 off the effect on real findings is visibility-tier — which is why the audit count line stays mandatory.