Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,15 @@ acquire diff --> parse supported lockfiles --> parse + index --> bounded evidenc
- `review.rs`: orchestration; enforces acquisition, model-aware context, request,
provider-attempt, output-token, and worst-case token-exposure budgets before calls;
one UTF-8 byte counts as one projected token rather than using an optimistic ratio;
reviews every evidence batch outside hosted inference. Hosted inference uses a
reviews every evidence batch outside hosted inference. A diff that materializes more
than 24 source batches enters a deterministic large-review route on every surface,
including `--diff-file`: at most 24 requests, at most four concurrent requests, and
an exact hunk receipt committed by SHA-256 before provider contact. Security,
authorization, configuration, policy, billing, migration, release-control, and
executable vendor hunks require direct source evidence. Exact-evidence summaries are
limited to supported dependency metadata, provenance-bound generated output, and
low-risk non-security churn. Missing or invalid receipt coverage fails closed.
Smaller hosted reviews use a
bounded, schema-validated planner over deterministic candidate digests, always
including boundary, high-risk, and global-synthesis evidence. A planner outage or
invalid response retains its complete usage and cost records, then falls back to the
Expand Down Expand Up @@ -194,9 +202,12 @@ additions (violations are `kind: contentPolicy`). Content policy is on by defaul
planner and scorer input, worst-case token exposure, and projected cost across
cascade or consensus before provider contact.
12. Every completed review envelope records source-batch coverage when batching runs.
Synthesis requests remain outside those counts. Bounded reviews expose selected and
total source-batch counts in compact output. Planner
fallback remains audit metadata and does not expose provider failure details to a PR.
Deterministic large reviews also record a plan hash and direct, semantic, and
unreviewed hunk counts. Every normalized hunk has exactly one disposition; evidence
identifiers bind the exact hunk digest, and any unreviewed hunk fails the gate.
Semantic coverage cannot resolve baseline findings. Bounded reviews expose selected
and total source-batch counts in compact output. Planner fallback remains audit
metadata and does not expose provider failure details to a PR.
13. Operational and provider virtual anchors expire after each run. Reviewable
PR-description and change-metadata anchors carry across unrelated incremental
reviews, and a same-head rerun with either anchor falls back to a full review.
Expand Down
7 changes: 4 additions & 3 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@
- `postil plan` deterministic config dry-run; `postil doctor`; exact-ref pre-push hook.
- Compact PR summaries with run links, retained policy-suppressed findings, and
provider-safe operational check text.
- Bounded large-change review with repeated manifests, format-specific directional
lockfile summaries, non-line change metadata, oversized-line segmentation, and
cross-batch synthesis.
- Bounded large-change review with deterministic hunk receipts, mandatory direct
coverage for security and control-plane changes, exact-evidence semantic summaries,
at most 24 requests, four-way concurrency, format-specific lockfile summaries,
oversized-line segmentation, and fail-closed incomplete coverage.
- `.coderabbit.yaml` translation for zero-cost migration.
- Model cascade + concurrent multi-model consensus over any OpenAI-compatible endpoint;
bounded retry with jittered backoff on transient provider errors.
Expand Down
21 changes: 20 additions & 1 deletion bench/src/harness.ts
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,13 @@ export const envelopeV1 = z.object({
selectedBatches: z.number().int().nonnegative(),
totalBatches: z.number().int().nonnegative(),
plannerFallback: z.boolean().default(false),
receipt: z.object({
planSha256: z.string().regex(/^[0-9a-f]{64}$/u),
totalHunks: z.number().int().nonnegative(),
directHunks: z.number().int().nonnegative(),
semanticHunks: z.number().int().nonnegative(),
unreviewedHunks: z.number().int().nonnegative(),
}).optional(),
}).optional(),
reviewAdmission: z.object({
providerAttempts: z.number().int().nonnegative(),
Expand Down Expand Up @@ -952,7 +959,19 @@ function evaluateEnvelope(c: BenchmarkCase, env: Envelope, exitCode: number | un
failures.push("bounded review did not complete a non-fallback planner selection");
}
const plannerUsage = env.modelUsage?.filter((usage) => usage.role === "reviewPlanner") ?? [];
if (plannerUsage.length !== 1) {
const receipt = env.reviewCoverage?.receipt;
if (receipt !== undefined) {
if (
receipt.totalHunks !==
receipt.directHunks + receipt.semanticHunks + receipt.unreviewedHunks ||
receipt.unreviewedHunks !== 0
) {
failures.push("deterministic bounded review receipt is incomplete");
}
if (plannerUsage.length !== 0) {
failures.push(`deterministic bounded review recorded ${plannerUsage.length} planner usage event(s)`);
}
} else if (plannerUsage.length !== 1) {
failures.push(`bounded review recorded ${plannerUsage.length} planner usage event(s), expected 1`);
}
}
Expand Down
15 changes: 14 additions & 1 deletion bench/src/live.ts
Original file line number Diff line number Diff line change
Expand Up @@ -789,7 +789,20 @@ export function boundedCoverageFailure(
if (coverage.plannerFallback) {
return "bounded review used planner fallback";
}
if (plannerUsage !== 1) {
if (coverage.receipt !== undefined) {
if (plannerUsage !== 0) {
return `deterministic bounded review recorded ${plannerUsage} planner usage event(s)`;
}
if (
coverage.receipt.totalHunks !==
coverage.receipt.directHunks +
coverage.receipt.semanticHunks +
coverage.receipt.unreviewedHunks ||
coverage.receipt.unreviewedHunks !== 0
) {
return "deterministic bounded review receipt is incomplete";
}
} else if (plannerUsage !== 1) {
return `bounded review recorded ${plannerUsage} planner usage event(s), expected 1`;
}
} else if (coverage.selectedBatches !== coverage.totalBatches || plannerUsage !== 0) {
Expand Down
9 changes: 8 additions & 1 deletion bench/src/livemodels-score.ts
Original file line number Diff line number Diff line change
Expand Up @@ -290,12 +290,19 @@ export function scoreLiveCase(args: {
const coverage = env.reviewCoverage;
const boundedCoverage = coverage?.mode === "bounded";
const plannerUsage = modelUsage.filter((entry) => entry.role === "reviewPlanner");
const deterministicReceiptValid = coverage?.receipt !== undefined &&
coverage.receipt.totalHunks ===
coverage.receipt.directHunks +
coverage.receipt.semanticHunks +
coverage.receipt.unreviewedHunks &&
coverage.receipt.unreviewedHunks === 0 && plannerUsage.length === 0;
const coverageValid = coverage !== undefined &&
coverage.totalBatches > 0 &&
coverage.selectedBatches > 0 &&
coverage.selectedBatches <= coverage.totalBatches &&
(coverage.mode === "bounded"
? coverage.selectedBatches < coverage.totalBatches && plannerUsage.length > 0
? coverage.selectedBatches < coverage.totalBatches &&
(deterministicReceiptValid || (coverage.receipt === undefined && plannerUsage.length > 0))
: coverage.selectedBatches === coverage.totalBatches && !coverage.plannerFallback && plannerUsage.length === 0) &&
(c.admission.expectedCoverage === undefined || coverage.mode === c.admission.expectedCoverage);
const usageValid =
Expand Down
2 changes: 1 addition & 1 deletion bench/src/scorer-eval.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ function boundedScorerFixture() {
const ordinaryFile = (ordinal: number) => {
const path = `src/ordinary/segment-${ordinal}.ts`;
const lines = Array.from(
{ length: 2_200 },
{ length: 200 },
(_, line) => ordinal === 0 && line === 0
? "+export const accessPermissionLabel = 'Account access'; // ordinary display copy"
: `+export const ordinary_${ordinal}_${line} = ${ordinal + line}; // ordinary source behavior`,
Expand Down
27 changes: 26 additions & 1 deletion bench/src/scorer-eval.ts
Original file line number Diff line number Diff line change
Expand Up @@ -856,6 +856,13 @@ export function reviewCoverageFailure(
selectedBatches?: unknown;
totalBatches?: unknown;
plannerFallback?: unknown;
receipt?: {
planSha256?: unknown;
totalHunks?: unknown;
directHunks?: unknown;
semanticHunks?: unknown;
unreviewedHunks?: unknown;
};
} | undefined;
if (coverage?.mode !== expected) {
return `review coverage mode ${String(coverage?.mode ?? "missing")} does not match ${expected}`;
Expand All @@ -879,7 +886,25 @@ export function reviewCoverageFailure(
if (coverage.plannerFallback !== false) {
return "bounded review did not complete a non-fallback planner selection";
}
if (plannerUsage !== 1) {
if (coverage.receipt !== undefined) {
const receipt = coverage.receipt;
if (
typeof receipt.planSha256 !== "string" ||
!/^[0-9a-f]{64}$/u.test(receipt.planSha256) ||
typeof receipt.totalHunks !== "number" ||
typeof receipt.directHunks !== "number" ||
typeof receipt.semanticHunks !== "number" ||
typeof receipt.unreviewedHunks !== "number" ||
receipt.totalHunks !==
receipt.directHunks + receipt.semanticHunks + receipt.unreviewedHunks ||
receipt.unreviewedHunks !== 0
) {
return "deterministic bounded review receipt is incomplete";
}
if (plannerUsage !== 0) {
return `deterministic bounded review recorded ${plannerUsage} planner usage event(s)`;
}
} else if (plannerUsage !== 1) {
return `bounded review recorded ${plannerUsage} planner usage event(s), expected 1`;
}
} else if (coverage.selectedBatches !== coverage.totalBatches || plannerUsage !== 0) {
Expand Down
Loading
Loading