feat: confidence score (0-5) + optional LLM judge

[lohitkolluri]

What

Every finding now carries a confidence score 0-5.

• Base mapping (deterministic, no LLM): registry/manifest/secrets/IaC detectors = 5, heuristic detectors (style, stale API, guidelines, graph, slop) = 3, vulnerabilities = 3 (manifest-only; reachability uplifts later), LLM-authored prose = 4.
• Optional LLM judge: when a BYOK LLM is configured + enabled for the repo, one batched call re-scores findings and stores confidence + judge_rationale. Best-effort — LLM failure keeps base confidence, never fails the review.
• Grounding filter: [confidence] drop_ungrounded = true drops findings with confidence <= 1.
• v18 migration: findings.confidence (INTEGER), findings.judge_rationale (TEXT).
• Dashboard: confidence badge cN colored by tier, judge rationale in tooltip.

Config

[confidence]
judge_tier1 = false   # also judge deterministic tier-1 findings
drop_ungrounded = false

Notes

• Detector Finding literals across bot + detectors now carry confidence/judge_rationale (serde-skipped when absent).
• Gates: fmt, clippy -D warnings, 219 lib tests pass, frontend build clean. postgres_smoke requires a live Postgres (env-only failure, not code).
• Branches off feat/review-comment-redesign (Phase 0, PR feat: baseline finding suppression + quality gates #23) since the judge runs after the baseline filter.

Follows the 7-phase plan (0→3→1→2→4→5→6).

[codasaurus-e0a6]

Codasaurus

Advisory draft — soft findings only. Codasaurus will not request changes; a maintainer still decides merge.

This PR updates 31 files across src, docs, svelte-dashboard. I spotted 5 things worth a quick look (not blockers). See the list below.

Findings

Severity	Finding
warning	Repeated code in src/bot/markdown.rs
warning	Unfinished TODO left in in src/bot/markdown.rs:1450
warning	Unfinished TODO left in in src/bot/markdown.rs:1463
warning	Unfinished TODO left in in src/bot/markdown.rs:1485
warning	Unfinished TODO left in in src/confidence.rs:25

Copy this into your AI coding agent

Paste this prompt into Cursor, Copilot, or another coding agent:

You are fixing Codasaurus review findings on a GitHub pull request.

## Rules
- Change only what is required to resolve the findings below.
- Do not refactor unrelated code or add features.
- Prefer the smallest correct diff.
- `tests/fixtures/golden/` and `golden/*/input.*` files intentionally contain sample secrets/TODOs for detector tests — do not "fix" those unless the PR is explicitly changing the fixture contract.
- Workspace crate imports in `tests/` (e.g. `use codasaurus::…`) are expected — do not add them to Cargo.toml.
- If a finding is a false positive, say so instead of changing code.
- After edits, summarize what you changed and why.

## Findings (priority order)
### 1. [warning] Unfinished TODO left in
- **Location:** `src/bot/markdown.rs`:1450
- **Detector:** `todo-leaks`
- **Issue:** A `TODO` / `FIXME` marker was committed. Finish the work or remove the marker before merge.
- **Fix:** Complete the implementation or remove the placeholder.

- **Evidence:** detector: "todo-leaks".into(),

### 2. [warning] Unfinished TODO left in
- **Location:** `src/bot/markdown.rs`:1463
- **Detector:** `todo-leaks`
- **Issue:** A `TODO` / `FIXME` marker was committed. Finish the work or remove the marker before merge.
- **Fix:** Complete the implementation or remove the placeholder.

- **Evidence:** let prompt = agen…[redacted] "Fix todos").expect("prompt");

### 3. [warning] Unfinished TODO left in
- **Location:** `src/bot/markdown.rs`:1485
- **Detector:** `todo-leaks`
- **Issue:** A `TODO` / `FIXME` marker was committed. Finish the work or remove the marker before merge.
- **Fix:** Complete the implementation or remove the placeholder.

- **Evidence:** detector: "todo-leaks".into(),

### 4. [warning] Unfinished TODO left in
- **Location:** `src/confidence.rs`:25
- **Detector:** `todo-leaks`
- **Issue:** A `TODO` / `FIXME` marker was committed. Finish the work or remove the marker before merge.
- **Fix:** Complete the implementation or remove the placeholder.

- **Evidence:** | "guidelines" | "todo-leaks" | "policy" => 3,

### 5. [warning] Repeated code
- **Location:** `src/bot/markdown.rs`
- **Detector:** `boilerplate`
- **Issue:** Repeated code blocks found (40 instances) — AI often generates repetitive code.
- **Fix:** Extract repeated blocks into reusable functions.


## PR context
Title: feat: confidence score (0-5) + optional LLM judge

Changes

Path	What changed
docs/codasaurus-toml.md	updated
src/bot/concern.rs	updated
src/bot/markdown.rs	updated
src/bot/policy.rs	updated
src/bot/provenance.rs	updated
src/bot/quality.rs	updated
src/bot/review/findings.rs	updated
src/bot/review/persist.rs	updated
…23 more

All files (31)

• docs/codasaurus-toml.md (modified)
• src/bot/concern.rs (modified)
• src/bot/markdown.rs (modified)
• src/bot/policy.rs (modified)
• src/bot/provenance.rs (modified)
• src/bot/quality.rs (modified)
• src/bot/review/findings.rs (modified)
• src/bot/review/persist.rs (modified)
• src/bot/review/pipeline.rs (modified)
• src/confidence.rs (added)
• src/config.rs (modified)
• src/db/migrations.rs (modified)
• src/db/models.rs (modified)
• src/detectors/graph.rs (modified)
• src/detectors/guidelines.rs (modified)
• src/detectors/hallucinated_imports.rs (modified)
• src/detectors/iac.rs (modified)
• src/detectors/license_drift.rs (modified)
• src/detectors/lockfile_drift.rs (modified)
• src/detectors/mod.rs (modified)
• src/detectors/phantom_deps.rs (modified)
• src/detectors/risky_patterns.rs (modified)
• src/detectors/security.rs (modified)
• src/detectors/slop.rs (modified)
• src/detectors/stale_api.rs (modified)
• src/detectors/style.rs (modified)
• src/detectors/vulnerabilities.rs (modified)
• src/gates.rs (modified)
• src/lib.rs (modified)
• src/llm/mod.rs (modified)
• svelte-dashboard/src/pages/app/ReviewDetail.svelte (modified)

[codasaurus-e0a6]

Context

Blast radius

Sensitive path: src/detectors/mod.rs (+1)

Blast details

High-sensitivity paths:

• src/detectors/mod.rs
• src/llm/mod.rs

Top imports in this PR:

• super:: (2 importers)

_{Bounded estimate from PR imports only.}

Related pull requests

• Review maturity: advisory gate, grounding, concerns, eval, learning guard #19: Review maturity: advisory gate, grounding, concerns, eval, learning guar (4 shared files)
• refactor: merge review overview into a single Findings table #22: refactor: merge review overview into a single Findings table (2 shared files)
• Copyable AI fix prompt + review comment polish #17: Copyable AI fix prompt + review comment polish

[codasaurus-e0a6]

Checks

Before merge

• No blocking findings
• Warning budget within limit
  • At most 0 warnings allowed. Clear the notes on the Files tab, or ask a teammate.
• PR has a title

Reviewer time ~35 min (4/5 · Complex · 31 files)

Commands

Type as plain text (GitHub Apps are not @-mentionable):

review · describe · summarize · improve · security · ask … · ignore <fp> · help

_{Full list: @codasaurus help}

[codasaurus-e0a6]

Codasaurus

Commit e2b3702: advisory notes only (soft findings). Codasaurus is not requesting changes — 2 inline comments.

[codasaurus-e0a6]

Note · Unfinished TODO left in

Do this: Complete the implementation or remove the placeholder.

Why it matters

A TODO / FIXME marker was committed. Finish the work or remove the marker before merge.

detector: todo-leaks
source: tier1
snippet: | "guidelines" | "todo-leaks" | "policy" => 3,

Dismiss / fix commands

fingerprint: a44b36868e15

@codasaurus ignore a44b36868e15 · 👎 to dismiss