TOOL-013: PR-Agent on NaN inference as a uniform reusable review workflow

Goal

Stand up PR-Agent (qodo-ai/pr-agent, MIT) as a reusable workflow in this repo, backed by NaN cluster inference, applied uniformly across all personal repos. Motivation: the free CodeRabbit tier gives line-by-line reviews only on public repos; three private repos (resume, biwenger-mvp, knowledge) get summaries only, and a merge burst has already exhausted its per-developer rate limit once.

Prior art: research-pr-review-agents.md (2026-06-11) surveyed 12 tools and ranked PR-Agent as the BYO-key fallback, rejecting it because "local-model quality drops below the bar". That objection evaluated BYO as "Ollama on homelab hardware". Cluster inference is a different quality tier, so the rejection no longer holds. PR-Agent itself is actively maintained (last commit 2026-08-06).

Where review quality actually comes from

Ranked by leverage, because the instinct is to reach for the prompt first and that is the smallest lever:

Context assembly — the dominant factor. A reviewer that sees only the changed hunk cannot find a cross-file bug; no prompt recovers information that never entered the window. PR-Agent's patch-extension and file-prioritisation logic is the part worth relying on rather than reimplementing.
Model class — a large reasoning model finds defects a small one cannot, at equal context.
Static analysis fed into the prompt — a large share of what reads as "AI quality" in CodeRabbit is linter and SAST output rendered as prose. This is mostly redundant here: ruff, mypy strict, BATS, hadolint, gitleaks and per-repo validators already run in CI. Our marginal gap is reasoning-level defects, not lint.
Prompt and output format — real, but it buys noise control, severity thresholds and formatting, not new findings.

Consequence for this ticket: invest in configuring context and models correctly; treat prompt tuning as the last pass, once real reviews exist to tune against.

Notable free win: config.repo_context_files defaults to ["AGENTS.md"], so each repo's behavioural SSOT enters the review prompt automatically. Standing orders, the English-only rule and the no-auto-merge rule become review criteria with no extra work.

Model configuration

Three models doing three different jobs, rather than one job three times:

[config]
model            = "deepseek-v4-flash"   # reasoning + long context for large diffs
fallback_models  = ["qwen3.6"]           # independent failure mode — see below
model_weak       = "gemma4"              # descriptions, commit summaries, cheap passes
response_language = "en-US"

Why not deepseek → mimo → qwen3.6. deepseek-v4-flash and mimo-v2.5 draw on the same metered monthly pool. Falling back from one to the other does not survive pool exhaustion — the failure mode the fallback exists for. qwen3.6 is documented as rate-limited only, which is the independent lane. mimo adds nothing here: it is omnimodal and the payload is text.

Quota is not a constraint at this volume. A 500-line diff is roughly 20K tokens; even at a hundred reviews a month with multiple passes this stays far below the pool.

Verify before pinning: NaN models are not in LiteLLM's registry, so custom_model_max_tokens must be set explicitly, and max_model_tokens (default 32000) raised if long-context review is wanted.

Uniform rollout, exclusion by path not by repo

One reusable workflow here, a short caller per repo, one shared config. Sensitive material is excluded by path glob, so the rule lives in a file rather than in memory:

[ignore]
glob = ["50_work/**", "10_projects/fae-brain/**", "10_projects/openkm-brain/**"]

Rationale: the diff leaves our infrastructure only at the model call, and path filters run before it. This is stricter than the CodeRabbit App, which holds standing read access to every repo in scope.

CodeRabbit: overlap, measure, then retire one

Do not drop CodeRabbit on day one. It costs nothing on public repos and carries the highest measured recall of the tools surveyed. The standing rule caps autonomous reviewers at two, so:

Run PR-Agent and CodeRabbit in parallel on public repos for a bounded window (suggest 3 weeks / ~10 PRs)
Record which tool found what, and how much noise each produced
Retire one on evidence, restoring the cap at two

Steps

Reusable workflow here (on: pull_request [opened, synchronize] + issue_comment for slash commands), pinned action tag, permissions pull-requests: write, contents: read
NAN_API_KEY as a secret; confirm it is never echoed in logs
Decide runner: GitHub-hosted (free minutes on public repos) vs the self-hosted runner for private repos
Shared .pr_agent.toml defaults, with the path-exclusion glob
Pilot on biwenger-mvp first — private, zero sensitivity, active PR flow
Validate empirically: open a PR with a known defect and confirm it is found. This is the real acceptance test and it does not depend on the model-reference refresh
Roll out to the remaining repos via the short caller
Only then: tune extra_instructions for noise and format

Acceptance

A PR in a pilot repo receives inline review comments generated through NaN, path exclusions demonstrably prevent excluded files from reaching the model, and a provider outage degrades the review rather than blocking the PR.

Notes

Runs in parallel with AI-001 (model reference refresh). It does not block: model behaviour here is verified by opening a test PR, which is faster than waiting on documentation.
knowledge has had 8 PRs in its lifetime and none since 2026-06-20 — it is not PR-driven, so the path glob covers a rare bulk-operation case rather than daily flow.

Reference

10_projects/dotfiles/research/research-pr-review-agents.md · 10_projects/ai-strategy/adversarial-review-2026-08.md §9.

Spec written ahead of implementation: specs/TOOL-013-pr-agent-nan-review/ — PR #789.

Implementation-grade detail lives there: model routing by role (and why deepseek → mimo is not a valid fallback — same metered pool), runner split with the fork-PR reasoning, scripted per-repo secret distribution, path-glob exclusion, and the bounded CodeRabbit overlap before retiring either.

Two open questions deliberately left unanswered in the spec: whether resume PII diffs should reach the cluster, and the missing NaN telemetry outside the hermes microVM.

This ticket stays in Backlog — the spec is preparation; pickup is the implementation PR.

docs(spec): PR-Agent on NaN inference as a uniform review workflow (#789

Measured the premise today instead of assuming it, because a run on the public repo reported Plan: Pro Plus and that could have undercut the whole ticket.

It does not. The premise holds. Across the last 8 merged PRs in each of the three private repos:

repo	private	CodeRabbit summaries	inline comments
`resume`	yes	0	0
`biwenger-mvp`	yes	0	0
`knowledge`	yes	0	0

Zero artifacts of any kind over 24 PRs. So whatever plan label appears on the public repo does not extend to the private ones, and the gap this ticket exists to close is real and total — not "summaries only" as the proposal states, but nothing at all.

One honest caveat before anyone treats this as settled: zero artifacts is consistent with two different causes — the tier genuinely excluding private repos, or the CodeRabbit app simply never having been installed on them. Same symptom, very different fix: the second costs one click and would shrink this ticket to almost nothing. Worth confirming in the CodeRabbit dashboard which repos are actually in scope before starting the 39 tasks.

Separately, a stronger argument for this ticket than the coverage gap. #906 fired live on PR #918 today:

CodeRabbit	pass	0		Review rate limited

And it went both ways within one hour on that same PR — the first push got a real review with three actionable findings, a later push got the rate-limited no-op, both reported as a green check. So the failure is per-push, and the merge gate cannot tell which one a given commit received.

That reframes the value here. A second reviewer is worth having less for review quality than for redundancy against a silent no-op: a reviewer that sometimes skips without saying so is worse than one that is merely less capable, because the first is indistinguishable from success. That argument applies to the public repos too, which the coverage argument does not — and it may justify a considerably smaller first slice than the full uniform rollout.

Rate limits: the real numbers, and what they imply for the design

The proposal states "quota is not a constraint at this volume" and reasons only about the
monthly token pool. That half holds. The other half was missing, and the vault snapshot
21-nan-cloud.md (verified June 2026) does not carry it: it lists RPM only for embeddings
(60), TTS (15) and STT (10), and nothing for /v1/chat/completions.

The live docs at nan.builders/docs do:

Limit	Value	Scope
Requests per minute	60	per key (confirmed by owner)
Concurrent requests	5	per key (confirmed by owner)
Token quota	500M/month per member	shared across models

Monthly volume is a non-issue — a 500-line diff is ~20-40K tokens, so the pool covers
roughly 15-25K reviews/month. Three orders of magnitude of headroom. Shape is the
binding constraint, not volume. 60 RPM does not bite at solo-dev cadence; 5 concurrent
does.

Consequence 1 — the limits are per key, so a dedicated CI key isolates the budget

With a per-key limit, the reviewer sharing the member key with the hermes gateway means
both draw on the same 5 concurrent slots — a review firing during a cron inference batch
is the plausible collision. Minting a separate key for CI removes that contention
entirely and reduces the concurrency cap's job to protecting against self-contention
(several PRs reviewed at once).

Open decision: confirm whether additional keys can be issued on the membership. If yes,
this is the preferred split and it should land before the pilot.

Consequence 2 — the mechanism is a concurrency cap, not a queue

A durable queue was considered and rejected. The argument against it is this ticket's own
finding from #906: a reviewer that silently no-ops is worse than a weaker one, because it
is indistinguishable from success. A queue that defers a review past the merge recreates
that failure inside our own tooling — and under the PR-stewardship rule, a review arriving
after disposition is worthless. The correct degradation under a rate limit is a loud
skip ("review skipped: rate limited", visible on the PR), never silent deferral.

Escalation ladder, each rung only if the previous one proves insufficient:

In-job retry with exponential backoff — PR-Agent rides LiteLLM, which already retries 429s
concurrency: group with cancel-in-progress: false — this is a queue, native to
Actions, no infrastructure. Promoted from optional to mandatory from day one. Honest
limitation: it serializes per repo, not across repos; acceptable at solo-dev volume
Loud skip + /review slash command as manual dequeue — zero infrastructure
Fallback model on an independent lane — see the caveat below
Only if measured pain persists: the hermes scheduler is the natural queue substrate, since
it already schedules authenticated inference jobs. Never a new broker in the workflow
layer — that is one more stateful service to operate, monitor and back up

Correction to the model-routing rationale

The proposal justifies fallback_models = ["qwen3.6"] on the grounds that "qwen3.6 is
documented as rate-limited only, which is the independent lane". That is not in the
reference doc. qwen3.6 and gemma4 carry no quota row at all, and the Quotas & Limits
section states the token quota is shared across all models. With per-key limits the
independent-lane reasoning does not hold for a fallback using the same key either.

This strengthens the gate already in tasks.md: do not pin model names or quota classes
until re-verified against AI-001 (mlorentedev/knowledge #150).

Secret distribution collapses to two lines

The "scripted per-repo secret distribution" task is already built. NAN_API_KEY is in
secrets/registry.yaml (plane: app, backend: bw, item nan-api-key in
Dotfiles/apps, 90d rotation), and dotf secrets sync ci already pushes a repo's ci:*
secrets to GitHub Actions (ADR-028/ADR-029). The change is adding ci:mlorentedev/<repo>
to the entry's consumers and running one command.

Not verified, and blocked. dotf secrets verify aborts before resolving anything:

Error: secret "GITHUB_PERSONAL_ACCESS_TOKEN": bw.folder "apps" is not in the ratified
taxonomy (Dotfiles/apps, Dotfiles/infra)

Scoping by id (dotf secrets verify NAN_API_KEY) fails identically — registry validation
runs at load time, before resolution. This contradicts the command's documented contract
("reports a per-var status", "ids/var-names scope it"): a health check that cannot report
health because one unrelated entry is malformed. Filed separately; it blocks this
ticket's secret-distribution step.

Key placement

A — GitHub Actions secret via dotf secrets sync ci. Already built, rotation already
registered. Cost: copies of the key across N repos.
B — in-cluster only, runner/webhook reaches the gateway, key never leaves. Natural for
the Gitea leg; gates the GitHub leg on infrastructure that does not exist yet, and public
fork PRs must not touch self-hosted runners (already noted in the spec).

Decision: A now for GitHub, B later for Gitea. When the forge exists, the Gitea leg
distributes no key at all.

Gitea leg — in scope, deferred, and gated

PR-Agent supports Gitea natively (official installation docs; provider fixes shipped
Dec 2025 and Jan 2026), so this needs no second tool.

Uniform policy, mechanism decided later. The concurrency cap applies identically to both
forges. Whether the mechanism is identical depends on the deployment shape: via Gitea
Actions it is the same concurrency: block; via the PR-Agent webhook server there is no
such primitive and the cap must live inside the service — which is also one more
long-running stateful service in the cluster, to be classified under ADR-061 before it is
built, not after.

Gated on mlorentedev/kubelab #1077 (IDP-034, forge sequencing) and informed by #504
(OPS-D015, Gitea Actions evaluation). Tracked in its own ticket.

Revised sequencing

Phase 0 — collapse scope before building: confirm in the CodeRabbit dashboard which
repos are actually in scope (the 0-artifacts-over-24-PRs finding is consistent with the
app never having been installed), and land AI-001 to refresh 21-nan-cloud.md
Phase 1 — pre-flight: smoke the endpoint, resolve the exact LiteLLM model string, set
custom_model_max_tokens, and measure the limit empirically — fire N requests until a
429 appears or does not, and determine whether PR-Agent issues its completions serially or
in parallel per command. If serially, one review occupies ~1 slot and 5-concurrent is far
less threatening
Phase 2 — GitHub, small slice: reusable workflow + biwenger-mvp pilot with a
deliberate defect, continue-on-error, and the concurrency: group from day one. Not the
uniform rollout — the CI: CodeRabbit's free-tier quota silently skips review on some PRs, and the merge gate cannot tell #906 redundancy argument applies to public repos too and justifies a
smaller first slice
Phase 3 — Gitea, once the forge is standing

Ticket stays in Backlog: this is preparation, pickup is the implementation PR.

feat(ci): make a green check mean reviewed, not merely un-failed (#1019)

docs(spec): disclose that TOOL-013's config preceded its spec (#786)

feat(ci): add PR-Agent on NaN inference, so review capacity stops gat…

fix(ci): stop PR-Agent cancelling its own review when a bot comments (#…

feat(ci): enable the inline suggestions this tool was adopted for (#786)

docs(spec): record the TOOL-013 follow-on found by its first live runs (

feat(ci): make the reviewer check harness compliance by default (#786)

feat(ci): enable the inline suggestions this tool was adopted for (#786)

docs(spec): record the TOOL-013 follow-on found by its first live runs (

fix(ci): make the push-commands guard fail when the feature goes inert (

feat(ci): make the reviewer check harness compliance by default (#786) (

feat(ci): review every push, because the doctrine already says we do (#…

feat(ci): stop reviewing release PRs, and stop demanding a review of …

fix(ci): add a same-tier fallback reviewer, measured to have its own …

fix(ci): cap the reviewer's inference demand, and queue instead of fa…

fix(ci): move PR-Agent off the retired mimo-v2.5 model (#472)

fix(ci): move PR-Agent off the retired mimo-v2.5 model (#413)

fix(ci): move PR-Agent off the retired mimo-v2.5 model (#89)

fix(ci): move PR-Agent off the retired mimo-v2.5 model (#124)

TOOL-013 archive review (FAIL, landed in #2324) is addressed by #2325: inline_key_issues = true (F1), comment drift (F2), vendor/** (F3), plus the attestation text and triage skill.

AC1 live evidence, still owed: PR-Agent loads its config from main, so #2325 is reviewed under the old config. The evidence is the first PR reviewed after #2325's merge sha. Expect a COMMENTED review by github-actions with inline comments, and a Guide that keeps only lines-0 findings (harness compliance) or has no findings row. Then rewrite verification.md AC1 from it and run dotf spec review TOOL-013-pr-agent-reviewer from a fresh worktree.

AC1 live evidence: inline delivery after #2325 (`7186bc0a`, merged 2026-10-11T02:25Z)

Every PR-Agent review since that merge, read from the REST API. Corrected: the first version of this comment left out #2326 and #2327, which were reviewed first.

PR	Review	Head	Inline comments	Findings kept in the summary
#2326	`COMMENTED` 02:30Z, review `5481615000`	`3df52387`	2	none
#2327	`COMMENTED` 02:34Z, review `5481624169`	`44e8809d`	1	none
#2329	`COMMENTED` 03:18Z	`e3086b93`	1	the DoD evidence finding (path: PR description)
#2332	`COMMENTED` 03:20Z	`47fa3db2`	1 (`system.go:206-208`, "Incomplete Fix")	the DoD evidence finding
#2333	none (summary comment only)	`fc0fa741`	0	its one finding has path "PR description" and no line
#2334	`COMMENTED` 03:41Z, review `5481759720`	`71df0865`	2 (`manifest_requires_test.go`, "Vacuous guard" and "Weak assertion")	none

Findings with a file and line arrive as inline comments under a COMMENTED review.
Findings without a line (path "PR description") stay in the summary.
That matches the owner's decision of 2026-10-11, recorded in the handoff: lines-0 findings stay in the summary.
fix(converge): re-run records-harness after tools installs an agent #2333 is the control case: no finding had a location, so no review object was created.

Next, per the spec, rewrite specs/TOOL-013-pr-agent-reviewer/verification.md AC1 from this evidence, then run dotf spec review TOOL-013-pr-agent-reviewer.

AC1 live evidence: the first PR reviewed after 7186bc0 carries inline findings. On #2359, review 5498320909 github-actions[bot] COMMENTED 7ef4b35 2026-10-13T03:10:11Z (id, author, state, head, submitted) is a COMMENTED review by github-actions with 3 inline comments: cli/internal/env/roles.go:61, cli/internal/cmd/env_role_test.go:9 and specs/PLAT-001b-one-entrypoint/verification.md:94. The summary comment keeps the ticket-compliance table. Read with gh api repos/mlorentedev/dotfiles/pulls/2359/reviews and .../pulls/2359/comments. Next: rewrite AC1 in TOOL-013 verification.md, then dotf spec review TOOL-013-pr-agent-reviewer.

Navigation Menu

Repository navigation

TOOL-013: PR-Agent on NaN inference as a uniform reusable review workflow #786

Goal

Where review quality actually comes from

Model configuration

Uniform rollout, exclusion by path not by repo

CodeRabbit: overlap, measure, then retire one

Steps

Acceptance

Notes

Reference

Rate limits: the real numbers, and what they imply for the design

Consequence 1 — the limits are per key, so a dedicated CI key isolates the budget

Consequence 2 — the mechanism is a concurrency cap, not a queue

Correction to the model-routing rationale

Secret distribution collapses to two lines

Key placement

Gitea leg — in scope, deferred, and gated

Revised sequencing

AC1 live evidence: inline delivery after #2325 (`7186bc0a`, merged 2026-10-11T02:25Z)

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

TOOL-013: PR-Agent on NaN inference as a uniform reusable review workflow #786

Description

Goal

Where review quality actually comes from

Model configuration

Uniform rollout, exclusion by path not by repo

CodeRabbit: overlap, measure, then retire one

Steps

Acceptance

Notes

Reference

Activity

mlorentedev commented on Aug 7, 2026

mlorentedev commented on Aug 11, 2026

mlorentedev commented on Aug 15, 2026

Rate limits: the real numbers, and what they imply for the design

Consequence 1 — the limits are per key, so a dedicated CI key isolates the budget

Consequence 2 — the mechanism is a concurrency cap, not a queue

Correction to the model-routing rationale

Secret distribution collapses to two lines

Key placement

Gitea leg — in scope, deferred, and gated

Revised sequencing

mlorentedev commented on Oct 11, 2026

mlorentedev commented on Oct 11, 2026

AC1 live evidence: inline delivery after #2325 (7186bc0a, merged 2026-10-11T02:25Z)

mlorentedev commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

AC1 live evidence: inline delivery after #2325 (`7186bc0a`, merged 2026-10-11T02:25Z)