ClawSweeper does not review fresh openclaw/openclaw PRs: per-target exact-review cap of 1 (PR #348) starves new PRs

[yetval]

Summary

ClawSweeper is not reviewing freshly opened openclaw/openclaw PRs. The bot is alive and keeps re-reviewing older PRs, but brand-new PRs go unreviewed for hours. Root cause: PR #348 introduced a per-target exact-review concurrency cap of 1, which serializes all openclaw/openclaw reviews onto a single slot held under a 130 minute lease. On the busiest repo, re-review and retry churn keeps that single slot occupied, so new PRs wait behind the entire backlog.

Impact

Observed on 2026-06-22:

PR	opened (UTC)	clawsweeper review comments at 22:03Z
openclaw/openclaw#95852	20:14	0 (open ~1h50m)
openclaw/openclaw#95860	20:56	0 (open ~1h7m)

About a dozen other PRs opened in the same window were also unreviewed (#95864, #95863, #95862, #95859, #95857, #95856, #95855, #95854, #95848, #95844, #95842), while the dispatch stream was dominated by repeat re-reviews of old PRs (#73079, #91502, #94272 each dispatched multiple times). This is throughput starvation of new PRs, not an outage and not an author block (the same author's prior PRs #95614 and #95432 were reviewed normally).

Root cause

Newly opened PRs are reviewed through the event-driven exact-review path: webhook to ExactReviewQueue (dashboard/worker.ts) to a per-item repository_dispatch. The admission loop in ExactReviewQueue.alarm() enforces a per-target cap:

// dashboard/worker.ts, alarm()
for (const item of pending) {            // pending sorted by createdAt ASC
  if (admitted.length >= slots) break;
  const active = activeTargets.get(item.decision.targetRepo) || 0;
  if (active >= DEFAULT_EXACT_REVIEW_TARGET_MAX_CONCURRENT) continue;   // cap
  ...
}

with (dashboard/worker.ts:74-77):

const DEFAULT_EXACT_REVIEW_QUEUE_MAX_CONCURRENT = 4;     // global
const DEFAULT_EXACT_REVIEW_TARGET_MAX_CONCURRENT = 1;    // per target repo
const DEFAULT_EXACT_REVIEW_EXECUTION_LEASE_MS = 130 * 60 * 1000;

DEFAULT_EXACT_REVIEW_TARGET_MAX_CONCURRENT = 1 and its enforcement line are new in PR #348 (fix(ci): bound ClawSweeper review fanout). The same commit also dropped the global queue cap 8 to 4 and WORKER_BUDGET 64 to 32:

git show f69d9de -- dashboard/worker.ts
-const DEFAULT_EXACT_REVIEW_QUEUE_MAX_CONCURRENT = 8;
+const DEFAULT_EXACT_REVIEW_QUEUE_MAX_CONCURRENT = 4;
+const DEFAULT_EXACT_REVIEW_TARGET_MAX_CONCURRENT = 1;
...
+      if (active >= DEFAULT_EXACT_REVIEW_TARGET_MAX_CONCURRENT) continue;

Because every openclaw/openclaw PR and issue shares the same targetRepo, the per-target cap means at most one openclaw exact review runs at a time regardless of free global slots. The slot is held up to 130 minutes and is constantly re-occupied by re-review and retry churn of older items, and pending items admit oldest-enqueued-first, so new PRs starve. Before #348 there was no per-target cap (up to 8 concurrent).

This is compounded by the scheduled lanes failing to rescue missed PRs: the broad sweep pages open items created asc and stops at shouldStopSaturatedPlanScan once the due backlog fills capacity (now much smaller after config/automation-limits.json cut workers 128 to 32, exact_review 32 to 4, and sweep.yml shard_count 89 to 22), so it re-reviews the oldest items and never reaches the newest. selectDueCandidates also prepends all weekly-overdue items first, leaving zero slots for the hot_pull_request bucket where fresh PRs live when the overdue backlog exceeds capacity.

Each piece is an intentional throttle, not an item-dropping bug, but together they make the time-to-first-review for a fresh openclaw PR far longer than its actionable window. The single highest-leverage line is the per-target cap.

Reproduction

Drop this test in test/ on current main and run node --test test/repro-fresh-pr-starvation.test.ts. It drives the real ExactReviewQueue:

import assert from "node:assert/strict";
import { generateKeyPairSync } from "node:crypto";
import test from "node:test";
import { ExactReviewQueue } from "../dashboard/worker.ts";

class MemoryDurableStorage {
  values = new Map(); alarmAt = null;
  async get(k) { return this.values.get(k); }
  async put(k, v) { this.values.set(k, v); }
  async delete(k) { this.values.delete(k); }
  async list() { return new Map(this.values); }
  async getAlarm() { return this.alarmAt; }
  async setAlarm(a) { this.alarmAt = a; }
  async deleteAlarm() { this.alarmAt = null; }
}
function enqueue(id, n, repo) {
  return new Request("https://q/enqueue", { method: "POST", body: JSON.stringify({
    delivery_id: id, decision: { targetRepo: repo, targetBranch: "main", itemNumber: n,
      itemKind: "pull_request", sourceEvent: "pull_request", sourceAction: "opened",
      supersedesInProgress: false } }) });
}
test("backlog of openclaw PRs only releases one review per cycle", async () => {
  const dispatched = [];
  const { privateKey } = generateKeyPairSync("rsa", { modulusLength: 2048,
    privateKeyEncoding: { type: "pkcs8", format: "pem" },
    publicKeyEncoding: { type: "spki", format: "pem" } });
  const orig = globalThis.fetch;
  globalThis.fetch = async (input, init) => {
    const u = new URL(String(input));
    if (u.pathname === "/repos/openclaw/clawsweeper/installation") return new Response(JSON.stringify({ id: 999 }));
    if (u.pathname === "/app/installations/999/access_tokens") return new Response(JSON.stringify({ token: "t" }));
    if (u.pathname === "/repos/openclaw/clawsweeper/dispatches") { dispatched.push(JSON.parse(String(init.body))); return new Response(null, { status: 204 }); }
    throw new Error("unexpected " + u);
  };
  try {
    const q = new ExactReviewQueue({ storage: new MemoryDurableStorage() },
      { CLAWSWEEPER_APP_CLIENT_ID: "Iv23", CLAWSWEEPER_APP_PRIVATE_KEY: privateKey,
        EXACT_REVIEW_QUEUE_MAX_CONCURRENT: "8" });  // pre-#348 global headroom
    for (const n of [95812, 95843, 95852, 95858, 95860]) await q.fetch(enqueue("d" + n, n, "openclaw/openclaw"));
    await q.fetch(enqueue("dg", 12, "openclaw/gogcli"));   // contrast target
    await q.alarm();
    const t = dispatched.map(p => p.client_payload.target_repo);
    assert.equal(t.filter(x => x === "openclaw/openclaw").length, 1);   // only 1 openclaw PR despite free slots
    assert.equal(t.filter(x => x === "openclaw/gogcli").length, 1);     // different target unaffected
    assert.ok(!dispatched.some(p => p.client_payload.item_number === 95860)); // fresh PR starved
  } finally { globalThis.fetch = orig; }
});

Result: with 8 global slots free and 5 openclaw PRs pending, only 1 openclaw PR dispatches per cycle (the fresh #95860 is left waiting), while a different-target item dispatches in the same cycle. A second alarm() with no completion dispatches no further openclaw review (the slot stays leased up to 130 minutes). This isolates the per-target cap, not global exhaustion, as the cause.

Suggested fix

1. Make DEFAULT_EXACT_REVIEW_TARGET_MAX_CONCURRENT configurable per target and set openclaw/openclaw to at least the global cap (smallest change, quickest mitigation).
2. Or exempt freshly opened PRs from the per-target cap (or give them a small reservation) so re-review churn of old items cannot crowd out new PRs.
3. Reserve a fresh-PR slice in the scheduled broad sweep so a saturated backlog still reaches the newest PRs (fetchOpenItemPage ordering plus shouldStopSaturatedPlanScan).
4. Reconsider the global cap 8 to 4 and worker budget 64 to 32 if API pressure allows, since the throttle was tuned for total API load rather than per-repo review latency.

Related

• Exact-review capacity queue times out before issue/PR reviews run #310 (exact-review capacity queue timeout) and github activity workflow can drop issue/PR events before ClawSweeper dispatch #311 (activity workflow dropping events) are distinct earlier mechanisms, both predate fix(ci): bound ClawSweeper review fanout #348.

[clawsweeper]

Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

Current main implements the central fix through #354: production now lets one target use all four exact-review slots, and the queue/test/docs paths cover that behavior. The broader worker-budget tuning in #350 is adjacent and not required to resolve the per-target cap starvation reported here.

I found the merged PR that appears to have closed this: #354: fix(queue): use available exact-review capacity per target.

So I’m closing this as already implemented rather than keeping a duplicate issue open.

Review details

Best possible solution:

Keep the merged current-main queue-capacity implementation from #354 and treat #350 as optional broader throughput tuning.

Do we have a high-confidence way to reproduce the issue?

Yes from source: with the prior production target cap of 1, same-target pending reviews were skipped while a single openclaw/openclaw lease was active. I did not run the provided repro because this is a read-only review, but current main now has focused regression coverage for the fixed path.

Is this the best way to solve the issue?

Yes: making the per-target exact-review cap configurable and setting production to the existing global cap is the narrowest maintainable fix for the reported idle-slot starvation without increasing global API pressure.

Security review:

Security review: No PR patch security review applies to this issue.

AGENTS.md: found and applied where relevant.

What I checked:

• Repository policy read: Target AGENTS.md was read fully; its automation guidance led this review to inspect queue code plus workflow/default tuning before closing. (AGENTS.md:1, 54eedf801125)
• Production capacity source: Production dashboard configuration now sets EXACT_REVIEW_QUEUE_MAX_CONCURRENT and EXACT_REVIEW_TARGET_MAX_CONCURRENT both to 4, so openclaw/openclaw can consume the existing four exact-review slots instead of being capped at one. (dashboard/wrangler.toml:42, 54eedf801125)
• Queue admission source: ExactReviewQueue.alarm computes targetCapacity from the environment and compares active work against that configured target capacity during admission, replacing the hard-coded target cap path introduced earlier. (dashboard/worker.ts:460, 54eedf801125)
• Queue wake source: The next-wake calculation also receives targetCapacity and wakes pending same-target work while capacity remains available, avoiding a stranded pending item when the target is below its configured cap. (dashboard/worker.ts:1266, 54eedf801125)
• Regression coverage source: The dashboard worker tests now cover dispatching four same-target exact reviews when target capacity is 4 and waking again while target capacity remains. (test/dashboard-worker.test.ts:403, 54eedf801125)
• Documentation source: docs/limits.md documents that production sets EXACT_REVIEW_TARGET_MAX_CONCURRENT to 4 so an OpenClaw backlog can drain without raising the global queue cap. (docs/limits.md:116, 54eedf801125)

Likely related people:

• vincentkoc: Authored the merged cap-introducing change and the merged queue-capacity fix, and recent history/blame shows ownership of the exact-review queue and dashboard defaults involved here. (role: recent area contributor; confidence: high; commits: f69d9debf487, 54eedf801125, ac8f2c5e61c2; files: dashboard/worker.ts, dashboard/wrangler.toml, test/dashboard-worker.test.ts)
• Peter Steinberger: Recent file history shows Peter authored the durable exact queue expansion path that the current ExactReviewQueue work builds on. (role: adjacent queue contributor; confidence: medium; commits: 30163dfd667b, dc824915bb6c; files: dashboard/worker.ts, dashboard/wrangler.toml)

Codex review notes: model internal, reasoning high; reviewed against 54eedf801125; fix evidence: merged PR #354, commit 54eedf801125, main fix timestamp 2026-06-23T11:27:37Z.

[yetval]

CC @vincentkoc as Code Owner of PR.

[yetval]

Resolved by #354 (merged 2026-06-23, commit 54eedf8), with #353 as a complementary self-heal fix.

#354 replaces PR #348's hard per-target exact-review cap of 1 with a configurable per-target capacity that defaults to the full queue capacity, so a single target (openclaw/openclaw) can now use all available exact-review slots instead of being throttled to one while the rest sit idle. It also fixes the alarm-wake scheduling so pending same-target items are retried while target capacity remains, closing the starvation tail.

On current main, DEFAULT_EXACT_REVIEW_TARGET_MAX_CONCURRENT now matches DEFAULT_EXACT_REVIEW_QUEUE_MAX_CONCURRENT, so there is no per-target throttle below the global cap. Closing as fixed.