[prompt-clustering] Prompt Clustering Analysis - 2026-10-10

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-10 to 2026-10-09)
Total Tasks Analyzed: 559 copilot-agent PRs
Clusters Identified: 10
Overall Merge Rate: 84.4% (472 merged / 82 closed / 5 open)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 7 — general CLI / compiler / test-runtime improvements (232 PRs, 41.5% of all tasks). This is a broad catch-all spanning day-to-day feature and bugfix work on the core CLI and test suite.
• Highest Success Rate: Cluster 3 — firewall version bump / release pinning (100% merged, 13/13).
• Most Review-Intensive: Cluster 4 — safe-outputs/ledger validation work averages 5.1 reviews per PR, the highest of any cluster, reflecting the sensitivity of safe-output correctness.
• Outliers: 1 outlier cluster flagged (Cluster 9, task-level model routing) — merge rate 11.7 points below the repo-wide baseline with 22 PRs, clearing both the size (≥15) and gap (≥10 pts) thresholds.
• Note on scope: Workflow turn-count/duration/cost enrichment (Phase 2) was skipped this run — PR bodies in this repository are written as agent-authored change summaries rather than raw task prompts with a discoverable marker, and matching aw_info.json turn counts to these PRs would require a separate workflow-run correlation pass. Findings below rely on PR text, comments, and reviews only.

Cluster Analysis

Cluster 7: General CLI / Compiler / Test-Runtime Work

• Size: 232 tasks (41.5% of total)
• Success Rate: 84.9%
• Avg Comments / Reviews: 7.9 / 3.8
• Top Keywords: cli, coverage, setup, review, runtime, test, detection, path
• Characteristics: The largest and most heterogeneous cluster — everyday compiler, CLI, and test-infrastructure changes that don't cluster into a sharper theme on vocabulary alone.
• Example PRs: Preserve frontmatter formatting during workflow upgrades #59969, Add frontmatter formatting command #59971, Reject NUL bytes in Git paths and consolidate validation tests #60043, Upload activation run info as a GHE-compatible artifact #60057, Fix Dependabot configuration syntax #60122

---

Cluster 4: Safe-Outputs / Ledger / Output Validation

• Size: 67 tasks (12.0%)
• Success Rate: 82.1%
• Avg Comments / Reviews: 8.2 / 5.1 (highest review load of any cluster)
• Top Keywords: safe, output, safe output, ledger, outputs, target, safe outputs, validation
• Characteristics: Work on the safe-outputs write-sink and ledger subsystem — correctness-critical, draws heavy reviewer scrutiny.
• Example PRs: Prevent Jira smoke workflow failures when credentials are unavailable #60199, Validate external safe-output secrets before activation #60267, Add additive global footers to safe-output bodies #60275, Enforce add-labels target restrictions #60362, Require update-release for agent-generated release notes #60701

---

Cluster 8: MCP Tools / Web / Shell / Claude SDK Integration

• Size: 62 tasks (11.1%)
• Success Rate: 88.7%
• Avg Comments / Reviews: 8.3 / 4.8
• Top Keywords: tool, mcp, web, tools, prompt, shell, sdk, claude
• Characteristics: MCP server/tool wiring and engine SDK integration (Claude, shell, web tools). Second-highest success rate among sizable clusters.
• Example PRs: Fix static GitHub agent enclave startup #59878, Disable forcePublicRepos for enclave-only GitHub MCP backends #60016, Fix dangling determine-automatic-lockdown step reference in static enclave sink-visibility policy #60338, Prevent Claude tool results from triggering failure classifiers #60349, Prevent AI credits rate-limit false positives from MCP echoes #61425

---

Cluster 0: Daily AI Usage / Cost Accounting & Guardrails

• Size: 50 tasks (8.9%)
• Success Rate: 82.0%
• Avg Comments / Reviews: 8.1 / 3.6
• Top Keywords: usage, daily, accounting, aic, daily aic, guardrail, credits, ai
• Characteristics: Daily AI-credit accounting, usage guardrails, cost ledgers. Largest average file footprint (118.7 files/PR) — these tasks touch broad swaths of generated lock files.
• Example PRs: Restore Copilot usage accounting without the agent sandbox #60134, Fix daily AIC snapshot completeness and quota handling #60136, Prevent daily AIC guardrail blocks after pre-agent failures #60355, Fix daily AI Credits gate accounting fallback #60371, Preserve zero-credit threat detection accounting #60380

---

Cluster 2: Model / Provider Pricing & Auto-Routing Config

• Size: 36 tasks (6.4%)
• Success Rate: 80.6%
• Avg Comments / Reviews: 5.5 / 3.0
• Top Keywords: model, pi, codex, pricing, gpt, auto, models, provider
• Characteristics: Provider/model pricing tables and auto-selection config (codex, gpt, pi engines) — adjacent to but distinct from Cluster 9's routing logic.
• Example PRs: Route Grok 4.5 and 4.6 through the Responses API #60139, Fix PR Sous Chef threat-detection model #60321, fix(codex): strip openai/ provider prefix from Codex model identifiers #60423, Fix Issue Monster’s unsupported Pi model #60461, Fix Issue Monster model compatibility with Pi #60561

---

Cluster 5: Work Queue / Dispatch / Coordinator

• Size: 31 tasks (5.5%)
• Success Rate: 80.6%
• Avg Comments / Reviews: 7.8 / 5.1
• Top Keywords: queue, work, work queue, claim, dispatch, coordinator, replay, ledger
• Characteristics: Work-queue claim/dispatch/replay semantics for multi-agent coordination.
• Example PRs: Add deterministic work-pool ledger reducer #64787, Remove work-pool ledger support #64850, Define deterministic Dispatch Coordinator Claim arbitration #64851, Dispatch Work Coordinator — implementation pending #64853, Add deterministic dispatch coordinator replay core #65030

---

Cluster 6: Logs / JSONL Caching / Multi-Target Runs

• Size: 31 tasks (5.5%)
• Success Rate: 90.3% (second-highest)
• Avg Comments / Reviews: 8.1 / 4.8
• Top Keywords: logs, jsonl, cached, cache, target, run, count, multi
• Characteristics: gh aw logs caching, checkpointing, and multi-target run collection. Small average file footprint (9.3 files/PR) — tightly scoped changes correlate with high success.
• Example PRs: Fix local cross-repository workflow resolution in gh aw logs #59859, Preserve partial logs before command timeouts #59964, Cache workflow run discovery payloads in logs JSONL #59988, Load cached logs JSONL once per command #60040, Apply logs timeout and count globally across concurrent targets #60058

---

Cluster 9: Task-Level Model Routing — ⚠️ OUTLIER

• Size: 22 tasks (3.9%)
• Success Rate: 72.7% (11.7 points below the 84.4% baseline)
• Avg Comments / Reviews: 7.7 / 5.0
• Top Keywords: routing, model, model routing, task, awf, router, endpoint, effort
• Characteristics: Task-level model/engine routing (classifier-driven model selection, OTEL routing attribution, routing contract tests). See root-cause section below.
• Example PRs: Support AWF task-level model routing in the Copilot engine #64403, Support AWF task-level model routing in the Copilot engine #64593, Accept router-selected reasoning efforts none and max in Copilot model routing #64840, Route model classification from workflow task only #64841, Route model classification from workflow task text #64859

---

Cluster 1: Failing GitHub Actions Job Triage

• Size: 15 tasks (2.7%)
• Success Rate: 86.7%
• Avg Comments / Reviews: 2.2 / 1.3 (lowest interaction — fast, narrow fixes)
• Top Keywords: actions job, failing actions, actions, job, failing, wip failing, wip, integration
• Characteristics: Narrowly-scoped CI-failure triage PRs (avg 2.0 files changed) — small, fast, low-friction.
• Example PRs: [WIP] Fix failing GitHub Actions job 'build' #60289, [WIP] Fix failing GitHub Actions job lint-go-golangci #60316, [WIP] Fix failing GitHub Actions job #60645, [WIP] Fix failing GitHub Actions job Integration: CMD Tests #60683, [WIP] Fix failing GitHub Actions job for CLI update command #60849

---

Cluster 3: Firewall Version Bump / Release Pinning

• Size: 13 tasks (2.3%)
• Success Rate: 100% (13/13 merged)
• Avg Comments / Reviews: 9.1 / 3.8
• Top Keywords: v0, v0 28, firewall, bump, version, release, pins
• Characteristics: Routine firewall/dependency version bumps (e.g., v0.28 pins). Largest average file footprint (290 files/PR, mostly regenerated lock files) but perfectly mechanical and always merges.
• Example PRs: Bump gh-aw-firewall to v0.28.15 #60020, Bump gh-aw-firewall to v0.28.16 and gh-aw-mcpg to v0.4.21 #60568, Bump gh-aw-firewall (AWF) to v0.28.17 #60945, Bump gh-aw-firewall to v0.28.20 #61527, Bump MCP Gateway to v0.4.25 #61661

Success Rate by Cluster

Cluster	Theme	Tasks	Share	Success Rate	Avg Comments	Avg Reviews
3	Firewall version bump / release pinning	13	2.3%	100.0%	9.1	3.8
6	Logs / jsonl caching / multi-target runs	31	5.5%	90.3%	8.1	4.8
8	MCP tools / web / shell / Claude SDK	62	11.1%	88.7%	8.3	4.8
1	Failing GitHub Actions job triage	15	2.7%	86.7%	2.2	1.3
7	General CLI / compiler / test-runtime	232	41.5%	84.9%	7.9	3.8
0	Daily AI usage / cost accounting	50	8.9%	82.0%	8.1	3.6
4	Safe-outputs / ledger / validation	67	12.0%	82.1%	8.2	5.1
2	Model/provider pricing & auto-routing config	36	6.4%	80.6%	5.5	3.0
5	Work queue / dispatch / coordinator	31	5.5%	80.6%	7.8	5.1
9	Task-level model routing (outlier)	22	3.9%	72.7%	7.7	5.0

Overall baseline: 559 tasks, 84.4% merge rate.

Key Findings

1. One dominant task category: 41.5% of all agent tasks fall into the general CLI/compiler/test-runtime catch-all (Cluster 7). Its vocabulary didn't separate further even after custom stop-word tuning, suggesting day-to-day maintenance work is genuinely heterogeneous rather than under-clustered.
2. Mechanical, narrowly-scoped tasks succeed almost universally: the two highest success-rate clusters — firewall version bumps (100%) and CI-failure triage (86.7%) — both have small, well-defined scopes and (for CI triage) the lowest comment/review counts in the dataset, indicating fast, low-friction review cycles.
3. Model-routing work is the clear outlier: Cluster 9 sits 11.7 points below baseline and is the only cluster meeting both outlier thresholds (≥15 PRs, ≥10-point gap).

Lowest-Merge Cluster Root Cause

• Cluster: 9 — Task-level model routing (classifier-driven model/engine selection, OTEL routing attribution, routing contract tests)
• Merge Rate Gap: 72.7% vs 84.4% overall (gap: 11.7 points)
• Primary Blocker: Scope/complexity mismatch from cross-PR dependency chains
• Confidence: Medium (n=6 non-merged PRs is a small sample, but the pattern is consistent)
• Evidence PRs: Export sparse model-routing attributes to OpenTelemetry #67240, Add golden coverage for routed model audit analysis #67253, Route model classification from workflow task only #64841, Keep harness routing outcomes advisory to model attribution #67280, Route model classification from workflow task text #64859, Support AWF task-level model routing in the Copilot engine #64403

Of the 6 non-merged PRs in this cluster, blockers break down as:

Blocker Category	Count	Share	Notes
Scope/complexity mismatch	3	50%	#67240 and #67253 both required repeated make merge-main cycles because they depended on sibling routing PRs (#67092, #67254, #67279) landing first; #67240 was also asked to shed unrelated work-queue changes mid-review. #64841 was closed after reviewer approvals once the design direction ("is this statically known? the compiler can conditionally generate the prompt") shifted.
Review friction (security/correctness)	2	33%	#67280 received an explicit CHANGES_REQUESTED over a trust-boundary gap ("agent-writable metadata can still establish trusted routing"). #64859 was flagged for an embedded multi-layer JS script bypassing standard CJS validation.
Other/unknown	1	17%	#64403 closed with zero comments or reviews — no visible interaction trail, likely superseded or abandoned before review began.

For comparison, the 16 merged PRs in this cluster averaged similar comment volume (3–14 comments) but far fewer merge-main round-trips reported in-thread, suggesting the non-merged PRs specifically clustered around the routing subsystem's busiest dependency window (classifier cost attribution, OTEL attribution, and contract-test goldens landing in close succession) rather than being intrinsically harder tasks.

Recommendations

Based on clustering analysis:

1. Sequence model-routing work explicitly: given 3 of 6 blocked PRs in Cluster 9 stalled on sibling-PR dependencies, stage routing-subsystem changes (classifier attribution → contract tests → OTEL export) as an explicit merge train rather than parallel PRs, to cut repeated rebase/merge-main cycles.
2. Keep CI-triage and version-bump tasks narrowly scoped: Clusters 1 and 3 show that small, single-purpose PRs merge almost without friction — continue issuing these as standalone tasks rather than bundling them into larger feature PRs.
3. Add a security/trust-boundary checklist for routing and attribution PRs: both review-friction cases in Cluster 9 (Keep harness routing outcomes advisory to model attribution #67280, Route model classification from workflow task text #64859) involved agent-writable metadata or embedded-script validation gaps — a pre-submission checklist for "does this data cross a trust boundary" could catch these before review.
4. Outlier-cluster mitigation (Cluster 9): split future task-level model-routing prompts into (a) attribution/telemetry plumbing and (b) trust-boundary/security-sensitive routing logic, so the latter gets a security-focused review pass up front instead of late-stage CHANGES_REQUESTED.

---

Generated by Prompt Clustering Analysis (Run: 38042617903)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 115.3 AIC · ⊞ 6.8K · ◷

• expires on Oct 11, 2026, 2:03 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #67692.