Skip to content

CI performance dashboard #1182

Description

Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.

Headline (run 2026-10-12 04:17 UTC)

Window: 2026-10-11 04:17 → 2026-10-12 04:17. 23 PRs merged through the merge queue. Steady state: merge-queue and PR latency match the last profile. No new finding clears the filing bar. One human-filed issue, #1414, has a PR open (#1415). Fresh numbers are posted on it.

Sample: run counts cover the full 24h: 379 runs (260 pull_request, 92 merge_group across 4 workflows, 23 push, 1 schedule, 3 workflow_dispatch). Run-level detail covers the 300 runs listed (3-page cap), which reach back to 15:38. Job detail covers 85 runs: all 22 CI merge_group runs listed, all 17 push runs, all 3 dispatch runs, the nightly schedule run, 34 of 101 CI pull_request runs (every 3rd), and 1–2 runs each of fail-fast, janitor, Audit and Pin check. Also 15 merged-PR timelines and the ruleset. 0 job logs. About 120 REST requests. GraphQL is blocked here.

metric 24h
PRs merged 23 (23 CI merge_group runs/day; 22 sampled, all success; no re-runs)
Merge queue enqueue → merge p50 8.0 / p90 8.3 / max 8.7 min (n=15)
Evictions 0 in 15 timelines
Bursts Largest was 3 entries (02:49 and 23:48), below max_entries_to_build 5.
merge_group created → ci-ok After #1386: p50 7.3 / p90 7.9 (n=19, range 7.07–7.90). All 22: p50 7.33 / p90 7.98 / max 9.87 (the max is from 2 pre-#1386 runs at 17:50 that ended on e2e-build-windows). Last job before ci-ok: coverage 16, Gradle e2e leg 4, e2e-build-windows 2 (pre-#1386).
PR run created → ci-ok Full runs: p50 7.45 / p90 8.9 / max 9.2 (n=12). The last job was coverage in 11 of 12. 22 of 34 sampled PR runs were drafts (only ci-ok).
push to main created → ci-ok p50 3.43 / p90 3.82 (n=17). An e2e-full leg was last in 10 of 17. 1 failure: cargo stable lock-v1 toolchain install at 22:58, already noted last run.
Linux queue wait p50 / p90 / max Depot 0.02 / 0.02 / 0.18 min (n=2,645)
macOS queue wait p50 / p90 / max 0.12 / 0.15 / 0.25 (n=91, nightly and dispatch only). No macOS job ran on PR, merge_group or push.
Windows queue wait p50 / p90 / max 0.03 / 0.05 / 0.07 (n=89)
Job-min/day (Linux / Windows / macOS) ~4.0k / 0.33k / 0.19k. By event: mg 30.8 L + 4.4 W per run (708 + 101/day); CI PR 11.4 per run averaged over drafts (~139 runs/day → ~1.6k); push 29.1 per run (~580/day, of which full tier 14.2); nightly 298 / 67 / 54; workflow_dispatch 3 full-matrix runs by a maintainer validating janitor branches (ci-janitor/go-work-symlinked-tmp ×2, agent/fix-ci-windows-pnpm-shims), ~700 / 162 / 132 in total; Merge queue fail-fast 7.6 hosted-Linux min per mg run (~175/day). The rise from ~2.7k comes from the dispatches and from PR volume (~139 CI PR runs vs 104). Per-run cost is unchanged.

Critical path (unchanged): clippy is 1.3 min (clippy 21 s, then cargo check --all-targets 41 s, p90 45 s). coverage has needs: clippy and runs 5.4 min, of which "Run tests with coverage" is p50 323 s / p90 357 s (n=22).

Watched, not filed (below the bar or not enough evidence):

  • Shard coverage into 2 partitions: at most ~1.3–1.8 min off every merge and full PR run. Still unconfirmed, because the log tool can't reach the compile/CLI-test part of the step.
  • clippy → coverage serialization: ~1.3 min per merge and per full PR run. coverage reads clippy's reuse/pr_unchanged outputs, so a fix would move those outputs into a cheaper job. Not filed: the saving is under the 2-min bar and the change is M.
  • coverage "Warm the coverage cache" on push: ~26 Depot min/day, below the bar.
  • Merge queue fail-fast: ~175 hosted-Linux min/day, off the critical path, and free on this public repo.
  • CI: skip PR re-test after a clean main merge (#1373) #1383 (pr_unchanged) is still not observable: 0 of 34 sampled PR runs took that path. Across three profiles, 116 PR runs have been either full 29-job runs or 1-job drafts.

Merge-queue settings (ruleset 24668460, read 04:2x): unchanged. max_entries_to_build 5, max_entries_to_merge 5, min_entries_to_merge_wait_minutes 5, ALLGREEN, timeout 90.

History

run (UTC) merges/24h MQ enq→merge p50/p90 (min) mg CI success p50 PR push→ci-ok p50/p90 mac queue p50/p90 linux queue p50/p90 job-min/day L/W/M
2026-10-12 04:17 23 (23 mg runs; 15 timelines) 8.0 / 8.3 (n=15, max 8.7, 0 evictions) 7.3 (n=19 after #1386, p90 7.9) 7.45 / 8.9 (n=12 full runs, created→ci-ok) 0.12 / 0.15 (nightly + dispatch only) 0.02 / 0.02 (Depot, n=2,645) ~4.0k / 0.33k / 0.19k (incl. 3 full-matrix dispatches; ~3.3k / 0.17k / 0.05k without; mg 30.8 / 4.4 / 0 per run)
2026-10-12 00:16 21 (20 mg runs, 1 in flight; 15 timelines) 7.9 / 8.3 (n=15, max 8.7, 0 evictions) 7.3 (n=16 after #1386, p90 7.9) 7.37 / 7.4 (n=18 full runs, created→ci-ok) 0.12 / 0.17 (nightly only) 0.02 / 0.02 (Depot, n=2,075) ~2.7k / 0.16k / 0.05k (mg 30.2 / 4.7 / 0 per run)
2026-10-11 16:17 15 (weekend; 15 mg runs, 1 merge since 01:02) 10.3 / 18.9 (n=15, max 19.1: 17:31 burst of 8, entries 6–8 waited a 2nd build round) 9.65 (n=15, p90 9.81) 7.35 / 7.8 (n=10 full runs, created→ci-ok) 0.12 / 0.15 (nightly only) 0.02 / 0.02 (Depot) ~1.9k / 0.18k / 0.05k (mg 28.5 / 7.2 / 0 per run)
2026-10-11 20:16 16 (15 timelines; 9 merges after #1386) 8.7 / 10.6 (n=15); 7.9 / 8.7 after #1386 (n=9) 7.3 after #1386 (n=9, p90 7.98); 9.65 before (n=7) 7.37 / 7.4 (n=10 full runs, created→ci-ok) 0.12 / 0.15 (nightly only) 0.02 / 0.02 (Depot, n=1,668) ~2.1k / 0.16k / 0.05k (mg 30.6 / 5.9 / 0 per run)
2026-10-11 12:17 28 (weekend; 30 mg runs) 10.3 / 10.7 (n=14, max 18.8 burst) 9.6 (n=30, p90 9.95) 7.2 / 8.1 (n=25, created→ci-ok) 0.12 / 0.18 (nightly/dispatch only) 0.02 / 0.02 (Depot) ~6.0k / 0.8k / 0.5k (mg 29 / 7.5 / 0 per run)
2026-10-09 20:16 ≥99 (8 since 16:16) last 78 / 112 (n=6, Linux backlog) 79.6 (n=6, 77–81) 23.3 / 65.5 (n=12; 51–79 after 17:45) 0.87 / 4.08 5.9 / 18.3 mg (16.6 p50 at 19h) ~150k / ~35k / 13.6k (carried over; CI per-run unchanged: mg 312 / 69 / 60)
2026-10-09 16:16 ≥99 first 84 / 509 · last 76 / 94 (n=13, burst drain) 22.3 (n=30) 21.8 / 24.6 (n=23) 0.67 / 2.73 0.07 / 2.12 ~150k / ~35k / 13.6k (CI 131k / 24k / 13.6k)
2026-10-09 12:24 72 first 107 / 168 · last 69 / 107 (n=8, burst drain) 23.1 (22.6 since 08:17) 21.5 / 22.3 (n=11, after #1166) 0.1 / 0.4 0.0 / 3.9 ~150k / ~40k / 9.4k (CI 106k / 19k / 9.4k, compat carried over)
2026-10-09 08:17 69 first 65 / 82 · last 64 / 69 (burst) 23.4 (21.5 since 04:17) 31.7 / 34.2 (n=58) 0.2 / 1.8 0.1 / 1.4 ~180k / ~40k / 10.1k (Gradle compat not re-measured)
2026-10-09 04:16 ~66 first 100 / 380 · last 86 / 142 (47 since 21:35) 25 (22 since 21:35) 31 / 33 (n=7) 0.2 / 1.9 0.1 / 3.3 ~263k / ~89k / 13.4k (Gradle carried over)
2026-10-09 00:17 72 first 48 / 290 · last 26 / 78 39.1 (22.6 after #1143) 47.2 / 56.9 (31.7 / 33.8 now) 0.2 / 1.7 0.4 / 3.7 295k / 92k / 10k (all workflows, 24h)
2026-10-08 21:50 49 80 / 157 44.5 (23.9 post-#1133) 43.2 / 71.9 (29.0 / 30.3 post-#1133) 22.0 / 55.5 1.2 / 12.8 97.5k / 28.3k / 12.0k (7-day avg)
2026-10-08 19:36 43 83 / 177 42.6 (22–25 after #1133) 41.6 / 78.5 1.1 / 69.8 0.1 / 7.6 139k / 21k / 21k (CI only, 24h; L/M/W order)

Open ci-perf issues by ROI

ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).

ROI issue saving status
~1 #1248 (settings) merge-queue build concurrency ~7.3 min for each entry past the 5th in a burst of more than 5. No such burst in 72h; the largest this window was 3. human decision (commented 10-12 00:20, no new comment today)
~0.5 #1414 full tier (e2e-full, yarn-berry-full, cargo-vex-matrix-full) runs on every push to main ~285 Depot job-min/day (~580 rounded per job). 29 of 50 push jobs, 14.2 of 29.1 job-min per push run. filed by a human 10-12 03:41. PR #1415 is open, merge state blocked. Fresh numbers.

Closed since the last profile: none. #1173 (test windows/macos redundant build) was closed not planned by a human on 10-11 19:12, together with PR #1396. It will not be refiled.

Open ci-perf / ci-janitor PRs: #1415 (fixes #1414).

Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect

PR merged before → after
#1392 Windows flake in notifier dead-endpoint test (ci-janitor) 10-11 19:23 Reliability. 0 merge_group failures since.
#1386 type-check Windows in the queue, in parallel 10-11 18:21 mg created → ci-ok 9.65 → 7.3 min (n=7 → 19, p90 7.9). Enqueue → merge 10.3 → 8.0. Windows min per mg run 8.2 → 4.4. Delivered.
#1383 skip PR re-test after a clean main merge 10-11 18:01 Not yet observable. 0 of 116 PR runs sampled across three profiles took the pr_unchanged path.
#1387 re-download truncated e2e-bin archives (ci-janitor) 10-11 17:50 0 failed merge_group runs since (22/22 this window).
#1378 compile tests on Windows in the merge queue 10-10 11:08 Added coverage but +2.5 min to the critical path. Fixed by #1386.
#1376 dead Depot runner doesn't cancel a merge-queue run 10-10 13:37 0 runner losses.
#1355 reuse the merge-queue verdict on main, trim coverage/sbt 10-10 08:57 Push-to-main run 22+ → 3.43 min (n=17). Delivered.
#1375 lean per-package-manager gate 10-09 21:29 Job-min per mg run 312 / 69 / 60 → 30.8 / 4.4 / 0. PR → ci-ok 21.8 → 7.45. Delivered.
#1362 Depot Linux runners 10-09 20:53 Linux queue wait p90 18.3 / 2.1 → 0.02 min. Delivered.
#1284, #1275 compat path filters 10-09 No compatibility workflow ran in the window. Delivered.
#1286 retry rustup installs (ci-janitor) 10-09 19:35 1 toolchain-install failure (10-11 22:58, cargo stable lock-v1). Re-passed on the next push.

Method notes

  • GraphQL is blocked in this environment, so merge-queue state comes from REST timelines of merged PRs.
  • Jobs are fetched with filter=all.
  • Queue wait = job started_at − created_at. Skipped jobs are excluded.
  • Critical path = merge_group run created_at → ci-ok completed_at.
  • Daily job-min = per-run mean × 24h run count per event. 24h counts come from total_count per event.
  • get_job_logs returns at most the last 5,000 lines, even with a larger tail_lines.

Generated by Claude Code

Activity

  1. mikolalysenko commented on Oct 8, 2026

    @mikolalysenko
    CollaboratorAuthor

    [agent] There are now two open dashboard issues: this one and #1175, both titled "CI performance dashboard" and labeled ci-perf-dashboard. The body says it is maintained in place, so the profiler routine should update one issue rather than open a new one each run. I'm not closing either one, because I can't tell which issue the routine will update next. Whoever owns the profiler should close the copy it no longer updates.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions