CI perf: e2e — 11 legs run under 20 s of tests each on their own runner, 82% overhead (~360 Linux job-min/day, 600 Depot starts/day) #1732

Measurement

Window: 2026-10-25 20:16 → 2026-10-26 20:16 UTC, jobs (filter=all) of all 108 CI runs listed in the window (47 merge_group, 47 PR, 13 push, 1 nightly). 75 runs ran the e2e matrix (18 legs per run).

The e2e matrix has 11 legs whose test step takes under 20 s (median). Each one still gets its own Depot runner, checkout, binary download and toolchain setup:

Totals over 24 h for these 11 legs: 825 jobs, 701 Linux job-min, of which 125 min (18%) is test time. 282 min is Depot runner start (runner start → first step), and the rest is set-up/checkout/download/toolchain.

Where the time goes

Per tiny leg: Set up job ~5 s, Checkout ~6–8 s, Download/Unpack the e2e binaries ~3 s, toolchain setup 1–9 s, test 0.4–18 s. Under load, the runner start dominates: the share of Depot jobs that took more than 15 s from runner start to first step was 0% in every hour with ≤130 jobs, and 31–53% in the 15:00–18:59 burst hours (439/1414, 169/352, 125/221, 74/197). mg runs with ≤6/31 slow jobs reached ci-ok in 4.48–5.25 min. Runs with 23–31/31 slow jobs took 6.4–12.8 min (run, run).

Root cause

#1178/#1234 merged suites that share a toolchain. The legs that are left differ only in which toolchain they install. The setup steps are already gated per matrix field (matrix.poetry, matrix.pdm, matrix.deno, …), and each field exports its own SOCKET_PATCH_*_E2E_REQUIRED / _VERSION. So one row can set several fields at once. The five e2e_vex_build rows run the same binary with different libtest filters, and libtest accepts several filters in one call.

Proposed fix

In .github/workflows/ci.yml e2e matrix (rows near lines 1222–1416 on main):

1. One VEX leg: replace the poetry / pdm / hatch / pipenv / deno e2e_vex_build rows with one row: test_filter: 'poetry:: pdm:: hatch:: pipenv:: deno:: --ignored', plus all five tool fields. Leave pip:: (2.4 min) on its own. Check that the Pipenv Setup uv / Install the Pythons steps don't conflict with the PDM/Hatch Install uv step. If they do, keep pipenv separate (4 legs → 1 instead of 5 → 1).
2. One JS leg: bun + pnpm + vlt rows → one row. The run step already loops over suite. vlt has its own Run vlt e2e tests step, which still runs when matrix.vlt is set.
3. One misc leg: uv + composer + dotnet → one row (suite: e2e_redirect_uv_build e2e_vendor_composer_build e2e_redirect_composer_build e2e_nuget_dotnet_build). This works when every suite uses the default --ignored filter.
   The per-row "ran no test" guard (line ~1859) must still fire for each suite/filter. Make the loop check each suite's own output, not the combined log.

Expected saving

• 8 fewer runners per run (18 → 10 e2e legs, 31 → 23 jobs per run): about 600 fewer Depot job starts per day at today's 75 runs.
• ~360 Linux job-min/day (8 legs × ~0.6 min fixed overhead × 75 runs). More under load, when runner start alone averages 0.34 min per tiny leg.
• Critical path: the combined legs take about 1.0–1.5 min (summed tests + setups). The Gradle redirect leg (p50 3.07, p90 5.26) stays the longest e2e leg, so there is no direct loss. Slow starts track load, so 26% fewer runner requests in a burst may lower the slow-start rate that pushed mg ci-ok from 4.5 to 6.5–12.8 min. That part is unproven (confidence 0.3).

Coverage and risk

Every test still runs in the same events (PR + merge_group); only the grouping changes. A failure now names a combined leg, so the run log has to show which tool failed (use a ::group:: per suite/filter). Required checks ci-ok / clippy don't change. Matrix-leg names aren't required checks. Risk: toolchain setup steps that conflict on PATH (uv versions, Node 24 for vlt vs pnpm's Node). Keep a conflicting tool in its own leg rather than forcing it.

Effort

M (matrix rework + run-step loop over suite/filter pairs; validate in one PR run).

ROI

saving ≈ 0.36k Linux job-min/day (×1) + ~1 min critical path when loaded (×0.3 confidence) → (0.36 + 0.3) × 0.6 confidence ÷ 2 ≈ 0.2

Generated by Claude Code