[kerdofficial]

Vectis CI was unstable and slow in several compounding ways. Runners were booked per job while GitHub assigns jobs by label, so runners serving other jobs burned a job's three attempts. Lost webhooks were never repaired. action_required runners never healed and held capacity, one transient GitHub error failed a runner, and offline listeners kept VMs for hours. The cloud ignored the macOS two-VM limit, and every machine lost its cloud connection every five minutes because the token refresh was never scheduled. Status responses were unbounded (1.5 million characters through MCP). Every job also downloaded the runner, and Windows jobs were several times slower than Linux, mostly because of Defender and bsdtar.

This PR combines the night's work (replacing #103-#108) in logical commits.

Reliability

• GitHub client: bounded retry with jitter for network errors, 5xx, 429 and rate-limit 403s, honoring retry-after/x-ratelimit-reset. Side-effecting POSTs are never repeated. Installation tokens are cached in memory. A "GitHub degraded" state replaces local failures.
• Webhooks: tolerant decoding instead of 400, and a two-minute cron redelivers failed deliveries (GitHub delivery IDs stay strings because they exceed safe integers).
• Scheduling: free slots are reported per environment, so the macOS limit is respected in the cloud. Only real failures count against the attempt budget.
• Lifecycle:
  • restart recovery (runner_recovery_pending -> runner_interrupted) with backoff;
  • 5 minute idle drain for every runner;
  • runner_offline when GitHub reports the runner offline;
  • visible listener exits;
  • a non-blocking command queue.
• Cloud connection: machine tokens carry a unique jti, so the Convex client schedules its refresh (already deployed to production, compatible with released apps). New environments are reported before repository connect.
• Status is bounded, with operation and activity endpoints, and MCP status is compact.

Speed and capacity

• Phase timings for every runner, shown as a desktop timeline.
• environment.provision builds derived environments from a ready one (optional per-repository setup script) with the runner preinstalled. Fresh images get the same. Provisioning guests shut down through the hypervisor (macOS falls back to sync plus a forced stop, recorded in the result).
• Windows guests get full Git for Windows (bash, GNU tar), zstd and PowerShell 7, a Defender policy exclusion for C:\ (AMSI blocks Defender cmdlets nondeterministically and Tamper Protection ignores them; the policy applies at next boot), and quieter background services.
• A shared VM memory budget replaces 1:1 CPU and summed-memory reservations: vCPUs are overcommitted, with a measured footprint and a host memory pressure gate.
• An optional warm pool of booted, credential-free VMs.
• Optional per-repository cache volumes (block devices, CoW forks per job; write-back only for push/schedule/dispatch on the default branch of the same repository).
• Optional GitHub runner scale set capacity mode per binding (preview API): GitHub assigns jobs to this machine's scale set and Vectis starts runners for the assigned level, without workflow changes. Restricted access policies stay on the demand mode.

Proof (dev deployment, real VMs, private sandbox repository, unless noted):

• Three parallel macOS jobs ran two at a time with no failed attempt.

• A simulated webhook outage was redelivered.

• kill -9 mid-run recovered the runner with no runner left on GitHub, and an idle runner was released after 5 minutes.

• The relay stayed connected across token expiry; it dropped every five minutes before, on main as well.

• Windows (679 MB pnpm store), base vs provisioned: pnpm install 105 s -> 28 s, cache save 25 s -> 16 s (GNU tar + zstd), whole job about 3 min -> 67 s, runner download removed.

• Linux, Windows and macOS derived environments each ran a real job with the preinstalled runner. A fresh Linux image prepared in 2 min 44 s.

• Warm pool takeover took 0.0 s instead of a 9.6 s cold start.

• The cache volume hit on the second run (setup-node from the tool cache), and a non-default branch could not write it back.

• Scale sets: Linux and macOS jobs ran unchanged workflows, macOS respected the two-VM limit, and pause, restart, crash and switching back to demand all recovered.

• pnpm check: 1612 tests pass.

• Scale set jobs ran end to end on Linux, macOS and Windows.

Not included: VZ memory snapshots. Measured with a Developer ID signed helper (restore reaches SSH in 1.8 s vs 5.5 s cold on Linux, 3.4 s vs 13 s on macOS), but per-slot identity, RNG reseeding and compatibility fallback make it a separate change.