ci: unblock the release job's artifact download, parallelize xz, and slim the CUDA install

[kenvandine]

Why

Release runs have been cancelling more often than not, and it looks like CUDA timing out. It isn't. In runs 30341691913 and 30364671144, every build job succeeded and only release died — both times ~14 minutes into Download artifacts, with 4 of 28 artifacts complete:

cancelled  16:07:54 -> 16:22:25  Download artifacts

Successful runs finish that same step in 3.5–4.5 min. CUDA itself is healthy: all 24 legs restore their ccache (76.8% hit rate, 0.13/0.50 GB), and warm Build steps are 2.5–5.1 min.

Fix 1 — the release job

Runner. ubuntu-slim is a container runner — the log reports VM Image — Source: Docker, Name: ubuntu:24.04 — and it was being asked to hold ~14 GB of artifacts plus the transient zip each download extracts from. Moved to a full ubuntu-24.04 VM.

Concurrency. actions/download-artifact@v7 starts every matched artifact at once. Its chunking is ineffective:

const downloadPromises = artifacts.map(artifact => ({
  promise: artifactClient.downloadArtifact(artifact.id, {...})   // starts NOW
}))
const chunkedPromises = chunk(downloadPromises, PARALLEL_DOWNLOADS)

The promises are constructed eagerly in .map() before chunk() runs, so PARALLEL_DOWNLOADS only paces how results are awaited. The log confirms it — all 28 Starting download lines land within 0.5 s. There is no concurrency input, so the only lever is separate pattern-scoped steps.

Verified the four patterns partition the real artifact list from run 30364671144:

llama-bin-*                        -> 4
llama-ubuntu-cuda-*-x64.tar.xz     -> 8
llama-ubuntu-cuda-*-arm64.tar.xz   -> 8
llama-windows-cuda-*               -> 8
unmatched: []   double-matched: []   total: 28

A pattern that matches nothing does not throw (only the name and artifact-ids paths do), so a dropped backend family still degrades gracefully into the existing completeness checks rather than failing the job.

Also added a ls -la / df -h step so the next stall is diagnosable instead of a silent 14-minute hang.

Fix 2 — xz threading

tar -cJf drives a single-threaded xz, and it was the single largest cost in the Linux CUDA jobs — larger than the compile:

	Pack artifacts	Install CUDA Toolkit	Build
ubuntu-cuda (8 legs)	54.0 min	35.6 min	24.4 min
ubuntu-cuda-arm64 (8 legs)	58.3 min	25.7 min	19.9 min

Switched to tar -I 'xz -T0'. Measured on a 2.9 GB binary payload:

	time	size
xz -T1 (runner default)	12m41s	719.8 MB
xz -T0 (32 cores)	47.7s	732.4 MB

Round-trip verified — xz -t passes, diff -r on the extracted tree is clean.

Two caveats: the archive is 1.8% larger, because threaded mode compresses independent blocks. And the runners are 4-core, not 32, so expect roughly 3–4× rather than 16× — about 6.8 min → ~2 min per leg, ~112 min → ~30 min of runner time per run.

Fix 3 — stop installing three quarters of the CUDA toolkit

The other column in that table. Both Linux CUDA jobs installed the cuda-toolkit-12-9 metapackage, which is Need to get 3935 MB of archives on every leg, ×16 legs, uncached.

Almost none of it is used. The build is -DGGML_STATIC=OFF, so ggml-cuda links CUDA::cudart, CUDA::cublas and CUDA::cuda_driver; the Bundle CUDA runtime libraries and Validate CUDA package contents steps additionally want libcurand and libnvJitLink. The metapackage also drags in Nsight Systems, Nsight Compute, cuFFT, cuSPARSE, cuSOLVER and NPP, which are neither compiled against nor shipped.

So the install is now a hand-resolved subset. I walked the dependency closure against the repo's own Packages.gz for both architectures:

	packages	download
cuda-toolkit-12-9 (x86_64)	61	3,885 MB
subset (x86_64)	16	1,174 MB
cuda-toolkit-12-9 (sbsa)	56	3,339 MB
subset (sbsa)	16	1,161 MB

(3,885 MB against the observed 3935 MB — the gap is the handful of deps that come from Ubuntu's own archive rather than NVIDIA's.)

The six roots, and why each is there:

package	reason
cuda-nvcc-12-9	the compiler; pulls cuda-crt, cuda-nvvm
cuda-cudart-dev-12-9	libcudart, and via cuda-driver-dev the libcuda.so stub behind CUDA::cuda_driver
cuda-cccl-12-9	thrust/cub headers included by ggml-cuda sources
libcublas-dev-12-9	libcublas + libcublasLt — 944 MB of the remaining 1.2 GB, and unavoidable
libcurand-dev-12-9	bundled and asserted on by Validate
libnvjitlink-dev-12-9	bundled and asserted on by Validate

Two things worth confirming rather than assuming:

• /usr/local/cuda still exists. It is an alternative installed by cuda-toolkit-12-9-config-common's postinst (update-alternatives --install /usr/local/cuda cuda /usr/local/cuda-12.9 129), and that package is a transitive dep of the subset. Unpacked the .deb to check.
• Nothing sneaks back in via Recommends. All 16 packages in the closure declare none, so the plain apt-get install behaves the same as --no-install-recommends here.

Since this is a hand-picked list, I also added a post-install existence check over nvcc and the exact headers and libraries the later steps consume. If NVIDIA ever repackages one of these, the job fails at the install step naming the missing file, rather than partway through a compile or — worse — at Validate after 20 minutes of work.

This also brings Linux in line with windows-cuda, which has always passed an explicit sub-packages list to Jimver/cuda-toolkit.

Expected effect: the ~61 min/run those two families currently spend in Install CUDA Toolkit should land in roughly the 20 min range. Bytes drop 70%; unpack time won't scale quite linearly, so I'd rather state the measured number and let the run confirm the rest.

Considered and rejected: caching the .debs with actions/cache. It would work, but the repo's Actions cache is already at 8.96 GiB of the 10 GiB limit — adding ~2.3 GiB of apt archives would evict the ccache entries that keep Build at 2.5–5.1 min. Not installing the bytes is strictly better than caching bytes we never needed.

What this does not fix

The real long pole is untouched. ubuntu-22-rocm runs 93–144 min while every CUDA leg finishes in 15–20, and release cannot start until it's done. Shaving the pack and install steps won't shorten a run that's gated on ROCm.

One other thing surfaced while investigating, not addressed here:

• The Actions cache is at 8.96 GiB of the 10 GiB repo limit (43 CUDA entries = 5.06 GiB; ROCm 3.76 GiB, including a single 2.61 GiB rocm-wheels blob). Each run writes ~3 GiB of fresh entries, so GitHub is silently LRU-evicting. Hasn't bitten the CUDA legs yet, but there's no headroom.

Separately: run 30390684307 was cancelled by hand at 20:05:49 and re-dispatched 11 minutes later. That one isn't a bug.

Testing

Workflow YAML parses, the release job's step list is as expected, and both rewritten Install CUDA Toolkit scripts pass bash -n. The package subset was verified by dependency-closure resolution against the live ubuntu2204/x86_64 and ubuntu2204/sbsa Packages.gz, not by eyeballing names — but the install itself only proves out on a real runner, so the CUDA legs of this PR's run are the actual check. The download-path change only exercises on schedule/workflow_dispatch and still needs a manual dispatch to confirm end to end; the xz change is covered by PR runs.

🤖 Generated with Claude Code

[kenvandine]

Summary of Review

This PR directly addresses the primary bottlenecks causing release job timeouts and cancellations in .github/workflows/release.yml.

Key Fixes Reviewed:

1. Release Job Environment & Download Batching:

   • Moving release from ubuntu-slim (container runner) to ubuntu-24.04 provides dedicated disk space and avoids IO saturation during extraction.
   • Partitioning artifact downloads into 4 pattern-scoped steps (llama-bin-*, llama-ubuntu-cuda-*-x64.tar.xz, llama-ubuntu-cuda-*-arm64.tar.xz, llama-windows-cuda-*) prevents actions/download-artifact@v7 from launching 28 parallel extractions simultaneously.
   • Added ls -la and df -h diagnostics for troubleshooting.

2. Multithreaded xz Compression:

   • Switching tar -cJf to tar -I 'xz -T0' drives xz across all available cores on the runner, cutting compression time per leg significantly.

3. CUDA Toolkit Slimming:

   • Replacing cuda-toolkit-12-9 (~3.9 GB) with the 6-package subset (cuda-nvcc-12-9, cuda-cudart-dev-12-9, cuda-cccl-12-9, libcublas-dev-12-9, libcurand-dev-12-9, libnvjitlink-dev-12-9) reduces apt payload size by ~70% (~1.2 GB).
   • The explicit post-install loop ensures compiler (nvcc), headers (cublas_v2.h, cub/cub.cuh), and runtime shared objects exist before starting the build.

LGTM!

[superm1]

Why even download and hash artifacts? Seems pretty expensive in first place.

Anyway lgtm