From 7ff69c9e535b506efc1d06f1cbd3cabf4549da5e Mon Sep 17 00:00:00 2001 From: Ken VanDine Date: Tue, 28 Jul 2026 21:50:42 -0400 Subject: [PATCH 1/2] ci: unblock the release job's artifact download and parallelize xz The release job has been cancelling mid-"Download artifacts" more often than not, which reads as a CUDA timeout but isn't: in runs 30341691913 and 30364671144 every build job succeeded and only `release` died, both times ~14 minutes into the download with 4 of 28 artifacts complete. Two causes: - `ubuntu-slim` is a container runner (the log reports "VM Image - Source: Docker, Name: ubuntu:24.04"), and it was being asked to hold ~14 GB of artifacts plus the transient zip each download extracts from. Move to a full ubuntu-24.04 VM. - actions/download-artifact starts every matched artifact concurrently. Its PARALLEL_DOWNLOADS chunking is ineffective because the promises are constructed eagerly in .map() before chunk() runs, so all 28 downloads begin within half a second. There is no concurrency input, so split the call into four pattern-scoped steps. The patterns partition the artifact list exactly: no unmatched and no double-matched names. A pattern matching nothing does not throw (only the `name` and `artifact-ids` paths do), so a dropped backend family still degrades into the existing completeness checks. Separately, `tar -cJf` drives a single-threaded xz and was the largest cost in the Linux CUDA jobs at ~6.8 min/leg, ~112 min per run across the 16 legs. Switch to `tar -I 'xz -T0'`. Measured on a 2.9 GB binary payload: 12m41s single-threaded vs 47.7s on 32 cores, with the archive 1.8% larger because threaded mode compresses independent blocks. The runners are 4-core, so expect roughly 3-4x there. This does not address the real long pole: ubuntu-22-rocm still runs 93-144 min while every CUDA leg finishes in 15-20, and `release` cannot start until it is done. Co-Authored-By: Claude Opus 5 --- .github/workflows/release.yml | 39 ++++++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 7c8446ec20df..398f52350fd1 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -276,7 +276,7 @@ jobs: pkgdir="llama-${{ steps.tag.outputs.name }}" mkdir -p "$pkgdir" cp -a build/bin/. "$pkgdir/" - tar -cJf llama-ubuntu-cuda-${{ matrix.sm }}-x64.tar.xz "$pkgdir" + tar -I 'xz -T0' -cf llama-ubuntu-cuda-${{ matrix.sm }}-x64.tar.xz "$pkgdir" - name: Upload artifacts uses: actions/upload-artifact@v6 @@ -434,7 +434,7 @@ jobs: pkgdir="llama-${{ steps.tag.outputs.name }}" mkdir -p "$pkgdir" cp -a build/bin/. "$pkgdir/" - tar -cJf llama-ubuntu-cuda-${{ matrix.sm }}-arm64.tar.xz "$pkgdir" + tar -I 'xz -T0' -cf llama-ubuntu-cuda-${{ matrix.sm }}-arm64.tar.xz "$pkgdir" - name: Upload artifacts uses: actions/upload-artifact@v6 @@ -949,7 +949,7 @@ jobs: permissions: contents: write # for creating release - runs-on: ubuntu-slim + runs-on: ubuntu-24.04 needs: - windows-cpu @@ -972,13 +972,42 @@ jobs: id: tag uses: lemonade-sdk/llama.cpp/.github/actions/get-tag-name@lemonade - - name: Download artifacts - id: download-artifact + # Downloaded in batches rather than one call: actions/download-artifact + # starts every matched artifact concurrently, and 28 parallel extractions + # of ~14 GB stalls the runner. + - name: Download artifacts (backends) uses: actions/download-artifact@v7 with: + pattern: llama-bin-* path: ./artifact merge-multiple: true + - name: Download artifacts (Ubuntu CUDA x64) + uses: actions/download-artifact@v7 + with: + pattern: llama-ubuntu-cuda-*-x64.tar.xz + path: ./artifact + merge-multiple: true + + - name: Download artifacts (Ubuntu CUDA arm64) + uses: actions/download-artifact@v7 + with: + pattern: llama-ubuntu-cuda-*-arm64.tar.xz + path: ./artifact + merge-multiple: true + + - name: Download artifacts (Windows CUDA) + uses: actions/download-artifact@v7 + with: + pattern: llama-windows-cuda-* + path: ./artifact + merge-multiple: true + + - name: Report downloaded artifacts + run: | + ls -la artifact + df -h . + - name: Move artifacts id: move_artifacts run: | From 7c053e2b38fbe5e3adc52091dabdb1d32d70ca3f Mon Sep 17 00:00:00 2001 From: Ken VanDine Date: Tue, 28 Jul 2026 22:01:36 -0400 Subject: [PATCH 2/2] ci: install only the CUDA components the build uses The Linux CUDA jobs installed the cuda-toolkit-12-9 metapackage, which pulls Nsight Systems, Nsight Compute, cuFFT, cuSPARSE, cuSOLVER and NPP along with the pieces ggml-cuda is actually compiled against. That is "Need to get 3935 MB of archives" on every one of the 16 Linux CUDA legs, none of it cached. Resolve the dependency closure by hand instead: nvcc, cudart-dev (which also supplies the libcuda.so stub behind CUDA::cuda_driver), cccl for the thrust/cub headers, and cublas/curand/nvjitlink -dev. That is 16 packages and ~1.2 GB against 61 packages and ~3.9 GB, and it still satisfies the Build, Bundle CUDA runtime libraries and Validate steps. None of the subset declares Recommends, so nothing sneaks back in. Add a post-install existence check over nvcc and the specific headers and libraries the later steps consume, so a repackaging upstream fails at the install step naming the missing file rather than partway through a compile. This matches what windows-cuda already does, where Jimver/cuda-toolkit is invoked with an explicit sub-packages list. Co-Authored-By: Claude Opus 5 --- .github/workflows/release.yml | 64 +++++++++++++++++++++++++++++++++-- 1 file changed, 62 insertions(+), 2 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 398f52350fd1..5d51ea44fd95 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -162,7 +162,43 @@ jobs: wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update - sudo apt-get install -y cuda-toolkit-12-9 cmake ninja-build patchelf + # Install only the components this build actually consumes rather than + # the cuda-toolkit-12-9 metapackage. That metapackage drags in Nsight + # Systems/Compute, cuFFT, cuSPARSE, cuSOLVER and NPP -- ~3.9 GB of + # archives per job, none of it compiled against or shipped. The subset + # below is ~1.2 GB and covers everything the Build, Bundle CUDA runtime + # libraries and Validate steps reference: + # cuda-nvcc the compiler (pulls cuda-crt, cuda-nvvm) + # cuda-cudart-dev libcudart + the libcuda.so stub (CUDA::cuda_driver) + # cuda-cccl thrust/cub headers included by ggml-cuda + # libcublas-dev libcublas + libcublasLt + # libcurand-dev bundled and asserted on by Validate + # libnvjitlink-dev bundled and asserted on by Validate + # The -dev packages depend on their runtime counterparts, and + # cuda-toolkit-12-9-config-common (a transitive dep) is what creates the + # /usr/local/cuda -> /usr/local/cuda-12.9 alternative used below. + sudo apt-get install -y \ + cuda-nvcc-12-9 \ + cuda-cudart-dev-12-9 \ + cuda-cccl-12-9 \ + libcublas-dev-12-9 \ + libcurand-dev-12-9 \ + libnvjitlink-dev-12-9 \ + cmake ninja-build patchelf + + # Fail here, with the name of the missing file, rather than partway + # through the compile if NVIDIA ever repackages one of these out from + # under the hand-picked list above. + for f in /usr/local/cuda/bin/nvcc \ + /usr/local/cuda/include/cublas_v2.h \ + /usr/local/cuda/include/cub/cub.cuh \ + /usr/local/cuda/lib64/libcudart.so \ + /usr/local/cuda/lib64/libcublas.so \ + /usr/local/cuda/lib64/libcublasLt.so \ + /usr/local/cuda/lib64/libcurand.so \ + /usr/local/cuda/lib64/libnvJitLink.so; do + [ -e "$f" ] || { echo "::error::CUDA install is missing $f"; exit 1; } + done - name: Set CUDA environment run: | @@ -320,7 +356,31 @@ jobs: wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update - sudo apt-get install -y cuda-toolkit-12-9 cmake ninja-build patchelf + # See the x86_64 job above for why this is a hand-picked subset rather + # than the cuda-toolkit-12-9 metapackage. On sbsa the metapackage is + # ~3.3 GB of archives; this subset is ~1.2 GB. + sudo apt-get install -y \ + cuda-nvcc-12-9 \ + cuda-cudart-dev-12-9 \ + cuda-cccl-12-9 \ + libcublas-dev-12-9 \ + libcurand-dev-12-9 \ + libnvjitlink-dev-12-9 \ + cmake ninja-build patchelf + + # Fail here, with the name of the missing file, rather than partway + # through the compile if NVIDIA ever repackages one of these out from + # under the hand-picked list above. + for f in /usr/local/cuda/bin/nvcc \ + /usr/local/cuda/include/cublas_v2.h \ + /usr/local/cuda/include/cub/cub.cuh \ + /usr/local/cuda/lib64/libcudart.so \ + /usr/local/cuda/lib64/libcublas.so \ + /usr/local/cuda/lib64/libcublasLt.so \ + /usr/local/cuda/lib64/libcurand.so \ + /usr/local/cuda/lib64/libnvJitLink.so; do + [ -e "$f" ] || { echo "::error::CUDA install is missing $f"; exit 1; } + done - name: Set CUDA environment run: |