cuda-nix

A fully-fledged example showing how to build a self-written CUDA application with Nix and exercise it on a real GPU in a NixOS integration test.

The app is a tiny, hand-written SAXPY kernel (y = a*x + y) compiled with nvcc from nixpkgs' cudaPackages. The integration test boots a NixOS test VM, starts a systemd-nspawn container with the host GPU bound in, runs the app and asserts on its output.

The GPU plumbing for the test driver is adapted from the NixOS test driver manual. The difference here is that we install our own CUDA program instead of the prebuilt pkgs.cudaPackages.saxpy.

Layout

.
├── flake.nix                         # packages, tests, devshell
├── cuda-app/
│   ├── saxpy.cu                      # the self-written CUDA program
│   ├── CMakeLists.txt                # build recipe 
│   └── default.nix                   # derivation that drives CMake
└── tests/
    ├── generic.nix                   # vendor-independent test (boots VM + container)
    ├── nvidia.nix                    # NVIDIA device bind mounts
    ├── amd.nix                       # AMD/ROCm device bind mounts
    ├── overlay.nix                   # patches nixos-test-driver for GPU access
    └── nixos-test-driver-gpu.patch

Building the app

No GPU is needed to compile the program — only nvcc — so this works in the ordinary Nix sandbox:

nix build .#cuda-app
./result/bin/saxpy      # running it *does* need an NVIDIA GPU + driver

Expected output on a working GPU:

Max error: 0.000000

How it's built

The build commands live in cuda-app/CMakeLists.txt, which just enables CMake's CUDA language and adds the saxpy target. We don't pin CUDA_ARCHITECTURES, so CMake falls back to nvcc's default (compute_52), which emits baseline SASS plus PTX. The driver JIT-compiles that PTX to whatever GPU it runs on, so the binary stays portable across cards. For production you'd set the target's CUDA_ARCHITECTURES property to embed native, arch-tuned SASS and skip the one-time JIT at first launch.

Because CMake provides the configure/build/install phases out of the box, cuda-app/default.nix only declares the inputs:

• nativeBuildInputs = [ cmake cudaPackages.cuda_nvcc ] — the build system plus a setup hook that lets CMake find the CUDA toolkit.
• buildInputs = [ cudaPackages.cuda_cudart ] — the CUDA runtime to link against.

There are no hand-written buildPhase/installPhase blocks.

The CMakeLists.txt also sets CUDA_RUNTIME_LIBRARY Shared. This matters: nvcc links the CUDA runtime statically by default, and such a binary cannot locate the host's NVIDIA driver and dies with "CUDA driver version is insufficient for CUDA runtime version". Linking the shared libcudart.so.12 from nixpkgs works because that library carries /run/opengl-driver/lib in its RUNPATH, so the real host driver is found at runtime (both directly and inside the test container).

Running the integration test

nix build .#test-cuda-nvidia -L     # NVIDIA
nix build .#test-cuda-amd    -L     # AMD / ROCm

-L streams the test driver / container logs so you can watch the SAXPY run.

These tests only run on a machine that actually has the GPU and whose Nix daemon is configured to expose the GPU to the build sandbox (see below). That is why they are exposed as packages rather than checks.

Host setup (required to run the tests)

The tests declare requiredFeatures.cuda / .nvidia-gpu / .amd-gpu, which translate into requiredSystemFeatures on the build derivation. The build will only be scheduled on a machine advertising those features, and the GPU device nodes / driver paths must be mounted into the Nix build sandbox.

On a NixOS host with an NVIDIA GPU, this single preset wires up both the system features and the sandbox mounts:

{
  hardware.graphics.enable = true;
  hardware.nvidia.open = true;          # or your usual NVIDIA driver config

  programs.nix-required-mounts.enable = true;
  programs.nix-required-mounts.presets.nvidia-gpu.enable = true;
  # ^ sets nix.settings.system-features to include "gpu" "nvidia-gpu" "cuda"
  #   and exposes /dev/nvidia*, /dev/dri and /run/opengl-driver to the sandbox.
}

After rebuilding, confirm the features are advertised:

nix show-config | grep system-features

For AMD/ROCm host setup and more background, follow the upstream tutorial: https://github.com/applicative-systems/nixos-test-driver-manual/blob/main/docs/tutorials/cuda-tests.md

Development shell

nix develop          # gives you cmake + nvcc + cudart on PATH
cmake -S cuda-app -B build && cmake --build build && ./build/saxpy

Notes

• allowUnfree = true and cudaSupport = true are set for the CUDA nixpkgs instance in flake.nix; the plain saxpy package itself depends on cudaPackages directly and builds regardless.
• tests/overlay.nix patches nixos-test-driver so it bind-mounts /run into /host/run, letting the container re-expose /run/opengl-driver. This is a temporary workaround carried over from the upstream manual.