fix(cdi): Include 32-bit libraries in discovery

[ehfd]

Superseding and takeover of #1968.

Closes #1968

Reviewer: @henry118 @cdesiniotis

Continues #1968 by @elibosley, whose two commits are included here unmodified. The third commit addresses @henry118's review: instead of gating compat32 discovery behind an opt-in flag, the container-side hook that creates the unsafe condition is made architecture-aware, so the 32-bit stack is exposed only where it can actually be used. Related to #563.

Background

ldcache.List() returns the 32-bit and 64-bit library sets separately, and the CDI lookup path discarded the 32-bit one. The default library locator also stopped at the first source that produced a match, so the linker cache could not contribute libraries from a compat32 directory once a library had been found in a predefined native path.

On a multilib host the generated spec therefore exposes the 64-bit driver stack while leaving a 32-bit application — a Steam or Proton title, a 32-bit CUDA or VDPAU consumer — without the matching vendor libraries, even though ELF32 driver libraries are installed and catalogued in the host's linker cache. Such applications fall back to software rendering or fail to start.

Why this is not behind an opt-in flag

The review on #1968 asked for an opt-in flag, mirroring --compat32 in libnvidia-container, because on musl containers the 32-bit and 64-bit directories are appended to a single /etc/ld-musl-<arch>.path with no architecture tagging: musl's dynamic linker loads the first file matching the requested name and fails instead of continuing the search, so a wrong-arch match breaks the workload.

That failure is real and is reproduced below. It is also entirely a property of the container, decided in our own update-ldcache hook, at a point where the architecture of every injected library is directly observable:

• glibc containers are unaffected either way. ld.so.cache entries are tagged with the architecture of the library they refer to, and ld.so rejects a wrong-class file and continues searching. This is what makes 32-bit support work at all, and it is why the fix belongs on the musl side only.
• --compat32 in the legacy path is not purely a gate. nvc_container.c resolves the container's own libs32_dir from its rootfs (/usr/lib/i386-linux-gnu, /usr/lib32, or /usr/lib) and mounts the 32-bit libraries there. CDI mounts host paths as-is and has no equivalent routing step, so the container-side hook is the only place where this can be resolved.
• A flag would also be the wrong shape for the problem. It is set on the host, once, while whether the 32-bit libraries are usable — or harmful — is a property of each container: an Alpine image and a multilib Ubuntu image on the same host need opposite answers.
• The opt-in already exists at the packaging level. The 32-bit driver stack is a separate, optional install on every distribution: the .run installer asks whether to install the 32-bit compatibility libraries, and Debian and Ubuntu ship them as a separate :i386 package. A host without it produces the same spec as today.

internal/ldconfig therefore classifies each injected directory by the ELF class of the libraries it contains (debug/elf) and:

• writes only native-class directories to /etc/ld-musl-<native>.path;
• writes 32-bit directories to /etc/ld-musl-i386.path (armhf on arm64), and only when a dynamic linker for that architecture is present in the container. In practice no such linker exists, so those directories are simply not searched;
• prepends newly added native driver directories ahead of the existing entries, so that an injected library wins a name lookup against a wrong-arch file that happens to sit in a directory musl already searches. This is the same precedence the glibc path already gives these directories through the 00-nvcr-*.conf drop-in;
• preserves musl's built-in /lib:/usr/local/lib:/usr/lib search path when it creates the file. Alpine ships no .path file, so the hook creates one, and creating it replaces the default search path. /usr/local/lib is silently dropped today; this is a pre-existing bug in the same function.

Classification reads the ELF header rather than matching path names such as i386-linux-gnu or lib32, so it also holds for hosts that do not follow those conventions.

The .path file is split on colons and newlines, exactly as musl's dynamic linker splits it, so the hook sees the same entries the linker will.

isMusl now also detects the musl dynamic linker, falling back to the /etc/alpine-release check it used before.

Verification

Bare-metal x86_64, Ubuntu 26.04, 2x Tesla P100-SXM2-16GB, driver 580.178.04, libnvidia-gl-580:i386 installed, Docker 29.7.2. Specs generated with nvidia-ctk cdi generate --mode=nvml into /etc/cdi under distinct vendors and requested with docker run --runtime=runc --device <vendor>/gpu=all.

Discovery. The generated spec grows from 58 to 83 hostPath entries: 25 ELF32 driver libraries under /usr/lib/i386-linux-gnu, their soname symlinks, and --folder /usr/lib/i386-linux-gnu{,/vdpau} for the update-ldcache hook.

The musl failure, reproduced. With the 32-bit libraries in the spec and the unmodified hook, in an Alpine container whose image ships a .path file:

# ldd /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
Error loading shared library libnvidia-glsi.so.580.178.04: Exec format error (needed by /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0)
Error loading shared library libnvidia-tls.so.580.178.04: Exec format error (needed by /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0)
Error loading shared library libnvidia-glcore.so.580.178.04: Exec format error (needed by /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0)

With this PR, same image and spec:

# ldd /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
	libnvidia-glsi.so.580.178.04 => /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.580.178.04
	libnvidia-tls.so.580.178.04 => /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.580.178.04
	libnvidia-glcore.so.580.178.04 => /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.580.178.04

/etc/ld-musl-x86_64.path in a stock Alpine container contains the native driver directories and no i386 entry, and no /etc/ld-musl-i386.path is created. In the image above, which ships a .path file, the native driver directories are prepended and the existing entries keep their order. An image that carries a 32-bit musl dynamic linker gets the 32-bit directories in /etc/ld-musl-i386.path and nothing else changes.

glibc containers. In ubuntu:24.04 and fedora, nvidia-smi -L is unchanged, the ldcache carries both architectures, the 00-nvcr-*.conf drop-in lists the 32-bit directories, and the 32-bit driver libraries resolve against each other (libX11/libXext are absent because the image has no i386 X11 packages):

libcuda.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so.1
libcuda.so.1 (libc6)        => /usr/lib/i386-linux-gnu/libcuda.so.1

# ldd /usr/lib/i386-linux-gnu/libGLX_nvidia.so.0
	libnvidia-glsi.so.580.178.04 => /usr/lib/i386-linux-gnu/libnvidia-glsi.so.580.178.04
	libnvidia-tls.so.580.178.04 => /usr/lib/i386-linux-gnu/libnvidia-tls.so.580.178.04
	libnvidia-glcore.so.580.178.04 => /usr/lib/i386-linux-gnu/libnvidia-glcore.so.580.178.04

32-bit workloads. In multilib images (ubuntu:26.04 and debian:trixie with libc6:i386, libvulkan1{,:i386}, libegl1{,:i386}, libx11-6{,:i386} and libxext6{,:i386}), a 32-bit process enumerates both P100s through Vulkan (vkCreateInstance, vkEnumeratePhysicalDevices, vkGetPhysicalDeviceProperties, served by the injected 32-bit libGLX_nvidia.so.0 ICD) and initialises the CUDA driver API (cuInit succeeds and reports both devices through the 32-bit libcuda.so.1). On main the same programs fail with VK_ERROR_INCOMPATIBLE_DRIVER and a failed dlopen, while their 64-bit builds pass either way. The ICD dlopens glvnd's libEGL.so.1 and the X11 client libraries of its own architecture, so the image has to provide those; without them it fails identically on main.

Unit tests cover the locator merge and the ldcache split (from #1968) and the ELF class split, path-file ordering, default-search-path preservation and musl detection. Tests for the touched packages pass, and golangci-lint run ./..., go vet ./... and gofmt are clean.

Notes for reviewers

• NVIDIA_CTK_LIBCUDA_DIR becomes /usr/lib/x86_64-linux-gnu:/usr/lib/i386-linux-gnu on multilib hosts. The variable already carries a list, and the native directory stays first, but any consumer taking the last entry would now get the 32-bit one.
• Replacing First with Merge in NewLibraryLocator widens every lookup that uses it, not only 32-bit ones: the ldcache is now consulted even when a predefined search path already matched. Deduplication and native-first ordering keep version inference, which uses the first result, unchanged.
• Hosts that pass explicit search paths — NixOS drives nvidia-ctk with --library-search-path — take the early return in NewLibraryLocator and are unaffected by the discovery change. Exposing the 32-bit stack there means passing a second search path for the driver's lib32 output; the hook change in this PR classifies those directories correctly, since it does not rely on path naming.
• Prepending in the musl .path file only applies to directories that are not searched already, so a driver directory that is also a system directory keeps its position.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[elibosley]

Thank you! I apologize for missing the earlier comments on my PR.

[ehfd]

@elibosley You're the author in the commits, so it's just a carry-over. I'll handle the rest! Thank you for your contribution!

[myeolenv]

Thanks for the contribution. We will review this PR soon.

[ehfd]

@henry118 Just a cordial reminder. I just would like this before v1.20.1 to blend with the other PRs.

[ehfd]

@henry118 I reduced the code footprint dramatically while doing the same work. FYI.

[henry118]

@ehfd Thanks for bearing with me. A few thoughts on the general approach:

1. I still think we want a flag here to gate this feature. Unconditionally discovering and mounting the full 32-bit lib set for every container is probably unnecessary for most use cases;
2. IMO we can skip the 32-bit injection for alpine entirely. This feels like solving for a use case we would never support (32-bit containers), and the host is always assumed 64-bit.