[Bug]:

[coventry]

Describe the bug

(NB: This is really a gvisor bug, IMO, but I don't have permission to file against google/gvisor. It breaks NVIDIA containers using runsc on my Dell GB10 Max Pro, though.)

nvidia-container-toolkit injects /run/nvidia-persistenced/socket into the OCI spec when the NVIDIA utility driver capability is enabled. With the runsc/gVisor runtime, container creation then fails because gVisor attempts to open the already-mounted Unix-domain socket as a regular file and receives ENXIO (No such device or address).

The outer Docker error is:

failed to create task for container: failed to create shim task:
OCI runtime create failed: creating container:
cannot create sandbox: cannot read client sync file:
waiting for sandbox to start: EOF

The actual gVisor gofer error is:

FATAL ERROR: error setting up FS:
mounting {Destination:/run/nvidia-persistenced/socket
Source:/run/nvidia-persistenced/socket
Options:[nosuid nodev rbind rprivate noexec]}:
open(".../merged/run/nvidia-persistenced/socket") failed:
no such device or address

The relevant OCI mount injected by NVIDIA Container Toolkit is:

{
  "destination": "/run/nvidia-persistenced/socket",
  "source": "/run/nvidia-persistenced/socket",
  "options": [
    "nosuid",
    "nodev",
    "rbind",
    "rprivate",
    "noexec"
  ]
}

The issue disappears if the utility driver capability is omitted.

I also confirmed the immediate gVisor-side cause: runsc/specutils.SafeSetupAndMount unconditionally executes:

f, err := os.OpenFile(dst, unix.O_CREAT, 0777)

for non-directory mountpoints. Since O_RDONLY == 0, this attempts a normal read-only open() of the existing Unix socket, which returns ENXIO.

A locally built runsc that first checks whether dst already exists and only calls OpenFile(..., O_CREAT, ...) when it does not exist successfully starts the previously failing container.

To Reproduce

1. Use a host with NVIDIA Container Toolkit, NVIDIA driver 580.173.02, and gVisor runsc release-20260817.0.

2. Configure runsc with nvproxy:

{
  "runtimes": {
    "runsc": {
      "path": "/usr/bin/runsc",
      "runtimeArgs": [
        "--nvproxy=true",
        "--nvproxy-allowed-driver-capabilities=all"
      ]
    }
  }
}

3. Run an NVIDIA-derived container image that has:

NVIDIA_DRIVER_CAPABILITIES=compute,utility,video

with:

docker run --runtime=runsc --gpus=all <image> ...

4. Container creation fails with:

waiting for sandbox to start: EOF

and the gVisor gofer log shows:

open(".../run/nvidia-persistenced/socket") failed:
no such device or address

5. Run the same image while overriding capabilities to exclude utility:

docker run --rm \
  --runtime=runsc \
  --gpus=all \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,video \
  <image> \
  true

This succeeds.

A compute-only configuration also succeeds.

nvidia-smi still works in the successful runsc container.

For comparison:

docker run --rm --runtime=runc --gpus=all ubuntu:24.04 nvidia-smi

works, and:

docker run --rm --runtime=runsc --gpus=all ubuntu:24.04 nvidia-smi

also works.

This indicates that nvproxy and the host NVIDIA driver are functional and that the failure is specifically triggered by the injected /run/nvidia-persistenced/socket mount.

Expected behavior

A GPU-enabled container should start successfully under runsc when the NVIDIA utility capability is enabled.

If mounting /run/nvidia-persistenced/socket into a gVisor container is unsupported, NVIDIA Container Toolkit and/or gVisor should handle that case explicitly rather than causing the gofer to terminate and exposing only the generic:

waiting for sandbox to start: EOF

error.

Environment (please provide the following information):

• nvidia-container-toolkit version:

NVIDIA Container Toolkit CLI version 1.20.0
commit: 5505e2f94d9aaa08561490db974ba3cd676af209

• NVIDIA Driver Version: 580.173.02
• Host OS: Ubuntu 24.04
• Kernel Version: 6.17.0-1031-nvidia
• Container Runtime Version:
  • Docker: 29.2.1
  • gVisor/runsc: release-20260817.0, OCI spec 1.2.1
• CPU Architecture: arm64
• GPU Model(s): NVIDIA GB10
• CUDA Version: 13.0 as reported by nvidia-smi

Hardware:

Dell Pro Max with GB10

Information info

• Output of nvidia-smi

$ nvidia-smi
Fri Aug 28 20:16:27 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.173.02             Driver Version: 580.173.02     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   43C    P8              3W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

runsc logs attached. docker container log empty.

Additional debugging

On the host, /run/nvidia-persistenced/socket is a Unix-domain socket.

A direct test confirms:

open(socket, O_RDONLY) -> ENXIO
open(socket, O_PATH)   -> succeeds

Inside a working runc container:

$ stat /run/nvidia-persistenced/socket
...
IO Block: 4096   socket
...

and /proc/self/mountinfo shows:

0:28 /nvidia-persistenced/socket /run/nvidia-persistenced/socket ...
- tmpfs tmpfs ...

The corresponding path in Docker's host-visible overlay root before entering the container mount namespace is an ordinary empty-file mountpoint placeholder.

The locally tested gVisor workaround was to change SafeSetupAndMount so that it only creates the destination file if the path does not already exist:

if _, err := os.Lstat(dst); err != nil {
    if !os.IsNotExist(err) {
        return fmt.Errorf("lstat(%q) failed: %v", dst, err)
    }

    f, err := os.OpenFile(dst, unix.O_CREAT, 0777)
    if err != nil {
        return fmt.Errorf("open(%q) failed: %v", dst, err)
    }
    f.Close()
}

With that modification to runsc/specutils/specutils.go, the previously failing container starts successfully.

runsc.log