Add support for injecting POSIX rlimits via nvidia-container-runtime config

[liuxu623]

Addresses #2037.

Motivation

Kubernetes provides no way to set per-container POSIX rlimits (kubernetes/kubernetes#3595), and the common advice for RDMA workloads — adding the IPC_LOCK capability — is silently ineffective for non-root containers: capabilities added via securityContext land in the bounding set but never in a non-root container's effective set, so ibv_reg_mr still fails with ENOMEM against the inherited RLIMIT_MEMLOCK (typically the 8 MiB systemd default of the container runtime daemon). Raising the limit on the daemon instead changes it for every container on the host, and containerd's per-runtime base_runtime_spec replaces the entire default spec, which is fragile across containerd upgrades.

nvidia-container-runtime already rewrites the OCI spec per container and is naturally scoped to exactly the workloads that need pinned memory, so this adds an opt-in config option there. See #2037 for the full analysis.

What this PR does

New rlimits option in the runtime config:

[nvidia-container-runtime]
rlimits = ["memlock=unlimited"]

• Entries have the form NAME=SOFT[:HARD]. Names are case-insensitive with an optional RLIMIT_ prefix; values are non-negative integers or unlimited/infinity (mapped to RLIM_INFINITY). HARD defaults to SOFT; SOFT > HARD and duplicate types are rejected.
• Implemented as an oci.SpecModifier (internal/modifier/rlimit.go) that merges the configured entries into spec.Process.Rlimits: an entry replaces an existing rlimit of the same type, other entries in the spec are preserved.
• Applies in all runtime modes (legacy/csv/cdi/jit-cdi) and is deliberately not gated on requested devices: workloads that need the limits (e.g. RDMA-only containers using an rdma/* resource) do not necessarily request GPUs.
• The toolkit container accepts --nvidia-container-runtime.rlimits / env NVIDIA_CONTAINER_RUNTIME_RLIMITS and renders the option into the installed config.toml, so GPU Operator users can enable it with no operator changes:

toolkit:
  env:
  - name: NVIDIA_CONTAINER_RUNTIME_RLIMITS
    value: memlock=unlimited

Notes

• Default behavior is unchanged: the option is omitempty and the modifier is skipped when unset (nvidia-ctk config default output is identical).
• This only takes effect where nvidia-container-runtime is the runtime (RuntimeClass or default runtime). The hook-only path (docker --gpus) cannot alter rlimits, since a prestart hook runs after the process limits are set.
• Also adds a []string case to the toolkit installer's optional-config switch; the existing []string-typed options previously fell through to the "Unexpected type" warning.

Testing

• Unit tests for entry parsing (valid/invalid forms, duplicates, soft>hard), spec merging (replace/append/nil-process), and modifier construction.
• go test ./internal/modifier/... ./api/config/... ./cmd/nvidia-ctk-installer/toolkit/... ./internal/runtime/... passes.
• Mechanism verified against a Kubernetes 1.36 / containerd 2.3.3 cluster where non-root GPU pods previously failed RDMA memory registration with the inherited 8 MiB limit.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[henry118]

@liuxu623 Thanks for the contribution.

I don't think this belongs in nvidia-container-runtime.

ulimit is a generic runtime/process setting, while the toolkit should stay focused on nvidia GPU-related configs. And CDI doesn't provide a way to configure rlimits either.

For this use case, an NRI plugin would be a better fit for the timebeing until KEP-5758 lands.