[Issue]: dlopen() of a truncated plugin .so causes SIGBUS crash instead of graceful fallback (NCCL_PROFILER_PLUGIN / NET / TUNER / ENV)

[AdamSabry1233]

How is this issue impacting you?

Application crash

Share Your Debug Logs

The two traces below are the same fault

UCX merely installed a signal handler in the multi-rank run — it is not
involved in the crash path. The gdb trace is the same fault caught directly,
with no UCX and no PyTorch present.

UCX-caught crash signature (16-rank DDP job, NCCL 2.30.1):

[worker-0:xxxxx:0:xxxxx] Caught signal 7 (Bus error: nonexistent physical address)
==== backtrace ====
... libc.so.6(dlopen+0x6f)
... libnccl.so.2(pncclCommInitRankConfig+0x298)
... (torch/PyTorch frames)

gdb, catching the fault directly (minimal repro, single GPU, no PyTorch, no UCX):

Thread 1 "all_reduce_perf" received signal SIGBUS, Bus error.
memset () at ../sysdeps/x86_64/multiarch/../multiarch/memset-vec-unaligned-erms.S:392
#0 memset ()
#1 0x0000155555523f0f in _dl_map_segments (header=0x1555405f0018, ...) at ./dl-map-segments.h:176
#2 _dl_map_object_from_fd (name="/path/to/trunc_2000.so", ...) at ./elf/dl-load.c:1258
#3 0x0000155555525529 in _dl_map_object (name="/path/to/trunc_2000.so", type=2, ...) at ./elf/dl-load.c:2268
#4 0x00001555555298dc in dl_open_worker_begin (...) at ./elf/dl-open.c:578

Fault address: 0x1555405f0008
info proc mappings (surrounding region):
0x1555405ef000 0x1555405f1000 size 0x2000 offset 0x2000 rw-p trunc_2000.so

readelf confirms the file is structurally inconsistent:

$ readelf -l trunc_2000.so
readelf: Error: the dynamic segment offset + size exceeds the size of the file
LOAD 0x0000000000002e68 0x0000000000003e68 ... FileSiz 0x1a0 RW
DYNAMIC 0x0000000000002e78 ... FileSiz 0x150 <- offset+size = 12232, file is only 2000 bytes

Steps to Reproduce the Issue

Minimal repro (single GPU, single process, bare command — no container,
no multi-node, no plugin-specific code):

1. Build any trivial valid shared library — no NCCL symbols needed at all:
   echo 'int f(int x){return x+1;}' > t.c
   gcc -shared -fPIC -o t.so t.c

2. Confirm baseline: NCCL loads and gracefully skips a valid-but-irrelevant .so
   (no profiler symbols found -> plugin disabled, job runs fine):
   NCCL_PROFILER_PLUGIN=/path/to/t.so ./build/all_reduce_perf -g 1 -b 8 -e 8 -n 1

3. Truncate the SAME file to a length that leaves its own program headers
   describing segments beyond the new (shorter) EOF. readelf -l t.so will
   flag this itself ("... exceeds the size of the file") -- for a ~15KB
   library, 2000 bytes reproduces it reliably:
   truncate -s 2000 t.so

4. Re-run the identical command:
   NCCL_PROFILER_PLUGIN=/path/to/t.so ./build/all_reduce_perf -g 1 -b 8 -e 8 -n 1

Observed: process dies with SIGBUS inside dlopen() (see backtrace above),
instead of the graceful "plugin not usable, continuing without it"
behavior seen in step 2.

Scope — full malformed-state matrix (tested on both NCCL versions,
identical results on each):

State	File size	Result
valid .so (control)	15128	graceful skip
truncated to 100 bytes	100	graceful skip
truncated to 2000 bytes	2000	SIGBUS crash
truncated to half	7564	SIGBUS crash
full size, random bytes at offset 8000-10000	15128	graceful skip
zero-length	0	graceful skip
non-ELF (text file renamed .so)	51	graceful skip
valid ELF header, all program headers zeroed	15128	graceful skip

Only files whose program headers describe segments extending past the
actual file length crash. Too-short files (can't contain the program-header
table) and same-size corruption both fail cleanly.

Other notes:

• Reproduces on NCCL 2.28.9 and 2.30.1 identically -- longstanding
  behavior in the shared plugin-loader code, not a version regression.
• Reproduces with a plugin file that never mentions NCCL/profiler symbols
  at all -- generic dlopen()/ELF-loading issue, not profiler-specific.
  By code inspection (see root cause) equally reachable via
  NCCL_NET_PLUGIN, NCCL_TUNER_PLUGIN, NCCL_ENV_PLUGIN.
• Reproduces with a single GPU, single rank, single process.
• Not intermittent: 100% reproducible.

NCCL Version

2.30.1+cuda13.0 (host install, driver 580.159.04) and 2.28.9+cuda12.8 (built from source, same repro)

Your platform details

The minimal repro above needs none of the following -- it reproduces with a
bare all_reduce_perf on one GPU. Listed for completeness only.

GPU: 8x NVIDIA H200 per node, 2 nodes, driver 580.159.04
Topology: GPUs via NVLink (NV18), NICs at NODE/PHB/PIX distance
Environment: originally observed under Slurm-on-Kubernetes (Soperator),
container nvcr.io/nvidia/pytorch:25.01-py3 via pyxis/enroot, with host
NCCL shadow-mounted to test 2.30.1; NCCL 2.28.9 built from source and
selected via LD_LIBRARY_PATH. Neither the container nor the mount is
required to reproduce.
Scalability: irrelevant -- crash occurs during dlopen() of the plugin path
before any collective communication begins. Reproduces from 1 rank up to
16 ranks (2 nodes x 8 GPUs) identically.

Error Message & Behavior

NCCL already degrades gracefully for every other unusable plugin file --
missing, zero-length, non-ELF, or a valid ELF with no matching plugin
symbols. In all of those cases it logs that the plugin couldn't be
loaded/matched and continues without it. A truncated ELF file is the one
malformed state that does NOT degrade gracefully: it crashes the whole
process instead. This report is about closing that single gap in otherwise
consistent behavior.

First error: no NCCL WARN is ever emitted -- the process receives SIGBUS
directly from the dynamic loader before NCCL's own plugin-load logging
path executes. The only visible signal is the crash handler output:
"Caught signal 7 (Bus error: nonexistent physical address)"
followed by a backtrace through dlopen() -> pncclCommInitRankConfig.

Actual: SIGBUS terminates the process. Under torch.distributed/torchrun,
this surfaces as ChildFailedError and kills the entire distributed job
(observed: 13/16 ranks died this way in a 2-node DDP job when the
profiler plugin binary was corrupted on disk).

Root cause (from src/plugin/plugin_open.cc, NCCL 2.28.9 source):
tryOpenLib() calls dlopen(name, RTLD_NOW | RTLD_LOCAL) directly, with no
stat()/access()/size/ELF-magic validation beforehand:

static void* tryOpenLib(char* name, int* err, char* errStr) {
  ...
  void *handle = dlopen(name, RTLD_NOW | RTLD_LOCAL);   // line 37

This function is shared by all four plugin types (NET, TUNER, PROFILER,
ENV) via openPluginLib(), so the exposure is not specific to the profiler
path. RTLD_NOW forces eager relocation at dlopen() time, which is when
glibc's _dl_map_segments() memset()s the bss-tail of the last LOAD segment
and faults on the unbacked (truncated-away) page.

Suggested fix direction: before calling dlopen() on a user-supplied plugin
path, verify the file is self-consistent -- specifically that
max(p_offset + p_filesz) across program headers does not exceed the actual
file size (per the matrix above, this is exactly the condition that
separates the crashing cases from the gracefully-handled ones). On
mismatch, treat it the same as any other unloadable plugin: log and
continue without it.