Fix Primus Llama-3.1-8B scaleout: NCCL_NET_PLUGIN for v26.4 RCCL overlay etc.

[i-kosarev]

Summary

Fixes required to run the Primus Llama-3.1-8B scaleout workload (PR ROCm#174
workload-1) on real multi-node AINIC hardware with a swappable RCCL overlay.
All hit during validation on mia1. Two are runtime, one is the overlay-injection
build fix.

Build — RCCL overlay actually takes effect

1. docker/primus_megatron_train_rccl_overlay.ubuntu.amd.Dockerfile: patch
   _rocm_sdk_libraries + global on-disk librccl sweep and gate.
   v26.4 splits ROCm libs into _rocm_sdk_libraries/lib (runtime .so) and
   _rocm_sdk_devel/lib (dev). torch maps librccl.so.1 from
   _rocm_sdk_libraries at runtime (soname wins once loaded), which the
   targeted /opt/rocm + torch/lib overwrites did not cover — so the
   candidate RCCL was built and shipped but the stock base librccl actually
   ran, silently invalidating the whole overlay validation. Stage 2 now sweeps
   every non-symlink librccl.so* on disk and overwrites it with the candidate
   (+add-needed librocm_smi64); the final gate asserts every on-disk librccl
   (not just 3 fixed paths) is RCCL 2.x with a baked git hash matching the built
   SHA. This is the injection robustness the RCCL-swap workflow depends on.

Runtime

2. GBS normalization for the 8B branch of
   scripts/primus_scaleout/megatron-lm/primus_megatron-lm_benchmark_report.sh.
   The 70B branch already normalizes GBS to the world size and passes explicit
   --micro_batch_size/--global_batch_size; the 8B branch did neither. At any
   node count where GBS % (MBS * world_size) != 0 (3-node/24-GPU: 512 % 96 != 0)
   Megatron aborts at startup with global batch size ... not divisible by micro batch size ... times data parallel size. Fix mirrors the 70B branch.
3. NCCL_NET_PLUGIN=none in assets/manifests/primus_llama-3.1-8b.template.json
   (both env blocks). rocm/primus:v26.4 defaults NCCL_NET_PLUGIN=librccl-anp.so;
   that ANP plugin is incompatible with a bundled RCCL overlay and RCCL init hangs.

Docs

4. references/gotchas.md — documents the GBS and NCCL_NET_PLUGIN pitfalls.

Validation

Primus Llama-3.1-8B scaleout, 3-node / 24-GPU MI355X (mia1). With the
_rocm_sdk_libraries sweep, the NCCL banner shows the candidate develop
86f60f9 (before the sweep it silently ran stock d34cbb6). AINIC RoCE
transport confirmed. 50 iters, 0 nan / 0 skipped, ~1550 TFLOP/s/GPU,
~30067 tokens/s/GPU (BF16). Without fix #2 the run aborts before iter 1; without
fix ROCm#3 RCCL init hangs.

Test plan

• Overlay librccl SHA in the runtime NCCL banner == candidate build SHA (not stock)
• Llama-3.1-8B scaleout completes 50 iters where GBS is not divisible by MBS * world_size
• NCCL banner shows AINIC RoCE transport
• JSON template parses; bash -n on the edited script passes
• Re-confirm on the default 2-node template shape (GBS divisible → normalize is a no-op)

🤖 Generated with Claude Code

[mkuznet1]

Used in d0fa669