Add SGLang disaggregated P/D inference and Primus Megatron-LM scaleout benchmarks + mad-slurm-multinode skill by mkuznet1 · Pull Request #174 · ROCm/MAD

mkuznet1 · 2026-07-01T18:00:05Z

Summary

This PR is the MAD side of a broader, parallel integration effort with
madengine to bring up multi-node disaggregated serving and scaleout training
end-to-end. On the madengine side, that effort included the native
sglang-disagg templated SLURM launcher's co-located-proxy / 2-node-minimum
topology (#149) and multi-node
distribution of a locally-built, unpushed Docker image (build-on-primary +
shared-tar load-on-workers, _ensure_local_image_available,
#142). On the MAD side, this PR
adds the model scripts, Dockerfiles, and the mad-slurm-multinode skill that
exercise both of those end-to-end.

Two new benchmarks carry that integration work, and are not the point in
themselves: SGLang disaggregated P/D inference (DeepSeek-R1) and Primus
Megatron-LM scaleout training (Llama-3.1 8B/70B/405B). The
mad-slurm-multinode skill packages the bootstrap so the same native path is
reproducible on any of the three cluster archetypes we validated against
(CX7/Mellanox-RoCE, AMD-AINIC/Pollara, Broadcom-Thor2-RoCE) without
cluster-specific code baked into scripts.

Branches off the current develop tip (Primus v26.4, #172).

Why madengine-native, not `run_xPyD_models.slurm` / `slurm_multi`

madengine has two distinct SLURM paths, and they are not peers:

Templated / native launchers (torchrun, vllm, sglang,
sglang-disagg, deepspeed, megatron, primus, ...) — first-class
citizens dispatched by _generate_launcher_command. madengine owns
orchestration: it emits the sbatch template, runs one madengine-managed
container per node, and the model script only maps roles.
slurm_multi — a self-managed escape hatch. Its own docstring says:
"Escape hatch for self-orchestrating multi-container SLURM topologies ...
This is NOT a peer of the templated launchers."
scripts/vllm_dissag/run_xPyD_models.slurm uses this path today, and it is
hardcoded to one cluster's layout (fixed paths, fixed partition/account,
static #SBATCH -N, Mellanox multi-rail directives baked into the script).

scripts/sglang_disagg/run.sh is the model-side entrypoint for path (1): it
maps the env contract _generate_sglang_disagg_command exports
(SGLANG_DISAGG_PREFILL_NODES, SGLANG_NODE_RANK, SGLANG_NODE_IPS, ...)
onto the existing launcher scripts. run.sh's co-located-proxy topology and
madengine #149's support for it were designed together, as two halves of the
same feature — not madengine shipping a capability first and MAD adding a
consumer for it afterwards.

Concretely, this also closes a real gap for this PR's own full-overlay
Dockerfiles: run_xPyD_models.slurm does a bare docker run $DOCKER_IMAGE_NAME with no build/distribution step, and slurm_multi's own
auto-pull only covers registry images (ci-*-tagged local builds are
skipped). A locally-built, unpushed overlay image only works multi-node
through madengine's native per-node run path (run_models_from_manifest ->
_ensure_local_image_available, madengine#142) — which is what run.sh
exercises on every node.

What's included

SGLang disaggregated P/D inference

Single full-overlay Dockerfile merging RCCL + MoRI + NIXL/Mooncake
KV-transfer into one build (no base-image chaining):
docker/sglang_disagg_inference_full_overlay.ubuntu.amd.Dockerfile, with an
optional ENABLE_RDMA62 build arg for hosts needing rdma-core v62 (OCI/CX7)
— merged from the previous separate *.oci-rdma62.* variant into one file
(same technique the Primus overlay already uses for RDMA_CORE_VERSION).
scripts/sglang_disagg/run.sh — native launcher entrypoint, in-container
rank-ordered node-IP discovery (ip_rendezvous.py), routes to the single
unified sglang_disagg_mori_io_ep.sh launcher for both MoRI and
Mooncake/NIXL backends (sglang_disagg_server.sh removed — dead code once
everything routes through mori_io_ep.sh, see basemam's review).
parse_to_csv.py rewritten for full metric extraction into the madengine
perf-CSV schema; backend tag now derives from RUN_MORI/KV_TRANSFER_BACKEND
instead of guessing from DP_MODE.
Resilient concurrency sweep in benchmark_xPyD.sh (fail-fast + per-point
retries, honors RUN_LOG_DIR from run.sh instead of a hardcoded
/run_logs path).

Primus Megatron-LM scaleout

docker/primus_megatron_train_rccl_overlay.ubuntu.amd.Dockerfile on the
rocm/primus:v26.4 base (RCCL from source + librccl fix, optional rdma-core
for Broadcom Thor2 bnxt_re).
Scaleout reporting scripts, reusing the existing
scripts/primus/megatron-lm/run.sh (+11 lines for the new model mappings)
rather than introducing a parallel run.sh variant.

`mad-slurm-multinode` skill

SKILL.md + reference docs + helper scripts
(detect_cluster_env.sh, preflight.sh, validate_manifest.sh) that
bootstrap madengine on a fresh node and drive the native launchers above.
Cluster-agnostic mad.env templates + manifest templates for
CX7/Mellanox-RoCE, AMD-AINIC/Pollara, and Broadcom-Thor2-RoCE.
Sanitized per-archetype walkthroughs (no real node names, queues, or
tokens).

Model configurations

models.json: SGLang disaggregated DeepSeek-R1 inference, and Primus
Megatron-LM scaleout training for Llama-3.1 8B/70B/405B.

Validated

Ran both workloads end-to-end on a CX7/Mellanox-RoCE (OCI) cluster via the
mad-slurm-multinode skill, specifically to validate the native-launcher +
local-image-distribution path and the ENABLE_RDMA62 Dockerfile merge:

Llama-3.1-8B / Primus (2 nodes, 16 GPUs): 8/8 configs green. No
rendezvous/DistStoreError issues.
DeepSeek-R1 / sglang-disagg, 2P2D (4 nodes, 32 GPUs, image built with
ENABLE_RDMA62=1): 56/56 config rows green across a con(8/16/32/64) x
isl(1024/8192) sweep.

Both runs used only manifest-level configuration (node exclude lists, NFS
mountpoint-vs-subdirectory bind-mount fix — see the skill's gotchas.md) with
zero changes to madengine or MAD core scripts, confirming the native
integration works as designed on this archetype.

Also validated separately on an AMD-AINIC/Pollara (gfx950) cluster:
DeepSeek-R1/sglang-disagg (2-node, 1P1D, MoRI + Mooncake KV-transfer) confirmed
end-to-end (56/56 config rows green), and Primus Llama-3.1 scaleout training
confirmed working on the same archetype as well.

…ooling Introduce single full-overlay Dockerfiles for SGLang disaggregated prefill/decode inference that merge the RCCL, MoRI, and NIXL/Mooncake KV-transfer layers into one build (no base-image chaining), plus the supporting run and benchmark scripts. Key changes: - New Dockerfiles: `sglang_disagg_inference_full_overlay.ubuntu.amd.Dockerfile` and the `*.oci-rdma62.*` variant (rdma-core v62 baked in for OCI-CX7 hosts). - `run.sh` native launcher with in-container rank-ordered node-IP discovery (`ip_rendezvous.py`), so `IPADDRS`/`SGLANG_NODE_IPS` need not be forwarded. - `parse_to_csv.py` rewritten for comprehensive metric extraction (best-throughput iteration) into the madengine perf-CSV schema. - Resilient benchmark sweep in `benchmark_xPyD.sh` (fail-fast + point retries) writing to the madengine-expected perf CSV path. - Updated `sglang_disagg_mori_io_ep.sh` / `sglang_disagg_server.sh` and README. Co-authored-by: Ilia Kosarev <Ilia.Kosarev@amd.com>

…ripts Add a Primus Megatron-LM multi-node scaleout training image (candidate RCCL built from source) and the benchmark scripts that drive it and convert training output into the madengine perf-CSV format. Key changes: - New overlay Dockerfile `primus_megatron_train_rccl_overlay.ubuntu.amd.Dockerfile` on the `rocm/primus:v26.4` base (RCCL built from source + librccl smifix, optional rdma-core for Broadcom Thor2 bnxt_re). - New scaleout scripts under `scripts/primus_scaleout/megatron-lm/`: `run.sh`, `primus_megatron-lm_benchmark_setup.sh`, `primus_megatron-lm_benchmark_report.sh`, and `primus_megatron-lm_benchmark_report.py` (covers Llama-3.1 8B/70B/405B). Co-authored-by: Ilia Kosarev <Ilia.Kosarev@amd.com>

Introduce the mad-slurm-multinode skill, which deploys and runs madengine performance tests on an unprepared SLURM cluster from scratch and launches multi-node runs from manifest templates. Key additions: - `SKILL.md` plus reference docs (cluster-types, deploy-bootstrap, manifests, launch-and-results, gotchas) and helper scripts (`detect_cluster_env.sh`, `preflight.sh`, `validate_manifest.sh`). - Cluster-agnostic `mad.env` templates for the CX7/Mellanox-RoCE, AMD-AINIC/Pollara, and Broadcom-Thor2-RoCE archetypes. - Manifest templates for the supported workloads: `sglang_disagg_deepseek-r1` and Primus Megatron-LM scaleout `primus_llama-3.1-8b`/`-70b` (on `rocm/primus:v26.4`). - Sanitized end-to-end example walkthroughs (no real node names, queues, or tokens) for each archetype. Co-authored-by: Ilia Kosarev <Ilia.Kosarev@amd.com>

Add model entries in `models.json` for the new inference and training workloads. Key additions: - SGLang disaggregated DeepSeek-R1 inference configuration. - Primus Megatron-LM scaleout training configurations for Llama-3.1 8B/70B/405B, each with its Dockerfile, scripts, and perf-result tracking. Co-authored-by: Ilia Kosarev <Ilia.Kosarev@amd.com>

Copilot

Pull request overview

This PR adds two new multi-node benchmark workloads (SGLang disaggregated prefill/decode inference and Primus Megatron-LM scaleout training) plus a new mad-slurm-multinode skill to bootstrap a fresh SLURM cluster and run supported workloads, along with corresponding models.json registrations and Docker overlay images.

Changes:

Introduces a madengine-native SGLang disagg entrypoint (scripts/sglang_disagg/run.sh) with in-container rank-ordered IP discovery and updated benchmark parsing/CSV emission.
Adds Primus Megatron-LM scaleout benchmark runner + reporting scripts and a build-verified RCCL overlay image based on rocm/primus:v26.4.
Adds the mad-slurm-multinode skill documentation, templates, and static manifest validator to deploy/run these workloads on multiple SLURM cluster archetypes.

Reviewed changes

Copilot reviewed 33 out of 34 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
scripts/sglang_disagg/sglang_disagg_server.sh	Adds KV transfer backend selection/validation and DeepSeek-R1 configs for the non-MoRI launcher.
scripts/sglang_disagg/sglang_disagg_mori_io_ep.sh	Moves runtime build-layer responsibilities into the image and improves benchmark failure propagation; adds KV backend note.
scripts/sglang_disagg/run.sh	New madengine bridge entrypoint; maps env/topology and performs in-container IP rendezvous + optional weight staging.
scripts/sglang_disagg/README.MD	Documents the madengine entrypoint flow, overlays, and updated parsing path.
scripts/sglang_disagg/parse_to_csv.py	Rewritten to extract a fuller metric set and emit madengine perf CSV rows per metric/config.
scripts/sglang_disagg/ip_rendezvous.py	New stdlib-only TCP rendezvous helper to reconstruct rank-ordered node IPs in-container.
scripts/sglang_disagg/benchmark_xPyD.sh	Makes the sweep more resilient (retries, fail-fast), aligns parsing/output with `MAD_OUTPUT_CSV`.
scripts/primus_scaleout/megatron-lm/run.sh	New Primus scaleout runner selecting model + precision combinations based on detected device.
scripts/primus_scaleout/megatron-lm/primus_megatron-lm_benchmark_setup.sh	New setup script for Primus benchmark prerequisites (tokenizers, repo prep).
scripts/primus_scaleout/megatron-lm/primus_megatron-lm_benchmark_report.sh	New Megatron-LM benchmark driver that launches training and writes perf CSVs.
scripts/primus_scaleout/megatron-lm/primus_megatron-lm_benchmark_report.py	New parser to convert Primus training logs into CSV metrics (incl. running-average variants).
models.json	Registers new SGLang disagg DeepSeek-R1 workload and Primus Megatron-LM scaleout workloads.
docker/sglang_disagg_inference_full_overlay.ubuntu.amd.Dockerfile	Adds merged “full overlay” image build (RCCL+MoRI+RIXL/NIXL+Mooncake) with sanity checks.
docker/sglang_disagg_inference_full_overlay.oci-rdma62.ubuntu.amd.Dockerfile	OCI variant that additionally bakes rdma-core v62 for specific host stacks.
docker/primus_megatron_train_rccl_overlay.ubuntu.amd.Dockerfile	Adds build-verified RCCL-from-source overlay + optional rdma-core source build for Primus training images.
.gitignore	Un-ignores checked-in skill manifest template JSON files under the skill assets path.
.claude/skills/mad-slurm-multinode/SKILL.md	Adds the new SLURM bootstrap + run skill with required inputs, workflow, and responsibilities.
.claude/skills/mad-slurm-multinode/scripts/validate_manifest.sh	Adds a GPU-free static manifest validator (placeholders, env consistency, asset resolution).
.claude/skills/mad-slurm-multinode/scripts/preflight.sh	Adds a preflight checker for docker/SLURM/git/python/conda/GPU SMI/HF token prerequisites.
.claude/skills/mad-slurm-multinode/scripts/detect_cluster_env.sh	Adds a node inspection helper to propose archetype-specific network/RDMA settings.
.claude/skills/mad-slurm-multinode/references/manifests.md	Adds manifest anatomy + fill checklist guidance (secrets handling, mounts, interface consistency).
.claude/skills/mad-slurm-multinode/references/launch-and-results.md	Adds run/aggregation guidance and failure triage for multi-node madengine runs.
.claude/skills/mad-slurm-multinode/references/gotchas.md	Adds cross-cutting + per-workload pitfalls and validated workarounds/expectations.
.claude/skills/mad-slurm-multinode/references/deploy-bootstrap.md	Adds detailed idempotent bootstrap steps for fresh nodes (clone, conda, install, env).
.claude/skills/mad-slurm-multinode/references/cluster-types.md	Documents archetype-specific transport settings (CX7/AINIC/Thor2) and validation guidance.
.claude/skills/mad-slurm-multinode/examples/thor2-bnxt-walkthrough.md	Adds sanitized end-to-end example for Broadcom Thor2 clusters.
.claude/skills/mad-slurm-multinode/examples/cx7-roce-walkthrough.md	Adds sanitized end-to-end example for CX7/Mellanox RoCE clusters.
.claude/skills/mad-slurm-multinode/examples/amd-ainic-walkthrough.md	Adds sanitized end-to-end example for AMD AINIC/Pollara clusters.
.claude/skills/mad-slurm-multinode/assets/manifests/sglang_disagg_deepseek-r1.template.json	Adds a filled-workload template manifest for SGLang disagg DeepSeek-R1.
.claude/skills/mad-slurm-multinode/assets/manifests/primus_llama-3.1-8b.template.json	Adds a filled-workload template manifest for Primus 8B scaleout.
.claude/skills/mad-slurm-multinode/assets/manifests/primus_llama-3.1-70b.template.json	Adds a filled-workload template manifest for Primus 70B scaleout.
.claude/skills/mad-slurm-multinode/assets/mad.env/mad.env.thor2-bnxt.template	Adds archetype `mad.env` template for Broadcom Thor2 environments.
.claude/skills/mad-slurm-multinode/assets/mad.env/mad.env.cx7-roce.template	Adds archetype `mad.env` template for CX7/Mellanox RoCE environments.
.claude/skills/mad-slurm-multinode/assets/mad.env/mad.env.amd-ainic.template	Adds archetype `mad.env` template for AMD AINIC/Pollara environments.

Comments suppressed due to low confidence (1)

scripts/sglang_disagg/sglang_disagg_mori_io_ep.sh:191

KV_TRANSFER_BACKEND is appended into PREFILL_MODEL_CONFIG/DECODE_MODEL_CONFIG and later executed via eval, but it is not validated. This allows invalid backends and can enable shell-token injection via KV_TRANSFER_BACKEND.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

raviguptaamd · 2026-07-03T00:29:56Z

Why is tis file added ?

raviguptaamd · 2026-07-03T00:30:49Z

Core sglanf_dissag is being changes needs thorough review

Fold the scaleout-only capabilities into the main Primus Megatron-LM benchmark scripts and remove the duplicated scripts/primus_scaleout tree (~940 lines of near-copy): - multinode-aware NNODES/GPUS_PER_NODE/NUM_GPUS with global_batch_size renormalization; single-node behavior is unchanged - run_primus() launch helper - amd-smi and MAD_SYSTEM_GPU_ARCHITECTURE device-detection fallback - Llama-3.1-405B branch with a missing-config guard - DeepSeek-V3 DeepEP opt-in via PRIMUS_USE_DEEPEP - trailing running-average throughput rows in the perf report (new run column) Repoint the three *_scaleout models in models.json to scripts/primus/megatron-lm/run.sh, keeping their rccl_overlay Dockerfile and scaleout tag.

Align the Mooncake/NIXL launcher topology with the MoRI EP launcher (sglang_disagg_mori_io_ep.sh): the router/proxy now runs on NODE_RANK=0 alongside the first prefill server instead of requiring a dedicated proxy node. Total nodes = xP + yD (was 1 + xP + yD), so node counting is consistent across both KV-transfer backends. - IP_ARRAY[0..xP-1] are prefill nodes, IP_ARRAY[xP..xP+yD-1] are decode. - Backend sglang servers listen on SERVER_PORT (3000); the router listens on ROUTER_PORT (2322) so prefill0 and the router can share NODE_RANK=0. - Proxy readiness loops and the socket barrier are reindexed accordingly. - Add an explicit out-of-range guard for NODE_RANK.

_get_run_metadata checked DP_MODE before RUN_MORI, so a Mooncake/NIXL run (RUN_MORI=0) with DP_MODE=1 was mislabelled "mori_dp". Decide on RUN_MORI first (DP_MODE only distinguishes mori_dp vs mori_io within the MoRI path), and fall back to the actual KV_TRANSFER_BACKEND (mooncake/nixl) otherwise. Defaults for RUN_MORI/DP_MODE aligned with run.sh (both 1).

R1: /run_logs must be a shared (NFS-backed) mount passed through the docker env because the disaggregated launchers write per-node readiness logs to /run_logs/${SLURM_JOB_ID} and the rank-0 proxy greps them across nodes. Disaggregated P/D always spans >=2 nodes (rank-0 prefill/proxy + at least one decode node), so a node-local /tmp fallback would silently break that cross-node rendezvous. run.sh now requires /run_logs and fails fast with an actionable message when it is missing or not writable (the previous silent /tmp fallback is removed). C3: benchmark_xPyD.sh no longer re-hardcodes /run_logs; it honors the RUN_LOG_DIR exported by run.sh (single source of truth), so its log lands in the same directory and the tee cannot fail the sweep.

The DeepSeek-R1 disagg template mounted /run_logs from a bare relative ./slurm_output/run_logs, mirroring the PR-174 review finding that a non-shared /run_logs breaks the cross-node readiness rendezvous (run.sh now requires /run_logs and fails fast otherwise, see commit a3c8bb5). A relative path resolves against whatever CWD the per-node madengine process happens to have, which is not guaranteed to be the same shared rundir on every node. Use ${SLURM_SUBMIT_DIR} instead of a manual <FILL_...> placeholder or a literal $WORKDIR. SLURM sets SLURM_SUBMIT_DIR to the directory `sbatch` was invoked from; madengine's slurm deployment runs `sbatch` with no `cwd=` override (deployment/slurm.py), and the skill's Step 6 always launches from $WORKDIR/rundir, so SLURM_SUBMIT_DIR already equals the shared rundir with zero manual filling. It propagates end-to-end (job script -> srun, no --export restriction -> final `docker run`, since additional_docker_run_options is concatenated unquoted and console.sh() runs with env=None i.e. the full inherited environment) so it reliably expands at every node. $WORKDIR itself is only a doc-only convention; nothing in mad.env/madengine actually exports it as a real env var. Drop the redundant /run_logs entry from docker_mounts: it was always excluded at runtime (container_runner._extract_additional_mount_targets skips duplicate container targets already present in additional_docker_run_options), and docker_mounts values are rendered through shlex.quote(), which single-quotes the whole string and would have silently blocked ${SLURM_SUBMIT_DIR} expansion there anyway. Document the mechanism (and that run.sh's /run_logs check only verifies writability, not actual cross-node sharedness) in references/gotchas.md so it isn't rediscovered per-cluster.

Adapt the fixes from #1 to the current branch. - docker/primus_megatron_train_rccl_overlay.ubuntu.amd.Dockerfile: after the targeted /opt/rocm + torch/lib overwrites, sweep every non-symlink librccl.so* on disk and overwrite it with the candidate (+add-needed librocm_smi64), and tighten the final gate to assert EVERY on-disk librccl (not just 3 fixed paths) is RCCL 2.x with a baked git hash matching the built SHA. v26.4 splits ROCm libs into _rocm_sdk_libraries/lib (runtime) and _rocm_sdk_devel/lib (dev); torch maps librccl.so.1 from _rocm_sdk_libraries at runtime, which the targeted overwrites did not cover, so the stock base librccl silently ran and invalidated the overlay validation. - assets/manifests/primus_llama-3.1-8b.template.json: set NCCL_NET_PLUGIN=none in both env blocks. rocm/primus:v26.4 defaults NCCL_NET_PLUGIN=librccl-anp.so, which deadlocks RCCL init when the image carries a bundled RCCL overlay. - references/gotchas.md: document the NCCL_NET_PLUGIN pitfall, and the GBS normalization requirement (adapted to this branch: the GBS fix from the PR's scripts/primus_scaleout/megatron-lm/primus_megatron-lm_benchmark_report.sh is already generalized here by scaleout_gbs_override in scripts/primus/megatron-lm/primus_megatron-lm_benchmark_report.sh after the primus_scaleout consolidation, so no per-branch code change is ported). Ref: #1

…gatron-lm The primus_scaleout/megatron-lm tree was consolidated into scripts/primus/megatron-lm (commit 72e7a71; models.json repointed the *_scaleout models to scripts/primus/megatron-lm/run.sh), but the skill still referenced the removed path. Fix the stale `scripts/primus_scaleout/megatron-lm/run.sh` references: - primus_llama-3.1-8b.template.json / primus_llama-3.1-70b.template.json: built_models.scripts -> scripts/primus/megatron-lm/run.sh - references/deploy-bootstrap.md: the asset-existence check -> same path Model names (..._scaleout), the scaleout terminology, and the scaleout_gbs_override helper are intentionally left unchanged; only the filesystem paths moved.

…nto _rocm_sdk_devel/bin) Adapted from mkuznet1/MAD PR#1 commit 6475da3. v26.4 ships ROCm as pip wheels; the amdclang++ wrapper in _rocm_sdk_devel/bin/ looks for its clang++/clang-23 helpers next to itself, but they live only in _rocm_sdk_devel/lib/llvm/bin/. RCCL's device-code compile calls bin/amdclang++ directly (--offload-device-only) and fails: amdclang++: binary '.../_rocm_sdk_devel/bin/clang++' does not exist. Symlink clang/clang++/clang-23 into bin/ so the wrapper resolves. Guarded to be a no-op on v26.3 / non-wheel bases.

basemam

This is stale code and no longer used will be deleted in the sglang_disagg PR

Do not use. Functionalty already merged in sglang_disagg_mori_io_ep.sh

basemam

Duplicate and not needed simply add your model to run_xPyD_models.slurm

* Fix SGLang disagg on ionic/RoCE fabrics: preserve AINIC transport env MOONCAKE_COOKBOOK's set_env_vars.sh assumes a Mellanox/mlx5 fabric and, on ionic-based RoCE clusters (mia1), disables RDMA (NCCL_IB_DISABLE=1), hardcodes mlx5 device names, and mangles NCCL_SOCKET_IFNAME. Save the launcher-provided transport iface before sourcing, then re-assert NCCL_IB_DISABLE=0, IBDEVICES from IB_DEVICES, and NCCL/GLOO socket ifnames afterward. Also derive host_ip from the transport interface instead of the first hostname -I address. Validated on mia1 (DeepSeek-R1-0528, 1P+1D disagg, TP=8, RCCL 78e8ba0 overlay + AINIC ionic RDMA), SLURM job 11211: both servers fired up, router bound on 2322, full benchmark sweep 100% request success. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com> * sglang_disagg_server: source host_ip from IPADDRS[NODE_RANK] Re-deriving the node IP via ip addr/hostname -I risks picking a different NIC on multi-homed nodes, causing a mismatch between what peers registered in the router/barrier (built from IPADDRS) and what --host / socket_barrier --local-ip binds to. Source host_ip from IPADDRS[NODE_RANK] instead — the rank-ordered, post-rendezvous list that run.sh already resolved and passed in. Keep a hostname -I fallback for environments where IPADDRS is unset. Remove the now-redundant IFS/read of IP_ARRAY in the Cluster Topology section; the array is already populated at host_ip derivation time. Suggested-by: mkuznet1 Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> --------- Co-authored-by: ilkosare@amd.com <ilkosare@amd.com@mia1-vm-amd-prj3-k8s-004.amd3.mia.tensorwave.lan> Co-authored-by: Claude Opus 4 <noreply@anthropic.com>

sglang_disagg_mori_io_ep.sh is a functional superset of server.sh (models.yaml-driven config, mori/mooncake/nixl backends via KV_TRANSFER_BACKEND, DP_MODE support), and run_xPyD_models.slurm already always uses it. run.sh was the only remaining consumer of server.sh, so route both RUN_MORI branches to the unified launcher and drop the duplicate. Also port server.sh's KV_TRANSFER_BACKEND allowlist guard (against eval injection) into mori_io_ep.sh, and update the README and mad-slurm-multinode skill docs accordingly. Addresses PR ROCm#174 review feedback (basemam #1: stale/duplicate server.sh).

The two full-overlay Dockerfiles were ~92% identical (270 vs 250 lines, only 23 lines differed): the rdma-core-v62 variant was a straight copy of the base overlay plus one appended stage. Fold that stage into the base Dockerfile behind an empty-by-default ARG (ENABLE_RDMA62), mirroring the existing RDMA_CORE_VERSION-gated optional stage in docker/primus_megatron_train_rccl_overlay.ubuntu.amd.Dockerfile, and drop the now-redundant variant file. Net: one Dockerfile instead of two near-duplicates, -231 lines. Also drop vendor/cluster-specific naming ("OCI-CX7") from the affected docs and manifest template, describing the requirement generically (hosts needing a newer rdma-core than the base image ships) to match the skill's existing hardware-archetype convention rather than naming a specific cluster.

mkuznet1 · 2026-07-08T14:42:39Z

This PR is out of Draft — updated the description above to lead with the
actual point of the change (not just "two new benchmarks"), and it now covers
the review feedback so far. TL;DR:

This PR is the MAD side of a parallel integration effort with
madengine — the sglang-disagg templated launcher's co-located-proxy
topology (madengine#149) and multi-node local-image distribution
(_ensure_local_image_available, madengine#142) were built alongside the
MAD-side model scripts, Dockerfiles, and skill in this PR, using SGLang
disaggregated DeepSeek-R1 inference and Primus Megatron-LM scaleout
training as the two workloads that exercise it end-to-end. See the
"Why madengine-native, not run_xPyD_models.slurm / slurm_multi" section
for the full reasoning.
sglang_disagg_server.sh is removed — @basemam, this addresses your first
review comment; run.sh now routes both MoRI and Mooncake/NIXL through the
single sglang_disagg_mori_io_ep.sh launcher.
The duplicate *.oci-rdma62.* Dockerfile variant is gone too — merged into
the base full-overlay Dockerfile behind an ENABLE_RDMA62 build arg (same
pattern the Primus overlay already uses for RDMA_CORE_VERSION).
Both workloads were re-validated end-to-end on a CX7/Mellanox-RoCE cluster
after these changes (see the description's "Validated" section), and the
sglang-disagg path was separately confirmed on an AMD-AINIC/Pollara
(gfx950) cluster too — no MAD/madengine core-code changes were needed,
only manifest-level config.

@basemam @raviguptaamd — could you take another pass, especially on the
"Why madengine-native" section, which is meant to directly answer "why is
run.sh needed" / "duplicate, just use run_xPyD_models.slurm"?
cc @lcskrishna

This PR moved the PyYAML install for models.yaml parsing from a runtime `pip install pyyaml` into the full-overlay Dockerfile's build layer, but 16 pre-existing pyt_sglang_disagg_* model entries in models.json still use the base (non-overlay) docker/sglang_disagg_inference.ubuntu.amd.Dockerfile via run_xPyD_models.slurm, which always routes through this same script and does not install PyYAML at build time either. Add an explicit `import yaml` check before the eval-based YAML parse, so a missing PyYAML fails with a clear message instead of leaving MODEL_* flags silently unset (Copilot).

- primus_megatron-lm_benchmark_report.sh: when scaleout_gbs_override adjusts global_batch_size for NUM_GPUS>8, the GBS variable is now reassigned via the new effective_global_batch_size() helper before the perf CSV is written, in all three scaleout branches (8B/70B/405B). Previously the perf CSV kept reporting the original, pre-override GBS even when the actual run used the renormalized value. - mad.env.{cx7-roce,amd-ainic,thor2-bnxt}.template: warn on stderr when ~/.huggingface/token is missing or empty before exporting MAD_SECRETS_HFTOKEN, so a misconfigured token is caught immediately instead of failing obscurely later during HF downloads. Uses a warning (not exit), since these files are meant to be sourced and an exit would kill the caller's shell.

…dels The previous fail-fast fix (15dfa69) only turned a missing PyYAML into a clearer error, but didn't actually restore the ability for the 16 pre-existing pyt_sglang_disagg_* models.json entries (which use the base, non-overlay docker/sglang_disagg_inference.ubuntu.amd.Dockerfile via run_xPyD_models.slurm) to run at all, since that image still lacks PyYAML. Restore a runtime `pip install pyyaml` as a fallback when `import yaml` fails -- matching the behavior on develop before this PR moved the install into the full-overlay Dockerfile's build layer -- and only fail fast if the runtime install itself doesn't succeed (e.g. no network).

- run.sh: NNODES no longer falls back to WORLD_SIZE. Elsewhere in this repo WORLD_SIZE means total ranks (GPUS_PER_NODE*NNODES), not node count, so a stray WORLD_SIZE would make ip_rendezvous.py wait for far too many peers and hang. Fall back to SLURM_JOB_NUM_NODES, then xP+yD, both of which are node counts. - sglang_disagg_mori_io_ep.sh: quote the interpolated model_name/config_path in the "model not found" error path of the YAML-to-shell snippet. That string is eval'd by the shell, so unescaped values could break the launcher or inject shell tokens; it now reuses the same shlex.quote helper (moved above its first use) as the rest of the emitted exports.

The comment above _tcp_discover_ipaddrs claims ip_rendezvous.py is "shared (duplicated, kept identical) with scripts/vllm_dissag/", but no such copy exists there. Same stale reference already removed from the ip_rendezvous.py docstring; drop it here too so it doesn't send maintainers looking for a non-existent file (Copilot). Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

- scripts/sglang_disagg/run.sh: default DP_MODE to 0 instead of 1, matching README.MD and sglang_disagg_mori_io_ep.sh's own default. With the old default, running this entrypoint for any non-DeepSeek model without an explicit DP_MODE=0 would fail fast against the launcher's DP_MODE=1 allowlist (DeepSeek-V3/R1 only). The registered DeepSeek-R1 model passes --dp-mode 1 explicitly, so its behavior is unchanged. - scripts/primus/megatron-lm/primus_megatron-lm_benchmark_report.sh: - remove a stale TRAIN_LOG at the start of the Llama-3.1-405B branch. That branch skips run_primus when the config is missing or the datatype is unsupported, so without clearing the log a re-run from the same directory could benchmark/parse a stale training log from a previous run. - default MBS/GBS to NA in the 405B branch before writing the perf CSV, so a missing config or unsupported dtype writes explicit NA markers instead of placeholder rows with empty batch_size/global_batch_size fields. - .claude/skills/mad-slurm-multinode/assets/manifests/primus_llama-3.1-70b.template.json: add NCCL_NET_PLUGIN=none to both context.docker_env_vars and deployment_config.env_vars, matching the 8B template and references/gotchas.md. On rocm/primus:v26.4 RCCL-overlay images the base librccl-anp.so plugin must be disabled or RCCL can hang on the first collective, so the 70B multi-node perf run never starts.

_get_run_metadata() defaulted DP_MODE to "1", which this PR had changed from the pre-PR default of "0". With RUN_MORI defaulting to "1", a run parsed without DP_MODE exported (standalone/offline parsing) was tagged mori_dp instead of mori_io, disagreeing with run.sh's DP_MODE=0 default. Restore the "0" default so the metadata backend tag matches the launcher.

Copilot

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 1 comment.

mkuznet1 and others added 4 commits July 1, 2026 17:51

i-kosarev requested a review from Copilot July 1, 2026 18:26

Copilot started reviewing on behalf of i-kosarev July 1, 2026 18:27 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

raviguptaamd requested changes Jul 3, 2026

View reviewed changes

i-kosarev mentioned this pull request Jul 3, 2026

Fix Primus Llama-3.1-8B scaleout: NCCL_NET_PLUGIN for v26.4 RCCL overlay etc. mkuznet1/MAD#1

Closed

5 tasks

mkuznet1 added 8 commits July 3, 2026 12:34

basemam suggested changes Jul 7, 2026

View reviewed changes

i-kosarev and others added 3 commits July 7, 2026 16:02

mkuznet1 marked this pull request as ready for review July 8, 2026 14:42

mkuznet1 requested a review from gargrahul as a code owner July 8, 2026 14:42

Copilot AI review requested due to automatic review settings July 8, 2026 14:42

mkuznet1 requested review from Rohan138, amathews-amd, coketaste and ppalaniappan-amd as code owners July 8, 2026 14:42

Copilot AI review requested due to automatic review settings July 9, 2026 12:09

Copilot started reviewing on behalf of mkuznet1 July 9, 2026 12:10 View session