Skip to content

Add DeepSeek-V4-Pro, Qwen3.5, and MiniMax-M3 inference recipes (scripts/)#173

Open
raviguptaamd wants to merge 8 commits into
ROCm:developfrom
raviguptaamd:add-deepseekv4-recipe
Open

Add DeepSeek-V4-Pro, Qwen3.5, and MiniMax-M3 inference recipes (scripts/)#173
raviguptaamd wants to merge 8 commits into
ROCm:developfrom
raviguptaamd:add-deepseekv4-recipe

Conversation

@raviguptaamd

@raviguptaamd raviguptaamd commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds three inference blueprints/recipes, linked from the top-level README Blueprints table:

  1. DeepSeek-V4-Pro disaggregated prefill/decode (P/D) inference on AMD Instinct MI355X, under scripts/DeepSeekV4/. Covers two transport backends:
    • ATOM + Mooncake (KV-cache transfer over RDMA)
    • SGLang + MoRI-IO
  2. Qwen3.5-35B-A3B single-node aggregated serving/benchmark (SGLang / vLLM), under scripts/Qwen3.5/. Recommended default is SGLang + MXFP4 + TP1 (8 single-GPU instances/node) — a 3B-active MoE fits one GPU, so filling the node with independent instances beats tensor-parallel. README includes a 4-step quickstart for the default.
  3. MiniMax-M3 disaggregated prefill/decode (P/D) inference on AMD Instinct MI355X, under scripts/MiniMax-M3/. ATOM MXFP4 config (minimaxm3-fp4-mi355x-atom-disagg, atomesh router + Mooncake RDMA) — the top-throughput MiniMax-M3 disagg config on InferenceX (~10,829 tok/s/GPU at 8k/1k, 2P1D+DPA). Engine code is copied verbatim from SemiAnalysisAI/InferenceX with a thin MAD wrapper (run.sh + cluster.yaml) so it runs standalone.

What's included

DeepSeekV4 (scripts/DeepSeekV4/)

  • README.md — usage and overview
  • cluster.yaml / model.yaml — cluster + model configuration
  • lib/ — launcher and helpers (run_disagg.sh, cfg.py, topo.py, check_accuracy.py, clean_node.sh, prompts.json, lib_inferencex.sh)
  • atom_disagg/ — ATOM+Mooncake env/server/launch/bench scripts
  • utils/bench_serving/ — serving benchmark client (throughput/latency)
  • docs/proxy_and_disagg.md — proxy + disaggregation notes
  • run_atom_disagg.sh / run_sglang_disagg.sh — entrypoints

Qwen3.5 (scripts/Qwen3.5/)

  • README.md — quickstart + recommended default (SGLang·MXFP4·TP1)
  • cluster.yaml / model.yaml — cluster + model config (MXFP4 + FP8), engine flags preset
  • lib/ — launcher and helpers (run_engine.sh, replica_entry.sh, placement.py, cfg.py, clean_node.sh, check_accuracy.py, prompts.json, lib_inferencex.sh)
  • run_sglang.sh / run_vllm.sh / run_atom.sh — entrypoints
  • utils/bench_serving/ — serving benchmark client

MiniMax-M3 (scripts/MiniMax-M3/)

  • README.md — nodes required, prerequisites (incl. amdgpu_peermem for GPUDirect RDMA), usage, topologies
  • run.sh / cluster.yaml — MAD launcher wrapper (supplies the per-node env + mounts InferenceX's job.slurm normally provides)
  • benchmarks/multi_node/amd_utils/ + benchmarks/benchmark_lib.sh — verbatim InferenceX ATOM code (server_atom.sh, models_atom.yaml, env_atom.sh, bench.sh, setup_deps.sh, server.sh, job.slurm, submit.sh, sync.py); models_atom.yaml drives all model flags
  • utils/bench_serving/ — serving benchmark client
  • configs/minimaxm3-fp4-mi355x-atom-disagg.yaml — config entry (all-TP4, STP; topologies 1p1d / 2p1d_dpa)

Topologies: 1p1d (2 nodes: 1 prefill + 1 decode) and 2p1d_dpa (3 nodes: 2 prefill + 1 decode + DP-attn).

@raviguptaamd
Add DeepSeek-V4-Pro disaggregated P/D inference recipe (scripts/DeepS…
83e63d0 
…eekV4)

Multi-node prefill/decode disaggregated serving/benchmark harness for
DeepSeek-V4-Pro (ATOM+mooncake primary; SGLang+MoRI experimental), ported
from InferenceX. Adds a Blueprints row to the top-level README.
@raviguptaamd @claude
Add Qwen3.5-35B-A3B aggregated inference recipe (scripts/Qwen3.5)
41070cb 
Single-node aggregated serving/benchmark harness for Qwen3.5-35B-A3B (3B-active
hybrid-GDN MoE) on AMD Instinct MI350 (gfx950), engines SGLang + vLLM.

Recommended default: SGLang + MXFP4 + TP1 (8 single-GPU instances/node) — a 3B-active
MoE fits one GPU, so fill the node with independent instances rather than tensor-parallel.
Findings: TP1 > TP2 > TP4; SGLang > vLLM; MXFP4 > FP8 (and ~half the HBM). Both engines
serve MXFP4 correctly (no garbling). ATOM excluded — crashes at ISL>=8192.

Code/scripts + README only (no result data). Engine flags baked into model.yaml per
AMD Day-0 recipe (SGLang --disable-radix-cache required for the hybrid-GDN arch;
vLLM --enable-expert-parallel --reasoning-parser qwen3). cluster.yaml uses placeholders.

Co-Authored-By: Claude <noreply@anthropic.com>
raviguptaamd and others added 6 commits July 8, 2026 19:10
@raviguptaamd @claude
Qwen3.5: add lib/ files (force-add past .gitignore lib/ rule)
c44d29b 
The repo .gitignore has `lib/` (Python build-dir convention), which silently
excluded scripts/Qwen3.5/lib/* from the prior commit. Force-add the 8 harness
lib files — run_engine.sh (core launcher), replica_entry.sh, cfg.py, clean_node.sh,
placement.py, check_accuracy.py, lib_inferencex.sh, prompts.json. Without these the
recipe cannot run.

Co-Authored-By: Claude <noreply@anthropic.com>
@raviguptaamd @claude
Qwen3.5 README: add highlighted 4-step Quickstart for the default config
61beced 
Makes the recommended SGLang+MXFP4+TP1 default runnable cold: download weights ->
edit cluster.yaml -> one run_sglang.sh command -> query. Spells out the prerequisites
(weights path, cluster.yaml placeholders, Slurm alloc) that were previously implicit.

Co-Authored-By: Claude <noreply@anthropic.com>
@claude
Add MiniMax-M3 MXFP4 disaggregated P/D inference recipe (scripts/Mini…
f2239a4 
…Max-M3)

Multi-node prefill/decode disaggregated serving/benchmark harness for
MiniMax-M3 (MXFP4) on AMD MI355X (gfx950) via ATOM (atomesh + mooncake RDMA),
ported from the InferenceX minimaxm3-fp4-mi355x-atom-disagg config (top-throughput
MiniMax-M3 disagg config on MI355X: ~10,829 tok/s/GPU at 8k/1k, 2P1D+DPA).

All workers TP4 (no EP; TP4 beats TP8/TP4-EP on gfx950); STP only (DECODE_MTP_SIZE=0).
Topologies 1p1d (2 nodes) and 2p1d_dpa (3 nodes). Mirrors the DeepSeekV4 harness
structure (model.yaml/cluster.yaml/lib/atom_disagg/utils). Scaffolded from InferenceX;
needs a live gfx950 dry-run once amd/MiniMax-M3-MXFP4 weights are staged.

Also adds the MiniMax-M3 row to the root README models table.

Co-Authored-By: Claude <noreply@anthropic.com>
@claude
MiniMax-M3: vendor InferenceX ATOM MXFP4 disagg code verbatim
32856d6 
Replace the hand-adapted MiniMax-M3 harness with the actual InferenceX ATOM
MXFP4 disaggregated benchmark, copied unmodified from SemiAnalysisAI/InferenceX
@ a174f20 (benchmarks/multi_node/amd_utils + utils/bench_serving). Config key
minimaxm3-fp4-mi355x-atom-disagg (atomesh + mooncake, all-TP4). Includes
models_atom.yaml (authoritative MiniMax-M3-MXFP4 flags), server_atom.sh,
env_atom.sh, bench.sh, setup_deps.sh, job.slurm/submit.sh/sync.py, the
bench_serving client, and the extracted config block under configs/.

Co-Authored-By: Claude <noreply@anthropic.com>
@claude
MiniMax-M3: make ATOM MXFP4 disagg recipe runnable standalone in MAD
5b40488 
Keep the InferenceX engine code verbatim (benchmarks/multi_node/amd_utils/,
benchmarks/benchmark_lib.sh, utils/bench_serving/ @ a174f20 — byte-identical,
models_atom.yaml drives all model flags) but preserve InferenceX's relative
path layout so bench.sh's ../../benchmark_lib.sh and REPO_ROOT/utils/bench_serving
resolve. Add a thin MAD wrapper (run.sh + cluster.yaml) that supplies the per-node
env contract + container mounts InferenceX's job.slurm/submit.sh normally provide,
mounting the recipe root as /workspace and ATOM_WS_PATH at amd_utils/.

README documents prerequisites (amdgpu_peermem for GPUDirect RDMA, weight staging,
RDMA fabric), usage (TOPO=1p1d|2p1d_dpa, ACTION=dry), and topologies. Wrapper is
syntax-checked; not yet dry-run on live MI355X.

Co-Authored-By: Claude <noreply@anthropic.com>
MiniMax-M3: trim README to plain runnable doc
da4c6c6
@raviguptaamd raviguptaamd changed the title Add DeepSeek-V4-Pro disaggregated P/D inference recipe (scripts/DeepSeekV4) Add DeepSeek-V4-Pro, Qwen3.5, and MiniMax-M3 inference recipes (scripts/) Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gargrahul gargrahul Awaiting requested review from gargrahul gargrahul is a code owner
@amathews-amd amathews-amd Awaiting requested review from amathews-amd amathews-amd is a code owner
@coketaste coketaste Awaiting requested review from coketaste coketaste is a code owner
@ppalaniappan-amd ppalaniappan-amd Awaiting requested review from ppalaniappan-amd ppalaniappan-amd is a code owner
@Rohan138 Rohan138 Awaiting requested review from Rohan138 Rohan138 is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@raviguptaamd