vllm_dissag: add Hunyuan-3.0 (Hy3-preview) bf16 MoRI-EP wideEP recipe#177
Open
raviguptaamd wants to merge 1 commit into
Open
vllm_dissag: add Hunyuan-3.0 (Hy3-preview) bf16 MoRI-EP wideEP recipe#177raviguptaamd wants to merge 1 commit into
raviguptaamd wants to merge 1 commit into
Conversation
Adds the disaggregated prefill/decode WideEP recipe for tencent/Hy3-preview (80-layer GQA MoE, 192 experts/8-active + 1 shared, 256K ctx) on the moriio connector (RUN_MORI=1 / CONNECTOR=moriio WIDE_EP=1). Validated (bf16, MoRIIO connector, MI300X/MI308): EP8 1P/1D : known-answer 20/20, NIAH 9/9 incl 256K deep -> PASS EP16 2P/2D : known-answer 20/20, NIAH 9/9 incl 256K deep -> PASS EP32 4P/4D : known MoRI-EP >2-pod all2all garbage bug (deferred; see HY3_RECIPE.md). Changes (minimal, follows existing DeepSeek-V3 moriio pattern): - run_xPyD_models.slurm: add Hy3-preview to VALID_MODELS + MORI_EP_VALID_MODELS gates; add BENCHMARK_SCRIPT=accuracy selector. - models.yaml: Hy3-preview entry + _hy3_recipe_env (GQA deltas vs DeepSeek/MLA: KV_BLOCK_SIZE=16, KV_CACHE_MEMORY_BYTES=48e9 for GQA KV @256k, AITER_MLA=0, per-role MoRI HT/LL backends + cudagraph NONE/PIECEWISE). - models.json: pyt_vllm_disagg_mori_hy3-preview_{1p1d,2p2d} (RUN_MORI=1, like the existing mori disagg entries). - HY_Accuracy_folder/: known-answer + NIAH(256K) accuracy suite -- the correctness gate for wideEP (mainline benchmark_niah caps at 35k and has no scored gate; wideEP can emit silent garbage that perf/NIAH-retrieval alone miss). - HY3_RECIPE.md: recipe doc, validated results, EP32 note, knob rationale.
raviguptaamd
requested review from
Rohan138,
amathews-amd,
coketaste,
gargrahul and
ppalaniappan-amd
as code owners
July 13, 2026 04:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the Hunyuan-3.0 (
tencent/Hy3-preview) bf16 recipe to thevllm_dissagMoRI-EP prefill/decode disaggregated WideEP stack. Hy3 is an 80-layer GQA MoE (192 routed experts + 1 shared, 256K native context). Follows the existing DeepSeek-V3 moriio single-home pattern — minimal deltas over the mainline.Validated on AMD MI300X/MI308 (gfx942) on both a fullsource MoRI-EP image and the mainline
docker/vllm_disagg_inference.ubuntu.amd.Dockerfile(MoRI v1.2.1 / AITER 0.1.16.post3 / vLLM 06/29).Validated configurations (bf16)
EP8 and EP16 are production-correct and 256K-capable. Mainline-image PR gate: 1P/1D known-answer 20/20.
EP32 note: 4P/4D hits the known MoRI-EP all-to-all
>2-podgarbage bug (silent — HTTP 200 + "successful requests" but degenerate repeated-token output), in the MoRI-EP all2all compute path (not the model/router/KV). Same class as the DeepSeek pre-fix issue; PR#176 defers the same. Use EP8/EP16 until the upstream MoRI fix lands. Accuracy MUST be gated separately — perf benchmarks are blind to this (BENCHMARK_SCRIPT=accuracy).Changes (+698 / −1, 8 files)
scripts/vllm_dissag/models.yaml—Hy3-previewentry +_hy3_recipe_envanchor (GQA knobs:KV_BLOCK_SIZE=16,KV_CACHE_DTYPE=fp8,KV_CACHE_MEMORY_BYTES,VLLM_ROCM_USE_AITER_MLA=0, prefill/decode cudagraph + MoRI backends).scripts/vllm_dissag/run_xPyD_models.slurm—Hy3-previewadded toVALID_MODELS+MORI_EP_VALID_MODELSgates;accuracy→benchmark_accuracy_hy3.shselector.models.json—pyt_vllm_disagg_mori_hy3-preview_{1p1d,2p2d}(RUN_MORI=1 convention).scripts/vllm_dissag/HY_Accuracy_folder/— known-answer + NIAH(256K) suite (the wideEP correctness gate).scripts/vllm_dissag/HY3_RECIPE.md— recipe doc, validated results, EP32 note, knob rationale.Test plan
vllm_disagg_inferenceDockerfile image (1P/1D 20/20)🤖 Generated with Claude Code