ROCm · raviguptaamd · Jul 12, 2026
@@ -3420,6 +3420,68 @@
     },
     "args": "-N 2 -n 2"
   },
+  {
+    "name": "pyt_vllm_disagg_mori_hy3-preview_1p1d",
+    "url": "",
+    "dockerfile": "docker/vllm_disagg_inference",
+    "scripts": "scripts/vllm_dissag/run_xPyD_models.slurm",
+    "data": "huggingface",
+    "n_gpus": "-1",
+    "owner": "mad.support@amd.com",
+    "training_precision": "",
+    "tags": [
+      "pyt",
+      "vllm",
+      "vllm_disagg",
+      "moriio",
+      "inference"
+    ],
+    "timeout": -1,
+    "distributed": {
+      "launcher": "slurm_multi"
+    },
+    "env_vars": {
+      "DOCKER_IMAGE_NAME": "<supply-your-image>",
+      "MODEL_NAME": "Hy3-preview",
+      "RUN_MORI": "1",
+      "RUN_DEEPEP": "0",
+      "xP": "1",
+      "yD": "1",
+      "BENCHMARK_COMBINATIONS": "1024/1024"
+    },
+    "args": "-N 2 -n 2"
+  },
+  {
+    "name": "pyt_vllm_disagg_mori_hy3-preview_2p2d",
+    "url": "",
+    "dockerfile": "docker/vllm_disagg_inference",
+    "scripts": "scripts/vllm_dissag/run_xPyD_models.slurm",
+    "data": "huggingface",
+    "n_gpus": "-1",
+    "owner": "mad.support@amd.com",
+    "training_precision": "",
+    "tags": [
+      "pyt",
+      "vllm",
+      "vllm_disagg",
+      "moriio",
+      "inference"
+    ],
+    "timeout": -1,
+    "distributed": {
+      "launcher": "slurm_multi"
+    },
+    "env_vars": {
+      "DOCKER_IMAGE_NAME": "<supply-your-image>",
+      "MODEL_NAME": "Hy3-preview",
+      "RUN_MORI": "1",
+      "RUN_DEEPEP": "0",
+      "xP": "2",
+      "yD": "2",
+      "BENCHMARK_COMBINATIONS": "1024/1024"
+    },
+    "args": "-N 4 -n 4"
+  },
   {
     "name": "pyt_vllm_disagg_mori_deepseek-v3-5layer",
     "url": "",

@@ -0,0 +1,69 @@
+# Hunyuan-3.0 (Hy3-preview) — MoRI-EP Disaggregated WideEP Recipe
+
+Enablement of **tencent/Hy3-preview** (Hunyuan-3.0 preview) on the vllm_dissag MoRI-EP
+prefill/decode disaggregated WideEP stack. Hy3 is an 80-layer **GQA** MoE: 192 routed
+experts (8 active/token) + 1 shared expert, 256K native context.
+
+## Validated configurations (bf16 weights)
+
+Measured on AMD MI300X/MI308 (gfx942) + the MoRIIO connector (`CONNECTOR=moriio WIDE_EP=1`):
+
+| Topology | EP | experts/rank | Known-answer | NIAH (54K/128K/256K) | Verdict |
+|----------|----|----|--------------|----------------------|---------|
+| 1P/1D | 8  | 24 | 20/20 (100%) | 9/9 incl 256K deep | ✅ PASS |
+| 2P/2D | 16 | 12 | 20/20 (100%) | 9/9 incl 256K deep | ✅ PASS |
+| 2P/1D, 1P/2D | 16 | — | perf 16/0 | — | ✅ serving |
+| 4P/4D | 32 | 6  | 0/20 (garbage) | — | ❌ see note |
+
+**EP8 and EP16 are production-correct and 256K-capable.** NIAH retrieves the needle at
+every depth up to 256K with fp8 KV cache + chunked prefill.
+
+**EP32 note:** 4P/4D hits the known MoRI-EP all-to-all `>2-pod` garbage bug (silent —
+HTTP 200 + "successful requests" but degenerate repeated-token output). Same class as the
+DeepSeek pre-fix issue; it is in the MoRI-EP all2all compute path, not the model/router/KV.
+Use EP8/EP16 until the upstream MoRI fix lands. **Accuracy MUST be gated separately** — perf
+benchmarks are blind to this (see `BENCHMARK_SCRIPT=accuracy`).
+
+## How to run
+
+```bash
+# EP16 2P/2D accuracy (known-answer + NIAH):
+export DOCKER_IMAGE_NAME=<your-moriep-image>
+export MODEL_NAME=Hy3-preview CONNECTOR=moriio WIDE_EP=1 EP_BACKEND=mori
+export xP=2 yD=2 BENCHMARK_SCRIPT=accuracy
+# ... via run_xPyD_models.slurm (slurm_multi launcher, -N 4 -n 4)
+
+# perf sanity:
+export BENCHMARK_SCRIPT=long_context BENCHMARK_COMBINATIONS=1024/1024
+```
+
+Or via `models.json`: `pyt_vllm_disagg_mori_hy3-preview_1p1d` / `_2p2d`.
+
+## Recipe knobs (models.yaml `Hy3-preview` → `_hy3_recipe_env`)
+
+The single-home recipe lives in `models.yaml`. Deltas vs the DeepSeek (MLA) recipe, because
+Hy3 is **GQA** (head_size 128):
+
+| Knob | Value | Why |
+|------|-------|-----|
+| `KV_BLOCK_SIZE` | 16 | block=1 has no valid ROCm attention backend under fp8 KV |
+| `KV_CACHE_DTYPE` | fp8 | KV cache in fp8 (does not degrade NIAH; 9/9 at 256K) |
+| `KV_CACHE_MEMORY_BYTES` | 48e9 | GQA KV ≈ 40 GiB @256K (vs MLA's compressed KV ~20 GiB); also skips the boot profiling forward |
+| `VLLM_ROCM_USE_AITER_MLA` | 0 | Hy3 is GQA (no MLA path); also avoids the fp8 MLA decode kernel GPU-fault |
+| `PREFILL_CUDAGRAPH_MODE` | NONE | prefill cudagraph capture deadlocks wide-DP |
+| `DECODE_CUDAGRAPH_MODE` | PIECEWISE | decode ITL win |
+| `PREFILL_MORI_BACKEND` | mori_high_throughput | InterNodeV1 |
+| `DECODE_MORI_BACKEND` | mori_low_latency | InterNodeV1LL |
+
+## Accuracy tooling (`HY_Accuracy_folder/`)
+
+`benchmark_accuracy_hy3.sh` runs three tiers (all greedy/deterministic):
+1. **known-answer** (`accuracy_eval.py --known`): ~20 factual/math prompts vs ground truth
+   — the gross-correctness gate that catches wideEP garbage (`0%` when the MoE path is wrong).
+2. **NIAH** (`niah_probe.py`): needle retrieval at 54K/128K/256K × depths 0.1/0.5/0.9 —
+   long-context KV-path correctness.
+3. **equivalence** (optional): greedy exact-match vs an EP16 golden.
+
+Note: the sanity probe uses a generous per-attempt budget (8×240s) because the FIRST
+inference after "Ready" triggers last-mile AITER JIT compile (minutes); a short probe
+would time out and abort the run.
@@ -0,0 +1,44 @@
+# HY_Accuracy_folder — Hy3 WideEP accuracy test suite
+
+Self-contained accuracy tooling for the MoRI-EP disaggregated stack. Validates that a
+disagg config produces *correct* output, not just HTTP-200 throughput — critical because
+some configs (e.g. EP32) emit silent KV-corruption garbage while perf benchmarks still
+report "successful". Used to establish EP16 = 100% vs EP32 = 0% on Hy3-preview.
+
+## Files
+| File | Purpose |
+|---|---|
+| `benchmark_accuracy_hy3.sh` | Orchestrator. Runs the tiers below + prints a combined PASS/FAIL verdict. Invoked by the launcher via `BENCHMARK_SCRIPT=accuracy`. |
+| `accuracy_eval.py` | Tier 1 — known-answer set (~20 factual/math/code prompts), scored vs ground truth. (Optional GSM8K mode behind `--gsm8k N`.) |
+| `niah_probe.py` | Tier 2 — needle-in-haystack long-context retrieval at 54k/128k/256k. |
+| `accuracy_probe.py` | Tier 3 — greedy exact-match equivalence vs a golden config (capture/compare). |
+| `benchmark_hold.sh` | Debug helper — `BENCHMARK_SCRIPT=hold` keeps the server up N hours for live poking (does not score). |
+
+## How to run
+Via the launcher (in-container, recommended):
+```bash
+export BENCHMARK_SCRIPT=accuracy        # selector wired in run_xPyD_models.slurm
+# ... normal MODEL_NAME=Hy3-preview / RUN_MORI=1 / xP / yD submit ...
+```
+Or manually against a serving disagg server (run INSIDE the container):
+```bash
+bash HY_Accuracy_folder/benchmark_accuracy_hy3.sh http://127.0.0.1:30000 <tag> [golden.json]
+```
+
+## CRITICAL — disagg request protocol
+The MoRIIO router only injects KV-routing (prefill/decode addressing) for requests that
+match the `vllm bench serve` client shape. Raw HTTP probes crash the decode engine with
+`KeyError: 'remote_host'`. The probes therefore send:
+- `"stream": true` (parse SSE chunks), and
+- an explicit `x-request-id` header.
+Run them from **inside the serving container** (`docker exec`) against `127.0.0.1:30000`,
+not from the host.
+
+## Scoring
+Greedy (temperature 0), deterministic. Known-answer scoring matches the **first answer
+line** (stop at `\n`) with **word boundaries**, so a short answer like `3` must appear as
+the answer token — not buried in over-generated continuation or coincidental garbage.
+
+## Status
+Known-answer tier validated (EP16 100% / EP32 0%). NIAH + equivalence tiers included;
+NIAH not yet run end-to-end at the time of writing.