From 127f4b26b034fef8fd774a1983f5ee654d6af951 Mon Sep 17 00:00:00 2001 From: raviguptaamd Date: Sun, 12 Jul 2026 23:49:59 +0000 Subject: [PATCH] vllm_dissag: add Hunyuan-3.0 (Hy3-preview) bf16 MoRI-EP wideEP recipe Adds the disaggregated prefill/decode WideEP recipe for tencent/Hy3-preview (80-layer GQA MoE, 192 experts/8-active + 1 shared, 256K ctx) on the moriio connector (RUN_MORI=1 / CONNECTOR=moriio WIDE_EP=1). Validated (bf16, MoRIIO connector, MI300X/MI308): EP8 1P/1D : known-answer 20/20, NIAH 9/9 incl 256K deep -> PASS EP16 2P/2D : known-answer 20/20, NIAH 9/9 incl 256K deep -> PASS EP32 4P/4D : known MoRI-EP >2-pod all2all garbage bug (deferred; see HY3_RECIPE.md). Changes (minimal, follows existing DeepSeek-V3 moriio pattern): - run_xPyD_models.slurm: add Hy3-preview to VALID_MODELS + MORI_EP_VALID_MODELS gates; add BENCHMARK_SCRIPT=accuracy selector. - models.yaml: Hy3-preview entry + _hy3_recipe_env (GQA deltas vs DeepSeek/MLA: KV_BLOCK_SIZE=16, KV_CACHE_MEMORY_BYTES=48e9 for GQA KV @256K, AITER_MLA=0, per-role MoRI HT/LL backends + cudagraph NONE/PIECEWISE). - models.json: pyt_vllm_disagg_mori_hy3-preview_{1p1d,2p2d} (RUN_MORI=1, like the existing mori disagg entries). - HY_Accuracy_folder/: known-answer + NIAH(256K) accuracy suite -- the correctness gate for wideEP (mainline benchmark_niah caps at 35k and has no scored gate; wideEP can emit silent garbage that perf/NIAH-retrieval alone miss). - HY3_RECIPE.md: recipe doc, validated results, EP32 note, knob rationale. --- models.json | 62 +++++ scripts/vllm_dissag/HY3_RECIPE.md | 69 +++++ .../vllm_dissag/HY_Accuracy_folder/README.md | 44 ++++ .../HY_Accuracy_folder/accuracy_eval.py | 243 ++++++++++++++++++ .../benchmark_accuracy_hy3.sh | 139 ++++++++++ .../HY_Accuracy_folder/niah_probe.py | 90 +++++++ scripts/vllm_dissag/models.yaml | 44 ++++ scripts/vllm_dissag/run_xPyD_models.slurm | 8 +- 8 files changed, 698 insertions(+), 1 deletion(-) create mode 100644 scripts/vllm_dissag/HY3_RECIPE.md create mode 100644 scripts/vllm_dissag/HY_Accuracy_folder/README.md create mode 100644 scripts/vllm_dissag/HY_Accuracy_folder/accuracy_eval.py create mode 100755 scripts/vllm_dissag/HY_Accuracy_folder/benchmark_accuracy_hy3.sh create mode 100644 scripts/vllm_dissag/HY_Accuracy_folder/niah_probe.py diff --git a/models.json b/models.json index af914f4..fdedf18 100644 --- a/models.json +++ b/models.json @@ -3420,6 +3420,68 @@ }, "args": "-N 2 -n 2" }, + { + "name": "pyt_vllm_disagg_mori_hy3-preview_1p1d", + "url": "", + "dockerfile": "docker/vllm_disagg_inference", + "scripts": "scripts/vllm_dissag/run_xPyD_models.slurm", + "data": "huggingface", + "n_gpus": "-1", + "owner": "mad.support@amd.com", + "training_precision": "", + "tags": [ + "pyt", + "vllm", + "vllm_disagg", + "moriio", + "inference" + ], + "timeout": -1, + "distributed": { + "launcher": "slurm_multi" + }, + "env_vars": { + "DOCKER_IMAGE_NAME": "", + "MODEL_NAME": "Hy3-preview", + "RUN_MORI": "1", + "RUN_DEEPEP": "0", + "xP": "1", + "yD": "1", + "BENCHMARK_COMBINATIONS": "1024/1024" + }, + "args": "-N 2 -n 2" + }, + { + "name": "pyt_vllm_disagg_mori_hy3-preview_2p2d", + "url": "", + "dockerfile": "docker/vllm_disagg_inference", + "scripts": "scripts/vllm_dissag/run_xPyD_models.slurm", + "data": "huggingface", + "n_gpus": "-1", + "owner": "mad.support@amd.com", + "training_precision": "", + "tags": [ + "pyt", + "vllm", + "vllm_disagg", + "moriio", + "inference" + ], + "timeout": -1, + "distributed": { + "launcher": "slurm_multi" + }, + "env_vars": { + "DOCKER_IMAGE_NAME": "", + "MODEL_NAME": "Hy3-preview", + "RUN_MORI": "1", + "RUN_DEEPEP": "0", + "xP": "2", + "yD": "2", + "BENCHMARK_COMBINATIONS": "1024/1024" + }, + "args": "-N 4 -n 4" + }, { "name": "pyt_vllm_disagg_mori_deepseek-v3-5layer", "url": "", diff --git a/scripts/vllm_dissag/HY3_RECIPE.md b/scripts/vllm_dissag/HY3_RECIPE.md new file mode 100644 index 0000000..4272ee7 --- /dev/null +++ b/scripts/vllm_dissag/HY3_RECIPE.md @@ -0,0 +1,69 @@ +# Hunyuan-3.0 (Hy3-preview) — MoRI-EP Disaggregated WideEP Recipe + +Enablement of **tencent/Hy3-preview** (Hunyuan-3.0 preview) on the vllm_dissag MoRI-EP +prefill/decode disaggregated WideEP stack. Hy3 is an 80-layer **GQA** MoE: 192 routed +experts (8 active/token) + 1 shared expert, 256K native context. + +## Validated configurations (bf16 weights) + +Measured on AMD MI300X/MI308 (gfx942) + the MoRIIO connector (`CONNECTOR=moriio WIDE_EP=1`): + +| Topology | EP | experts/rank | Known-answer | NIAH (54K/128K/256K) | Verdict | +|----------|----|----|--------------|----------------------|---------| +| 1P/1D | 8 | 24 | 20/20 (100%) | 9/9 incl 256K deep | ✅ PASS | +| 2P/2D | 16 | 12 | 20/20 (100%) | 9/9 incl 256K deep | ✅ PASS | +| 2P/1D, 1P/2D | 16 | — | perf 16/0 | — | ✅ serving | +| 4P/4D | 32 | 6 | 0/20 (garbage) | — | ❌ see note | + +**EP8 and EP16 are production-correct and 256K-capable.** NIAH retrieves the needle at +every depth up to 256K with fp8 KV cache + chunked prefill. + +**EP32 note:** 4P/4D hits the known MoRI-EP all-to-all `>2-pod` garbage bug (silent — +HTTP 200 + "successful requests" but degenerate repeated-token output). Same class as the +DeepSeek pre-fix issue; it is in the MoRI-EP all2all compute path, not the model/router/KV. +Use EP8/EP16 until the upstream MoRI fix lands. **Accuracy MUST be gated separately** — perf +benchmarks are blind to this (see `BENCHMARK_SCRIPT=accuracy`). + +## How to run + +```bash +# EP16 2P/2D accuracy (known-answer + NIAH): +export DOCKER_IMAGE_NAME= +export MODEL_NAME=Hy3-preview CONNECTOR=moriio WIDE_EP=1 EP_BACKEND=mori +export xP=2 yD=2 BENCHMARK_SCRIPT=accuracy +# ... via run_xPyD_models.slurm (slurm_multi launcher, -N 4 -n 4) + +# perf sanity: +export BENCHMARK_SCRIPT=long_context BENCHMARK_COMBINATIONS=1024/1024 +``` + +Or via `models.json`: `pyt_vllm_disagg_mori_hy3-preview_1p1d` / `_2p2d`. + +## Recipe knobs (models.yaml `Hy3-preview` → `_hy3_recipe_env`) + +The single-home recipe lives in `models.yaml`. Deltas vs the DeepSeek (MLA) recipe, because +Hy3 is **GQA** (head_size 128): + +| Knob | Value | Why | +|------|-------|-----| +| `KV_BLOCK_SIZE` | 16 | block=1 has no valid ROCm attention backend under fp8 KV | +| `KV_CACHE_DTYPE` | fp8 | KV cache in fp8 (does not degrade NIAH; 9/9 at 256K) | +| `KV_CACHE_MEMORY_BYTES` | 48e9 | GQA KV ≈ 40 GiB @256K (vs MLA's compressed KV ~20 GiB); also skips the boot profiling forward | +| `VLLM_ROCM_USE_AITER_MLA` | 0 | Hy3 is GQA (no MLA path); also avoids the fp8 MLA decode kernel GPU-fault | +| `PREFILL_CUDAGRAPH_MODE` | NONE | prefill cudagraph capture deadlocks wide-DP | +| `DECODE_CUDAGRAPH_MODE` | PIECEWISE | decode ITL win | +| `PREFILL_MORI_BACKEND` | mori_high_throughput | InterNodeV1 | +| `DECODE_MORI_BACKEND` | mori_low_latency | InterNodeV1LL | + +## Accuracy tooling (`HY_Accuracy_folder/`) + +`benchmark_accuracy_hy3.sh` runs three tiers (all greedy/deterministic): +1. **known-answer** (`accuracy_eval.py --known`): ~20 factual/math prompts vs ground truth + — the gross-correctness gate that catches wideEP garbage (`0%` when the MoE path is wrong). +2. **NIAH** (`niah_probe.py`): needle retrieval at 54K/128K/256K × depths 0.1/0.5/0.9 — + long-context KV-path correctness. +3. **equivalence** (optional): greedy exact-match vs an EP16 golden. + +Note: the sanity probe uses a generous per-attempt budget (8×240s) because the FIRST +inference after "Ready" triggers last-mile AITER JIT compile (minutes); a short probe +would time out and abort the run. diff --git a/scripts/vllm_dissag/HY_Accuracy_folder/README.md b/scripts/vllm_dissag/HY_Accuracy_folder/README.md new file mode 100644 index 0000000..6b439b2 --- /dev/null +++ b/scripts/vllm_dissag/HY_Accuracy_folder/README.md @@ -0,0 +1,44 @@ +# HY_Accuracy_folder — Hy3 WideEP accuracy test suite + +Self-contained accuracy tooling for the MoRI-EP disaggregated stack. Validates that a +disagg config produces *correct* output, not just HTTP-200 throughput — critical because +some configs (e.g. EP32) emit silent KV-corruption garbage while perf benchmarks still +report "successful". Used to establish EP16 = 100% vs EP32 = 0% on Hy3-preview. + +## Files +| File | Purpose | +|---|---| +| `benchmark_accuracy_hy3.sh` | Orchestrator. Runs the tiers below + prints a combined PASS/FAIL verdict. Invoked by the launcher via `BENCHMARK_SCRIPT=accuracy`. | +| `accuracy_eval.py` | Tier 1 — known-answer set (~20 factual/math/code prompts), scored vs ground truth. (Optional GSM8K mode behind `--gsm8k N`.) | +| `niah_probe.py` | Tier 2 — needle-in-haystack long-context retrieval at 54k/128k/256k. | +| `accuracy_probe.py` | Tier 3 — greedy exact-match equivalence vs a golden config (capture/compare). | +| `benchmark_hold.sh` | Debug helper — `BENCHMARK_SCRIPT=hold` keeps the server up N hours for live poking (does not score). | + +## How to run +Via the launcher (in-container, recommended): +```bash +export BENCHMARK_SCRIPT=accuracy # selector wired in run_xPyD_models.slurm +# ... normal MODEL_NAME=Hy3-preview / RUN_MORI=1 / xP / yD submit ... +``` +Or manually against a serving disagg server (run INSIDE the container): +```bash +bash HY_Accuracy_folder/benchmark_accuracy_hy3.sh http://127.0.0.1:30000 [golden.json] +``` + +## CRITICAL — disagg request protocol +The MoRIIO router only injects KV-routing (prefill/decode addressing) for requests that +match the `vllm bench serve` client shape. Raw HTTP probes crash the decode engine with +`KeyError: 'remote_host'`. The probes therefore send: +- `"stream": true` (parse SSE chunks), and +- an explicit `x-request-id` header. +Run them from **inside the serving container** (`docker exec`) against `127.0.0.1:30000`, +not from the host. + +## Scoring +Greedy (temperature 0), deterministic. Known-answer scoring matches the **first answer +line** (stop at `\n`) with **word boundaries**, so a short answer like `3` must appear as +the answer token — not buried in over-generated continuation or coincidental garbage. + +## Status +Known-answer tier validated (EP16 100% / EP32 0%). NIAH + equivalence tiers included; +NIAH not yet run end-to-end at the time of writing. diff --git a/scripts/vllm_dissag/HY_Accuracy_folder/accuracy_eval.py b/scripts/vllm_dissag/HY_Accuracy_folder/accuracy_eval.py new file mode 100644 index 0000000..9325655 --- /dev/null +++ b/scripts/vllm_dissag/HY_Accuracy_folder/accuracy_eval.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python3 +"""Accuracy evaluation for a running vLLM (disagg/EP) server — SCORED, not just consistency. + +Two tiers, both greedy (temperature=0) so results are deterministic and comparable +across EP/KV configs: + + Tier 1 KNOWN-ANSWER set : ~20 fixed factual/math/code prompts with EXPECTED answers, + scored by substring/normalized match. Fast (seconds). Catches gross + corruption (the EP32 `!!!` case scores 0%) AND wrong-but-fluent output. + + Tier 2 GSM8K : grade-school math word problems pulled from HuggingFace + (`datasets` is in the image; no lm-eval needed). N-shot prompting, extract the + final integer, exact-match against the gold answer -> real accuracy %. + +Unlike accuracy_probe.py (which checks if config A == config B), this scores against +GROUND TRUTH, so a single config gets an absolute number (e.g. "EP16 = 92% GSM8K"). + +Usage: + # Tier 1 only (fast) + python3 accuracy_eval.py --url http://127.0.0.1:30000 --model --known --out acc_ep16.json + # Tier 1 + GSM8K (40 problems, 5-shot) + python3 accuracy_eval.py --url http://127.0.0.1:30000 --model \ + --known --gsm8k 40 --gsm8k-shots 5 --out acc_ep16.json + +Notes: + * --model must be the SERVED name. On this stack that is the full model PATH + (e.g. /mnt/m2m_nobackup/models_blog/Hy3-preview) unless --served-model-name was set. + Use --auto-model to query /v1/models and use the first id. + * GSM8K download needs network + `datasets`. If offline, pass --gsm8k-file + with {"question","answer"} rows. +""" +import argparse, json, re, sys, urllib.request + +# --------------------------------------------------------------------------- +# Tier 1 — known-answer probes. `match` semantics: +# list of acceptable substrings (case-insensitive); pass if ANY appears in output. +# --------------------------------------------------------------------------- +KNOWN = [ + {"prompt": "The capital of France is", "accept": ["paris"]}, + {"prompt": "The capital of Japan is", "accept": ["tokyo"]}, + {"prompt": "The largest planet in the solar system is", "accept": ["jupiter"]}, + {"prompt": "Water freezes at a temperature of", "accept": ["0", "zero", "32"]}, + {"prompt": "Question: What is 2+2? Answer:", "accept": ["4", "four"]}, + {"prompt": "Question: What is 2*3? Answer:", "accept": ["6", "six"]}, + {"prompt": "Question: What is 10-7? Answer:", "accept": ["3", "three"]}, + {"prompt": "Question: What is the square root of 144? Answer:", "accept": ["12", "twelve"]}, + {"prompt": "The chemical symbol for gold is", "accept": ["au"]}, + {"prompt": "The author of 'Romeo and Juliet' is", "accept": ["shakespeare"]}, + {"prompt": "Complete the sequence: 2, 4, 8, 16,", "accept": ["32"]}, + {"prompt": "The boiling point of water at sea level in Celsius is", "accept": ["100"]}, + {"prompt": "Translate 'Hello' to French:", "accept": ["bonjour"]}, + {"prompt": "Roses are red, violets are", "accept": ["blue"]}, + {"prompt": "The first president of the United States was", "accept": ["washington"]}, + {"prompt": "The speed of light is approximately", "accept": ["300", "3", "186", "299"]}, + {"prompt": "In Python, the keyword to define a function is", "accept": ["def"]}, + {"prompt": "The opposite of 'hot' is", "accept": ["cold"]}, + {"prompt": "How many days are in a week? Answer:", "accept": ["7", "seven"]}, + {"prompt": "The chemical formula for water is", "accept": ["h2o", "h₂o"]}, +] + +_REQ_N = [0] + +def call(url, model, prompt, max_tokens, stop=None): + # Use the EXACT protocol of `vllm bench serve` (the proven-stable client that + # ran 128 concurrent reqs with 0 crashes), which raw probes did NOT: + # (1) stream=True + parse SSE chunks + # (2) an explicit x-request-id header (prefix like the bench client's + # request_id_prefix). The MoRIIO router embeds the prefill/decode routing + # into the request id; without a client-provided id the bare auto-id gets + # no kv_transfer injection -> decode KeyError remote_host -> crash. + _REQ_N[0] += 1 + payload = {"model": model, "prompt": prompt, "max_tokens": max_tokens, + "temperature": 0.0, "stream": True} + if stop: + payload["stop"] = stop + data = json.dumps(payload).encode() + headers = {"Content-Type": "application/json", + "x-request-id": f"acc-{_REQ_N[0]:06d}"} + req = urllib.request.Request(url.rstrip("/") + "/v1/completions", + data=data, headers=headers) + text = "" + with urllib.request.urlopen(req, timeout=600) as r: + for raw in r: + line = raw.decode("utf-8", "replace").strip() + if not line.startswith("data:"): + continue + chunk = line[len("data:"):].strip() + if chunk == "[DONE]": + break + try: + text += json.loads(chunk)["choices"][0]["text"] + except Exception: + pass + return text + +def resolve_model(url, model, auto): + if model and not auto: + return model + req = urllib.request.Request(url.rstrip("/") + "/v1/models") + with urllib.request.urlopen(req, timeout=60) as r: + ids = [m["id"] for m in json.loads(r.read()).get("data", [])] + if not ids: + raise RuntimeError("no model ids from /v1/models") + return ids[0] + +def _scored_hit(out, accept): + # Strict-but-fair match: score the FIRST LINE only (the answer; we stop at \n), + # with word boundaries so a short answer like "3" must appear AS a token, not + # buried in coincidental garbage. Scoring the whole first line (not a fixed char + # window) avoids cutting off valid longer answers like "...is 12.". Avoids both + # failure modes of bare substring-anywhere: (1) over-generated extra Q/A after the + # answer (excluded by stop=\n), (2) degenerate garbage with a stray digit + # (excluded by the word boundary). + head = out.strip().splitlines()[0].lower() if out.strip() else "" + for a in accept: + al = a.lower() + if re.search(r"(^|[^a-z0-9])" + re.escape(al) + r"([^a-z0-9]|$)", head): + return True + return False + +# ---- Tier 1 ---- +def run_known(url, model): + rows, ok = [], 0 + for item in KNOWN: + try: + # stop at newline / next "Question:" so we score the FIRST answer only, + # not 32 tokens of continuation. + out = call(url, model, item["prompt"], 32, stop=["\n", "Question:"]) + hit = _scored_hit(out, item["accept"]) + except Exception as e: + out = f"ERROR: {repr(e)[:100]}"; hit = False + ok += int(hit) + rows.append({"prompt": item["prompt"], "accept": item["accept"], + "got": out.strip()[:80], "pass": hit}) + return {"passed": ok, "total": len(KNOWN), "rate": ok / len(KNOWN), "rows": rows} + +# ---- Tier 2 GSM8K ---- +GSM_ANS = re.compile(r"####\s*(-?[\d,]+)") +NUM = re.compile(r"-?\d[\d,]*\.?\d*") + +def load_gsm8k(n, path=None): + if path: + rows = [json.loads(l) for l in open(path) if l.strip()][:n] + return [(r["question"], r["answer"]) for r in rows] + from datasets import load_dataset + try: + ds = load_dataset("openai/gsm8k", "main", split="test") + except Exception: + ds = load_dataset("gsm8k", "main", split="test") + out = [] + for i in range(min(n, len(ds))): + q = ds[i]["question"]; a = ds[i]["answer"] + m = GSM_ANS.search(a) + gold = m.group(1).replace(",", "") if m else None + out.append((q, gold)) + return out + +def build_fewshot(shots): + ex = [ + ("Natalia sold clips to 48 friends in April, and then she sold half as many clips in May. " + "How many clips did she sell altogether in April and May?", + "In April she sold 48. In May she sold 48/2 = 24. Altogether 48+24 = 72. The answer is 72."), + ("Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. " + "How much did she earn?", + "Per minute she earns 12/60 = $0.2. For 50 minutes she earned 50*0.2 = $10. The answer is 10."), + ("Betty is saving for a $100 wallet. She has half of the money she needs. Her parents give her $15 " + "and her grandparents twice as much as her parents. How much more does she need?", + "Half of 100 is 50. Grandparents give 2*15 = 30. Total now 50+15+30 = 95. She needs 100-95 = 5. The answer is 5."), + ("James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?", + "Each time he writes 3*2 = 6 pages. Twice a week is 6*2 = 12 pages. Per year 12*52 = 624. The answer is 624."), + ("Ten more than twice a number is 30. What is the number?", + "Twice the number plus 10 is 30, so twice the number is 20, so the number is 10. The answer is 10."), + ][:shots] + s = "" + for q, a in ex: + s += f"Question: {q}\nAnswer: {a}\n\n" + return s + +def extract_answer(text): + # prefer "The answer is X", else last number + m = re.search(r"answer is\s*\$?(-?[\d,]+\.?\d*)", text, re.I) + if m: + return m.group(1).replace(",", "").rstrip(".") + nums = NUM.findall(text) + return nums[-1].replace(",", "").rstrip(".") if nums else None + +def run_gsm8k(url, model, n, shots, path=None): + data = load_gsm8k(n, path) + fewshot = build_fewshot(shots) + rows, ok = [], 0 + for q, gold in data: + prompt = fewshot + f"Question: {q}\nAnswer:" + try: + out = call(url, model, prompt, 320, stop=["Question:", "\n\n"]) + pred = extract_answer(out) + hit = (pred is not None and gold is not None and + abs(float(pred) - float(gold)) < 1e-6) + except Exception as e: + out = f"ERROR: {repr(e)[:80]}"; pred = None; hit = False + ok += int(hit) + rows.append({"gold": gold, "pred": pred, "pass": hit, "got": out.strip()[:120]}) + return {"passed": ok, "total": len(data), "rate": ok / len(data) if data else 0, "rows": rows} + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--url", required=True) + ap.add_argument("--model", default="") + ap.add_argument("--auto-model", action="store_true", help="resolve model id from /v1/models") + ap.add_argument("--known", action="store_true") + ap.add_argument("--gsm8k", type=int, default=0, help="num GSM8K problems (0=skip)") + ap.add_argument("--gsm8k-shots", type=int, default=5) + ap.add_argument("--gsm8k-file", default=None) + ap.add_argument("--out", required=True) + a = ap.parse_args() + + model = resolve_model(a.url, a.model, a.auto_model) + report = {"model": model, "url": a.url} + print(f"[acc] model={model}") + + if a.known: + r = run_known(a.url, model) + report["known"] = r + print(f"[acc] KNOWN-ANSWER: {r['passed']}/{r['total']} ({100*r['rate']:.1f}%)") + for row in r["rows"]: + if not row["pass"]: + print(f" FAIL {row['prompt'][:40]!r} -> {row['got'][:40]!r}") + + if a.gsm8k > 0: + r = run_gsm8k(a.url, model, a.gsm8k, a.gsm8k_shots, a.gsm8k_file) + report["gsm8k"] = r + print(f"[acc] GSM8K ({a.gsm8k_shots}-shot): {r['passed']}/{r['total']} ({100*r['rate']:.1f}%)") + + json.dump(report, open(a.out, "w"), indent=2) + print(f"[acc] full report -> {a.out}") + # overall verdict line for quick scan + parts = [] + if "known" in report: parts.append(f"known={100*report['known']['rate']:.0f}%") + if "gsm8k" in report: parts.append(f"gsm8k={100*report['gsm8k']['rate']:.0f}%") + print(f"[acc] VERDICT {model}: " + " ".join(parts)) + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/vllm_dissag/HY_Accuracy_folder/benchmark_accuracy_hy3.sh b/scripts/vllm_dissag/HY_Accuracy_folder/benchmark_accuracy_hy3.sh new file mode 100755 index 0000000..7051547 --- /dev/null +++ b/scripts/vllm_dissag/HY_Accuracy_folder/benchmark_accuracy_hy3.sh @@ -0,0 +1,139 @@ +#!/bin/bash +# ============================================================================= +# benchmark_accuracy_hy3.sh - consolidated accuracy benchmark for Hy3 disagg/EP +# ============================================================================= +# Runs the self-contained accuracy tests (no external dataset / network needed) +# against a live vLLM disagg server, and emits one combined JSON + a verdict line. +# Accuracy is the central risk for WideEP (EP32 silently emits `!!!` while perf +# benchmarks still report 200 OK), so this is meant to run at EVERY config. +# +# Tiers (all greedy / temperature=0 -> deterministic, comparable across configs): +# 1. KNOWN-ANSWER (accuracy_eval.py --known) : ~20 factual/math/code prompts scored +# against ground truth. Gross-correctness gate; EP32 garbage -> 0%. +# 2. NIAH (niah_probe.py) : needle-in-haystack retrieval at +# 54k/128k/256k x depths -> long-context KV-path correctness. +# 3. EQUIVALENCE (accuracy_probe.py compare): greedy exact-match vs an EP16 golden +# (optional; only if a golden json is provided / present). +# +# Usage: +# benchmark_accuracy_hy3.sh # uses env defaults below +# benchmark_accuracy_hy3.sh [golden_json] +# +# Env (overridable): +# ACC_URL server url (default http://127.0.0.1:${BENCHMARK_PORT:-30000}) +# ACC_TAG label for outputs (default ${MODEL_NAME}_xP${xP}yD${yD} or 'run') +# ACC_OUT_DIR output dir (default /shared_inference/$USER/Tencent_HY3/accuracy) +# NIAH_LENGTHS context lengths (default "54000 128000 256000") +# NIAH_DEPTHS needle depths (default "0.1 0.5 0.9") +# ACC_GOLDEN golden json for equiv (default: skip if unset/missing) +# ACC_SKIP_NIAH =1 to skip NIAH (fast mode) +# ============================================================================= +set -uo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" + +ACC_URL="${1:-${ACC_URL:-http://127.0.0.1:${BENCHMARK_PORT:-30000}}}" +ACC_TAG="${2:-${ACC_TAG:-${MODEL_NAME:-run}_xP${xP:-1}yD${yD:-1}}}" +ACC_GOLDEN="${3:-${ACC_GOLDEN:-}}" +# Container runs as root ($USER empty); USER_NAME is plumbed in by the slurm. Fall back safely. +_ACC_USER="${USER:-${USER_NAME:-ravgupta}}" +ACC_OUT_DIR="${ACC_OUT_DIR:-/shared_inference/${_ACC_USER}/Tencent_HY3/accuracy}" +NIAH_LENGTHS="${NIAH_LENGTHS:-54000 128000 256000}" +NIAH_DEPTHS="${NIAH_DEPTHS:-0.1 0.5 0.9}" +mkdir -p "$ACC_OUT_DIR" + +echo "============================================================" +echo " Hy3 ACCURACY BENCHMARK" +echo " url=$ACC_URL tag=$ACC_TAG out=$ACC_OUT_DIR" +echo "============================================================" + +# Resolve served model id. The vllm-router (port 30000) does NOT proxy /v1/models +# (only /v1/completions), so /v1/models fails behind the router. Prefer the known +# served name = MODEL_PATH (full path on this stack), then ACC_MODEL override, and +# only fall back to /v1/models (works when hitting a raw vLLM serve port directly). +MODEL="${ACC_MODEL:-${MODEL_PATH:-}}" +if [ -z "$MODEL" ]; then + MODEL=$(python3 -c "import json,urllib.request;print(json.loads(urllib.request.urlopen('$ACC_URL/v1/models',timeout=30).read())['data'][0]['id'])" 2>/dev/null) +fi +if [ -z "$MODEL" ]; then + echo "ERROR: could not resolve model id (set ACC_MODEL or MODEL_PATH). url=$ACC_URL" >&2 + exit 1 +fi +# Sanity: confirm the server answers before the full suite. The FIRST disagg +# request can be very slow (final cold kernel warmup), so retry up to ~10 min +# with a long per-attempt timeout rather than failing on a one-shot 60s ping. +if ! python3 -c " +import json,urllib.request,time +ok=False +# Probe with a REAL (non-stream) completion — matches the working benchmark path. +# EP32 cold pipeline can take ~100s+ for the first generation, so use a generous +# per-attempt timeout and budget. Sanity errors are NOT suppressed (kept visible). +for i in range(8): # 8 x 240s = 32 min budget + try: + # Real completion, matching the proven-working warmup curl (vllm_disagg_mori_ep.sh:590). + # The FIRST inference after 'Ready' triggers last-mile AITER clang/gfx942 JIT (can take + # minutes) — the generous per-attempt timeout + budget rides through that cold compile, + # which is the actual failure mode (NOT request shape; model/stream both work once warm). + b=json.dumps({'prompt':'Who is AMD CEO?','max_tokens':10,'temperature':0,'top_k':1,'stream':False}).encode() + r=urllib.request.Request('$ACC_URL/v1/completions',data=b,headers={'Content-Type':'application/json','x-request-id':f'sanity-{i}'}) + resp=urllib.request.urlopen(r,timeout=240).read() + json.loads(resp)['choices'][0]['text']; ok=True; break + except Exception as e: + print(f'[sanity] attempt {i+1} not ready: {repr(e)[:120]}',flush=True); time.sleep(10) +print('ok' if ok else 'fail') +" | grep -q '^ok'; then + echo "ERROR: server not answering /v1/completions at $ACC_URL after retries (model=$MODEL)" >&2 + exit 1 +fi +echo "served model: $MODEL" + +# ---- Tier 1: known-answer (scored, fast) ---- +echo; echo "--- Tier 1: known-answer correctness ---" +python3 "$DIR/accuracy_eval.py" --url "$ACC_URL" --model "$MODEL" --known \ + --out "$ACC_OUT_DIR/acc_${ACC_TAG}.json" +T1=$? + +# ---- Tier 2: NIAH long-context retrieval ---- +if [ "${ACC_SKIP_NIAH:-0}" != "1" ]; then + echo; echo "--- Tier 2: NIAH long-context retrieval ($NIAH_LENGTHS) ---" + python3 "$DIR/niah_probe.py" --url "$ACC_URL" --model "$MODEL" \ + --lengths $NIAH_LENGTHS --depths $NIAH_DEPTHS \ + --out "$ACC_OUT_DIR/niah_${ACC_TAG}.json" +else + echo "--- Tier 2: NIAH skipped (ACC_SKIP_NIAH=1) ---" +fi + +# ---- Tier 3: equivalence vs golden (optional) ---- +if [ -n "$ACC_GOLDEN" ] && [ -f "$ACC_GOLDEN" ]; then + echo; echo "--- Tier 3: greedy equivalence vs golden ($ACC_GOLDEN) ---" + python3 "$DIR/accuracy_probe.py" compare --url "$ACC_URL" --model "$MODEL" \ + --golden "$ACC_GOLDEN" --out "$ACC_OUT_DIR/equiv_${ACC_TAG}.json" +else + echo "--- Tier 3: equivalence skipped (no golden; set ACC_GOLDEN to enable) ---" +fi + +# ---- combined verdict ---- +echo; echo "=== VERDICT [$ACC_TAG] ===" +python3 - "$ACC_OUT_DIR" "$ACC_TAG" <<'PY' +import json, os, sys +d, tag = sys.argv[1], sys.argv[2] +def load(p): + try: return json.load(open(p)) + except Exception: return None +acc = load(f"{d}/acc_{tag}.json") +niah = load(f"{d}/niah_{tag}.json") +equiv= load(f"{d}/equiv_{tag}_summary.json") or load(f"{d}/equiv_{tag}.json") +parts=[] +if acc and "known" in acc: + k=acc["known"]; parts.append(f"known={100*k['rate']:.0f}% ({k['passed']}/{k['total']})") +if niah: + parts.append(f"niah={100*niah.get('rate',0):.0f}% ({niah.get('passed',0)}/{niah.get('total',0)})") +if equiv and "exact" in equiv: + parts.append(f"equiv={100*equiv['exact']/equiv['total']:.0f}% ({equiv['exact']}/{equiv['total']})") +verdict = " | ".join(parts) if parts else "no results" +# pass/fail heuristic: known>=80% AND (no niah or niah>=80%) +ok = bool(acc and acc.get("known",{}).get("rate",0) >= 0.8) +if niah is not None: ok = ok and niah.get("rate",0) >= 0.8 +print(f" {verdict}") +print(f" STATUS: {'PASS' if ok else 'FAIL/REVIEW'}") +PY +echo "outputs: $ACC_OUT_DIR/{acc,niah,equiv}_${ACC_TAG}.json" diff --git a/scripts/vllm_dissag/HY_Accuracy_folder/niah_probe.py b/scripts/vllm_dissag/HY_Accuracy_folder/niah_probe.py new file mode 100644 index 0000000..c15cc0e --- /dev/null +++ b/scripts/vllm_dissag/HY_Accuracy_folder/niah_probe.py @@ -0,0 +1,90 @@ +#!/usr/bin/env python3 +"""NIAH (needle-in-a-haystack) long-context retrieval probe. + +Embeds a known fact ("the magic number") at varied DEPTHS inside long filler of +target token LENGTHS, then asks the model to retrieve it. Objective pass/fail +(did the exact number come back) — no LLM judge needed. This is the only test +that exercises the long-context KV path the way a 256K workload does, so it's the +direct accuracy answer for the 256K concern, run per EP/KV config. + +Token length is approximated from the served tokenizer if reachable, else ~4 chars/token. + +Usage: + python3 niah_probe.py --url http://127.0.0.1:30000 --model \ + --lengths 54000 128000 256000 --depths 0.1 0.5 0.9 --out niah_ep32.json +""" +import argparse, json, sys, urllib.request + +FILLER = ("The grass was green and the sky was clear. People walked along the " + "quiet streets as the day went on without anything unusual happening. ") +NEEDLE_TMPL = "\n\n>>> IMPORTANT: The magic access code for this session is {code}. Remember it. <<<\n\n" +QUESTION = "\n\nQuestion: What is the magic access code for this session? Answer with only the number.\n\nAnswer:" + +def approx_tokens_to_chars(ntok): # ~4 chars/token for English filler + return ntok * 4 + +def build_haystack(target_tokens, depth, code): + total_chars = approx_tokens_to_chars(target_tokens) + needle = NEEDLE_TMPL.format(code=code) + body_chars = max(0, total_chars - len(needle) - len(QUESTION)) + reps = body_chars // len(FILLER) + 1 + body = (FILLER * reps)[:body_chars] + cut = int(len(body) * depth) + return body[:cut] + needle + body[cut:] + QUESTION + +_REQ_N = [0] + +def gen(url, model, prompt, max_tokens=32): + # Stream + x-request-id (vllm bench serve protocol) so the MoRIIO router injects + # KV routing; raw non-streaming/no-id requests crash decode (KeyError remote_host). + _REQ_N[0] += 1 + b = json.dumps({"model": model, "prompt": prompt, "max_tokens": max_tokens, + "temperature": 0.0, "stream": True}).encode() + req = urllib.request.Request(url.rstrip("/") + "/v1/completions", data=b, + headers={"Content-Type": "application/json", + "x-request-id": f"niah-{_REQ_N[0]:06d}"}) + text = "" + with urllib.request.urlopen(req, timeout=1800) as r: + for raw in r: + line = raw.decode("utf-8", "replace").strip() + if not line.startswith("data:"): + continue + chunk = line[5:].strip() + if chunk == "[DONE]": + break + try: + text += json.loads(chunk)["choices"][0]["text"] + except Exception: + pass + return text + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--url", required=True) + ap.add_argument("--model", required=True) + ap.add_argument("--lengths", type=int, nargs="+", default=[54000, 128000, 256000]) + ap.add_argument("--depths", type=float, nargs="+", default=[0.1, 0.5, 0.9]) + ap.add_argument("--out", required=True) + a = ap.parse_args() + + code = "8675309" # fixed, distinctive 7-digit needle + rows, passed = [], 0 + for L in a.lengths: + for d in a.depths: + prompt = build_haystack(L, d, code) + try: + out = gen(a.url, a.model, prompt) + ok = code in out + except Exception as e: + out = f"ERROR: {repr(e)[:120]}"; ok = False + passed += int(ok) + rows.append({"length": L, "depth": d, "pass": ok, "got": out.strip()[:80]}) + print(f" len={L:>7} depth={d:.1f} {'PASS' if ok else 'FAIL'} got={out.strip()[:40]!r}") + total = len(rows) + summary = {"passed": passed, "total": total, "rate": passed / total if total else 0, "rows": rows} + json.dump(summary, open(a.out, "w"), indent=2) + print(f"[niah] retrieval {passed}/{total} ({100*passed/total:.0f}%) -> {a.out}") + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/vllm_dissag/models.yaml b/scripts/vllm_dissag/models.yaml index 23d6605..5d914fe 100644 --- a/scripts/vllm_dissag/models.yaml +++ b/scripts/vllm_dissag/models.yaml @@ -178,3 +178,47 @@ DeepSeek-R1: dp: "" decode: dp: "" + +# ============================ Hunyuan-3.0 (Hy3) — GQA MoE, wideEP ============================ +# tencent/Hy3-preview: 80-layer GQA MoE, 192 routed experts (8/tok) + 1 shared, 256K ctx. +# VALIDATED on CS-Austin MI308 (Thor2/bnxt) + the MoRI-EP fullsource stack: +# EP8 (1P/1D): known-answer 20/20, NIAH 9/9 incl 256K deep -> PASS +# EP16 (2P/2D): known-answer 20/20, NIAH 9/9 incl 256K deep -> PASS +# EP16 2P/1D & 1P/2D perf-validated (16/0). EP32 (4P/4D) hits the known MoRI-EP all2all +# >2-pod garbage bug (deferred; same as DeepSeek pre-fix) — use EP8/EP16 today. +# Recipe deltas vs DeepSeek (MLA): Hy3 is GQA (head_size 128), so: +# - KV_CACHE_MEMORY_BYTES sized larger (GQA KV ~40 GiB @256K vs MLA's compressed KV). +# - VLLM_ROCM_USE_AITER_MLA=0 (no MLA path; also avoids the fp8 MLA decode kernel fault). +# - KV_BLOCK_SIZE=16 (block=1 has no valid ROCm attn backend under fp8 KV). +# Per-role: prefill mori_high_throughput (InterNodeV1) + cudagraph NONE (capture deadlocks +# wide DP); decode mori_low_latency (InterNodeV1LL) + PIECEWISE cudagraph (ITL win). +_hy3_recipe_env: &hy3_recipe_env + VLLM_USE_V1: "1" + VLLM_ROCM_USE_AITER: "1" + VLLM_ROCM_USE_AITER_MOE: "1" + VLLM_ROCM_USE_AITER_MLA: "0" + VLLM_ROCM_USE_AITER_PAGED_ATTN: "0" + VLLM_ROCM_USE_AITER_RMSNORM: "1" + VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "0" + VLLM_USE_AITER_TRITON_SILU_MUL: "0" + KV_BLOCK_SIZE: "16" + KV_CACHE_DTYPE: "fp8" + KV_CACHE_MEMORY_BYTES: "48000000000" + GPU_MEMORY_UTILIZATION: "0.80" + VLLM_CUDAGRAPH_MODE: "PIECEWISE" + PREFILL_CUDAGRAPH_MODE: "NONE" + DECODE_CUDAGRAPH_MODE: "PIECEWISE" + CUDAGRAPH_CAPTURE_SIZES: "1 2 4 8 16 32 64 128 256" + VLLM_ALL2ALL_BACKEND: "mori_high_throughput" + PREFILL_MORI_BACKEND: "mori_high_throughput" + DECODE_MORI_BACKEND: "mori_low_latency" + MORI_SHMEM_HEAP_SIZE: "17179869184" + +# Hy3 is wideEP-only on this recipe (dp: blocks empty; moriio connector emits the +# validated wideEP serve flags from env: above). No tp: blocks. +Hy3-preview: + env: *hy3_recipe_env + prefill: + dp: "" + decode: + dp: "" diff --git a/scripts/vllm_dissag/run_xPyD_models.slurm b/scripts/vllm_dissag/run_xPyD_models.slurm index c71fc7e..b23673e 100755 --- a/scripts/vllm_dissag/run_xPyD_models.slurm +++ b/scripts/vllm_dissag/run_xPyD_models.slurm @@ -87,6 +87,7 @@ VALID_MODELS=( \ "DeepSeek-R1" \ "Qwen3-32B" \ "Qwen3-30B-A3B" \ + "Hy3-preview" \ ) # Models allowed for CONNECTOR=moriio WIDE_EP=1 (MoRI-EP; legacy RUN_MORI=1) @@ -94,6 +95,7 @@ MORI_EP_VALID_MODELS=( \ "DeepSeek-V3" \ "DeepSeek-V3-5layer" \ "DeepSeek-R1" \ + "Hy3-preview" \ ) # Models allowed for CONNECTOR=rixl WIDE_EP=1 EP_BACKEND=deepep (legacy RUN_DEEPEP=1) @@ -427,11 +429,15 @@ BENCHMARK_COMBINATIONS="${BENCHMARK_COMBINATIONS:-}" # Benchmark script selector: BENCHMARK_SCRIPT tag -> file run by the launcher. # sweep (default) -> benchmark_xPyD.sh (general concurrency sweep) # long_context -> benchmark_long_context.sh (per-shape warmup, c=1-first) +# accuracy -> HY_Accuracy_folder/benchmark_accuracy_hy3.sh +# (known-answer + NIAH long-context; the correctness gate for +# wideEP — perf benchmarks are BLIND to silent garbage output) BENCHMARK_SCRIPT="${BENCHMARK_SCRIPT:-sweep}" case "$BENCHMARK_SCRIPT" in sweep) BENCHMARK_SCRIPT_FILE="benchmark_xPyD.sh" ;; long_context) BENCHMARK_SCRIPT_FILE="benchmark_long_context.sh" ;; - *) echo "Error: invalid BENCHMARK_SCRIPT='$BENCHMARK_SCRIPT' (valid: sweep, long_context)" >&2; exit 1 ;; + accuracy) BENCHMARK_SCRIPT_FILE="HY_Accuracy_folder/benchmark_accuracy_hy3.sh" ;; + *) echo "Error: invalid BENCHMARK_SCRIPT='$BENCHMARK_SCRIPT' (valid: sweep, long_context, accuracy)" >&2; exit 1 ;; esac if [[ ! -f "$BENCHMARK_SCRIPT_FILE" ]]; then echo "Error: selected benchmark script '$BENCHMARK_SCRIPT_FILE' not found in $(pwd)." >&2