Skip to content

[NV] llm-d-vllm: optimize DSv4-Pro GB200 recipe configs#2242

Draft
ilmarkov wants to merge 1 commit into
SemiAnalysisAI:mainfrom
ilmarkov:imarkov/update-dsv4
Draft

[NV] llm-d-vllm: optimize DSv4-Pro GB200 recipe configs#2242
ilmarkov wants to merge 1 commit into
SemiAnalysisAI:mainfrom
ilmarkov:imarkov/update-dsv4

Conversation

@ilmarkov

Copy link
Copy Markdown

Tune decode and prefill vLLM flags based on validated GB200 NVL72 benchmarks showing +2-43% tok/s/GPU and 15-25% lower TPOT:

Decode (mid-curve-megamoe):

  • gpu-memory-utilization 0.85 -> 0.9 (larger KV cache pool)
  • Add max-model-len 9280 (tight ISL8192+OSL1024 bound)
  • max-num-seqs/batched-tokens/cudagraph-capture 512 -> 1024
  • Disable NCCL symmetric memory (standard NVLink path faster)
  • Add no-enable-flashinfer-autotune, rust frontend

Prefill (both recipes):

  • gpu-memory-utilization 0.9 -> 0.95
  • Add max-model-len 9280, max-num-seqs 16, max-num-batched-tokens 32768
  • Disable NCCL symmetric memory, enable rust frontend
  • Enable randomize-dp-dummy-inputs

EPP (both recipes):

  • Switch from max-score-picker to weighted-random-picker (threshold=0.1) for better load distribution under high concurrency

Low-latency decode: disable NCCL symmetric memory, add rust frontend.

Tune decode and prefill vLLM flags based on validated GB200 NVL72
benchmarks showing +2-43% tok/s/GPU and 15-25% lower TPOT:

Decode (mid-curve-megamoe):
- gpu-memory-utilization 0.85 -> 0.9 (larger KV cache pool)
- Add max-model-len 9280 (tight ISL8192+OSL1024 bound)
- max-num-seqs/batched-tokens/cudagraph-capture 512 -> 1024
- Disable NCCL symmetric memory (standard NVLink path faster)
- Add no-enable-flashinfer-autotune, rust frontend

Prefill (both recipes):
- gpu-memory-utilization 0.9 -> 0.95
- Add max-model-len 9280, max-num-seqs 16, max-num-batched-tokens 32768
- Disable NCCL symmetric memory, enable rust frontend
- Enable randomize-dp-dummy-inputs

EPP (both recipes):
- Switch from max-score-picker to weighted-random-picker (threshold=0.1)
  for better load distribution under high concurrency

Low-latency decode: disable NCCL symmetric memory, add rust frontend.
@ilmarkov ilmarkov changed the title llm-d-vllm: optimize DSv4-Pro GB200 recipe configs [NV] llm-d-vllm: optimize DSv4-Pro GB200 recipe configs Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant