Repository navigation
bench: B-1b attribution of Dynamo's time-to-first-token gain - #72
Draft
manmohanbrahma wants to merge 1 commit into
Draft
manmohanbrahma wants to merge 1 commit into
manmohanbrahma wants to merge 1 commit into
Conversation
oif bench --trace samples CPU per server process, py-spy profiles every Python process (nonblocking) and snapshots /metrics around each point; bench/attrib.py splits TTFT into engine core, output path and API layer. New stacks on the same vLLM image: vllm-api4 (--api-server-count 4) and vllm-rs (vLLM 0.33.0's own Rust frontend). Profile attrib. Every server now binds 127.0.0.1. Guide 18 section, unit test test_bench_attrib.sh. Signed-off-by: Manmohan Brahma <manmohan.brahma@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
B-1 measured Dynamo's TTFT up to 40% lower than vLLM's own server (Qwen3-0.6B, chat, 64 users) with the same engine and equal throughput. B-1b finds where that time goes before we claim it.
What changes
oif bench --trace: per point, CPU of every server process (bench/proc_sampler.py), py-spy profiles of every Python process (--nonblocking, pinnedPYSPY_VERSION=0.4.1), the server's/metricsbefore and after.bench/attrib.pysplits TTFT into engine core (queue + prefill), output path and API layer, and buckets the profiles (tokenize, detokenize, HTTP, JSON, ZMQ, asyncio).BENCH_VLLM_IMAGE, only when named in--stacks:vllm-api4(--api-server-count 4) andvllm-rs(vLLM 0.33.0's own experimental Rust frontend,VLLM_USE_RUST_FRONTEND=1; skipped when the image has novllm-rsbinary).attrib: chat and short prompts, concurrency 1 to 64, 3 repeats.Tests
tests/unit/test_bench_attrib.sh(synthetic metrics, CPU samples and a speedscope profile; on Linux a live process-tree sample). Passes on Ubuntu 24.04, with test_bench_report, test_bench_lock, test_rules, test_links and test_exec_bits. shellcheck v0.11.0: info-level only.Lab check (owner, not run yet)
Two passes on the W7900 (on the bare host while the VMs are off):
--profile attribfor timings, then--repeats 1 --trace; the attrib table per point. Draft until then.