perf(connection): raise MAX_TRANSMIT_SEGMENTS to 40 and MAX_TRANSMIT_DATAGRAMS to 80#636
perf(connection): raise MAX_TRANSMIT_SEGMENTS to 40 and MAX_TRANSMIT_DATAGRAMS to 80#636poka-IT wants to merge 1 commit inton0-computer:mainfrom
Conversation
…DATAGRAMS to 80 At MTU 1280 this means 51.2 KB per sendmsg(UDP_SEGMENT) call instead of 12.8 KB. The Linux kernel hard limit UDP_MAX_SEGMENTS is 64, so 40 stays within bounds. On uplink-saturated benchmarks I measured a meaningful throughput improvement on a Hetzner CCX23 box (single iperf3 -P 1, MTU 1280).
|
Thanks for the PR, could you post your benchmarks from before and after, including instruction on how to recreate them? These kind of changes tend to be delicate, so we'll want to try this out in some scenarios and see how comparable the results are. |
|
Thanks for taking a look. Here's the data, with enough context to reproduce it. Hardware / network setupTwo Hetzner CCX23 dedicated x86 VMs in
iperf3 driver: NumbersBench A is vanilla Uplink (client to exit), 5x30s, P=8:
Downlink (exit to client), 3x30s, P=8:
Within-session stddev: B uplink ±46 Mbps (2.5%), B downlink ±53 Mbps (2.2%). Tight enough that the +25% downlink is well above noise. CPU client: ~210-230% (≈2.2 cores). RSS: 27 MB (A) vs 30 MB (B), so the 4x pre-alloc growth per drive call is small enough that it doesn't show up materially at N=32 connections. Why it's asymmetricUplink shows no real change because our client-side architecture has a single sequential pump task. It's already drive-bound on its own loop, not on Downlink benefits because the exit side has 32 parallel pump tasks, each driving its own Connection. Each one's So the gain visible here is heavily workload-dependent: it shows up when several Connections drive transmit concurrently and the link has headroom. A single Connection doing bulk send on a saturated 25 GbE box (your typical bench setup) would probably show a different shape. Happy to run that variant if you want a more ReproducePatch (already in this PR): -const MAX_TRANSMIT_DATAGRAMS: usize = 20;
+const MAX_TRANSMIT_DATAGRAMS: usize = 80;
-const MAX_TRANSMIT_SEGMENTS: NonZeroUsize = NonZeroUsize::new(10).expect("known");
+const MAX_TRANSMIT_SEGMENTS: NonZeroUsize = NonZeroUsize::new(40).expect("known");Apply against [patch.crates-io]
noq = { path = "vendor/noq-fork/noq" }
noq-proto = { path = "vendor/noq-fork/noq-proto" }
noq-udp = { path = "vendor/noq-fork/noq-udp" }(Patching all three is required, otherwise Bench loop (runs on the client VM, exit running iperf3 -s on :49200): for i in 1 2 3 4 5; do
iperf3 -J -c <exit-tunnel-ip> -p 49200 -t 30 -P 8 \
| jq -r '.end.sum_sent.bits_per_second / 1e6'
sleep 1
doneRepeat with Caveats I noticedThe two benches above are cross-session (different VM pair, different day). Hetzner inter-session network capacity varies a lot: on a follow-up run, the same provisioning gave a REF that dropped from 16.9 Gbps to 7.4 Gbps downlink, and the tunnel throughput scaled down with it. Anything below ~10% delta needs paired same-session runs to be trustworthy. The +25% downlink here is well above that threshold, but I would not claim a stable absolute Mbps number across hardware. If a paired same-session A/B with a flag-toggled binary would carry more weight for you, I can run that. Memory: at N=32 Connections, the additional pre-allocated transmit space per drive call (4x growth) added ~3 MB total RSS in our setup. On something with thousands of Connections it would be more visible. RegressionsWe have 11 multi-conn QUIC integration tests covering the pump and accept loops; all pass unchanged after the patch. No new test added for the constants since there's no behavior change to assert, but happy to add a memory-footprint sanity test if you want one. Let me know if you want flamegraphs, raw iperf3 JSON, or a different shape of bench (single Connection, lossy link, low-bandwidth path). I can also run the same A/B on a kernel-WireGuard pairing for sanity check if that's useful. |
|
If I followed your description it seems these benches are from using your entire stack and tests a TCP connection established by i3perf which itself is tunnelled into a QUIC connection using the QUIC datagram extension. Unfortunately I have no idea what your entire stack is, you haven't told me yet :). I think ideally we'd be able to write a small self-contained perf tool in rust that only uses noq with no other components involved to compare the performance of these. There is already a |
|
Thanks, that pushback is fair. The original numbers were end-to-end through our entire stack (Warren is a userspace QUIC VPN we're building, using QUIC datagrams for the tunnel), which made it impossible to attribute the delta to this patch alone. I rebuilt the comparison so it's purely noq: HEAD vs HEAD~1 of this branch (so the only diff between the two binaries is the constants change), running Setup: Hetzner cpx32 (4 vCPU AMD shared, 8 GB, fsn1, kernel 6.1, debian-12). The patch governs Single bi-stream, MTU 1280 on both sides,
Ratio mean throughput 1.278x, ratio median 1.288x. Welch t-stat +13.3 (p < 0.0001). Client efficiency ratio (Mbps per 1% CPU): 1.287x. The CPU column is the part I think matters most for the merge decision. Client saturates 1 core (~99% of one vCPU on the current_thread tokio runtime) in both variants, with vanilla and patched within a fraction of a percent of each other. So the +28% throughput is not coming from the patch using more CPU. It's the same CPU budget moving more bytes, which is the expected signature of a Per-iteration:
Caveat: I had to fall back to cpx32 instead of the ccx23 from the original PR description (unrelated dedicated-core quota on my project at the time of test). Absolute throughput is therefore lower than the PR description, but the ratio sits inside the original 1.2-1.5x window. I can rerun on ccx23 if you want the exact same hardware as the claim. On extending the perf binary with a Reproduction script + raw CSVs available, can paste as a gist or open a follow-up PR adding a small (updated to add CPU/efficiency measurements from a follow-up paired run; absolute throughput is from a fresh session, hence marginally different from the first table I posted) |
|
Great that you think the existing perf tool is representative. I wasn't sure since in your first message you were going on about number of pump tasks and I didn't exactly follow what your setup and use was. But all the easier if the existing perf tool is representative. On localhost I absolutely expect this to improve the throughput. Localhost is not really lossy and has a huge amount of bandwidth. So you can have huge flow-control and congestion windows. Testing this out on various links that are not localhost is much more interesting. E.g. same datacenter, across the internet to an ISP. For example between a VPS in a datacenter in the UK and my home in Vienna I get these results: Running the server using: Running the client using: With the current noq main: With the changes from this PR: Now as you also observe, across different runs I also get different results, the best one I've seen with the increased constants was P90 at 178Mb/s, but then I also had one run with P90 at 147Mb/s. For noq main the P90 swung between 155 Mb/s and 166Mb/s. Interestingly not as wide a gap. But this is just some quick checks, not rigorous benchmarking or statistical analysis. Though it seems at least on this real link the difference is not as pronounced. This is why it would be good to get a bit more data on real links.l |
|
Ran the paired same-session bench on real network as you suggested. Two scenarios, N=16 alternating per variant, both variants built from the same fork tree (vanilla = Setup: 3x Hetzner CCX23 (4 vCPU dedicated AMD EPYC-Milan, 16 GB, kernel 6.1, debian-12), one server fsn1-dc14, two clients (one fsn1-dc14 for intra-DC, one hel1-dc2 for inter-region EU). Sysctl on all three: Bench loop: per variant, 16 iterations of Scenario A: intra-DC fsn1 (~0.5ms RTT, REF iperf3 ~16-17 Gbps)
Upload: ratio mean 1.168x (+16.8%), Welch t = +21.88, p ≈ 1e-22. CPU side, intra-DC:
Sender CPU drops noticeably, receiver CPU rises slightly because it has more bytes to process. Net: more throughput at the same or lower sender cost. Scenario B: inter-region fsn1 ↔ hel1 (26.3ms RTT measured, Hetzner EU backbone)
Upload: ratio mean 0.983x (-1.7%), Welch t = -8.95, p ≈ 4e-10. Statistically significant negative, but small in absolute terms (-6 Mbps on 365). So on this real link, the throughput is essentially flat, matching what you observed on UK ↔ Vienna. The plateau at ~365 Mbps is BDP-bound (RTT × default cwnd dominates), not syscall-bound. There's nothing the patch can do here on throughput alone. CPU side, inter-region:
Even at flat throughput, the sender uses meaningfully less CPU. This is the syscall reduction the patch is supposed to deliver, and it shows up cleanly even when bandwidth is BDP-bound. Takeaway
So the conclusion I'd argue for: this isn't a "free 1.3x throughput everywhere" patch, and your UK ↔ Vienna result was correct. But on links where throughput can scale (LAN, datacenter, intra-region), the patch matters; and on links where it can't, it still reduces sender CPU. Both regimes are non-regressing on CPU efficiency. The only nominally-negative number in the whole bench is -1.7% upload throughput on the inter-region scenario, which is at the edge of what 16 paired runs can reliably measure on a backbone link. If a synthetic-loss scenario ( Reproduction script and aggregate Python (Welch t-test, P10/P50/P90, Mbps/%CPU sender/receiver) available on request, or I can paste them as a follow-up PR adding a small |
2 participants
Description
Raises MAX_TRANSMIT_SEGMENTS from 10 to 40 and MAX_TRANSMIT_DATAGRAMS from 20 to 80 in
noq/src/connection.rs.At MTU 1280 this means 51.2 KB per
sendmsg(UDP_SEGMENT)call instead of 12.8 KB. The Linux kernel hard limitUDP_MAX_SEGMENTSis 64, so 40 stays comfortably within bounds.On uplink-saturated benchmarks I measured a meaningful throughput improvement on a Hetzner CCX23 box (single iperf3 -P 1, MTU 1280). I noticed this while looking at quinn issue 1572 and the ETHZ NSG 2024 thesis section 5.4 ("Patching Quinn"), which mentions that raising similar constants doubles msquic-style throughput in their bench.
Breaking Changes
None. Internal constants only.
Notes & open questions
Memory usage per drive call grows by 4x (40 vs 10 segments pre-allocated), which is an acceptable tradeoff for the throughput gain on saturated workloads. Lower-traffic connections still allocate on demand and should not see a difference.
Happy to add a benchmark to
bench/if useful, but the change is self-contained and the rationale matches the existing comment.Change checklist