#250 - Surface 802.3x pause state and counters - #252
Merged
Conversation
Pause is enabled by default on many mlx5 ports and caps raw-Ethernet throughput without incrementing any drop counter, so it reads as a slow transmitter rather than a misconfiguration. On a 400 GbE loopback it cost 21.7% of line rate while rx_discards_phy and rx_out_of_buffer stayed at 0. --check pause reports the ethtool pause parameters per NIC and, separately, whether the pause counters are non-zero, since pause that is enabled but never asserted costs nothing. It deliberately makes no per-direction claim: ethtool and systemd-networkd document their rx/tx pause naming with opposite senses, so the advice is to disable both. This covers the tuning-script half of #250. DAQIRI's own xstats dump still cannot see these counters, which is tracked in the same issue. Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
The tuning-script check added in the previous commit only helps if it is run
beforehand. It says nothing about whether pause fired during a run, and the
engines could not tell you either: mlx5 does not expose the pause counters
through rte_eth_xstats_get, so a throttled run left no trace in the shutdown
dump even though it prints every non-zero xstat. Across an entire benchmark
campaign not one application log mentioned pause while ethtool showed nine
non-zero pause counters on the same ports.
Read them from the kernel netdev instead, which works alongside a bound PMD
because mlx5 is bifurcated and keeps its netdev. src/net_pause.{h,cpp} wraps
ETHTOOL_GPAUSEPARAM and ETHTOOL_GSTATS with no DPDK or ibverbs dependency, so
it lives in daqiri_common and both raw engines use it: a warning at init when
pause is enabled, and the counters alongside the existing stats dump.
The counters are cumulative since boot, so the totals say nothing about a
single run -- a link with pause disabled still reports millions from earlier
traffic, which would claim a run was throttled when it was not. init records a
baseline and the dump reports the delta, matching the surrounding xstats, which
are also per-run.
No per-direction claim is made anywhere: ethtool and systemd-networkd document
their rx/tx pause naming with opposite senses, so the advice is to disable both.
Verified on a 400 GbE loopback: warns with pause enabled and reports the frames
exchanged during the run, stays silent when the delta is zero, and no-ops on a
port with no netdev (software loopback) rather than printing zeros.
Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
Contributor
|
| Filename | Overview |
|---|---|
| src/net_pause.cpp | Adds best-effort ethtool ioctl inspection, initialization baselines, and pause-counter delta logging. |
| src/net_pause.h | Defines the internal pause telemetry state and helper interface. |
| src/engines/dpdk/daqiri_dpdk_engine.cpp | Adds pause-state checks during initialization and pause counters to DPDK statistics output. |
| src/engines/ibverbs/daqiri_ibverbs_engine.cpp | Adds pause-state checks and counter reporting to the raw ibverbs engine lifecycle. |
| python/tune_system.py | Adds a focused host check for pause configuration and cumulative pause activity. |
| src/CMakeLists.txt | Compiles the new Linux-only pause telemetry implementation into the common library. |
| docs/tutorials/system_configuration.md | Adds operational guidance for checking and disabling Ethernet pause where appropriate. |
| docs/benchmarks/raw_benchmarking.md | Documents how pause counters explain throughput plateaus without packet drops. |
Reviews (7): Last reviewed commit: "#252 - Tighten pause documentation" | Re-trigger Greptile
Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
…ration Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
cliffburdick
requested changes
Aug 25, 2026
cliffburdick
left a comment
Collaborator
There was a problem hiding this comment.
Couple small comments. Good PR overall
Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
Member
Author
|
@cliffburdick thanks for reviewing, I just pushed the changes |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #250.
Found while chasing the ResNet benchmark's 308 Gb/s cap: 802.3x pause was eating 21.7% of
400 GbE line rate with every drop counter at zero, so it read as a slow transmitter. Two
things should have caught it and neither did.
tune_system.pyhad no pause check. Added--check pause: reports the pause parametersper NIC, and separately whether the pause counters are non-zero, since pause that never fires
costs nothing. Also Step 10 in the system configuration guide and a warning in the raw
Ethernet benchmarking guide.
Pause counters never reached the xstats dump. They are not in
rte_eth_xstats_getonmlx5 at all, so the dump could not print them.
src/net_pause.{h,cpp}reads them from thekernel netdev instead (
ETHTOOL_GPAUSEPARAM/ETHTOOL_GSTATS), and both raw engines callit: warn at init when pause is enabled, print the counters with the stats dump. Counters are
per-run deltas against an init baseline, because the raw values are cumulative since boot and
would call a run throttled when pause was off for it.
Test plan
0, no false "throttled" warning despite ~132M lifetime frames--check pauseparsing: both-off, both-on, rx-only, tx-only,RX/TX negotiatedlinesclang-formatclean,check_doc_refs.pyand mkdocs anchors pass