Skip to content

#250 - Surface 802.3x pause state and counters - #252

Merged
cliffburdick merged 11 commits into
mainfrom
250-surface-pause-counters
Sep 4, 2026
Merged

#250 - Surface 802.3x pause state and counters#252
cliffburdick merged 11 commits into
mainfrom
250-surface-pause-counters

Conversation

@chloecrozier

Copy link
Copy Markdown
Member

Closes #250.

Found while chasing the ResNet benchmark's 308 Gb/s cap: 802.3x pause was eating 21.7% of
400 GbE line rate with every drop counter at zero, so it read as a slow transmitter. Two
things should have caught it and neither did.

tune_system.py had no pause check. Added --check pause: reports the pause parameters
per NIC, and separately whether the pause counters are non-zero, since pause that never fires
costs nothing. Also Step 10 in the system configuration guide and a warning in the raw
Ethernet benchmarking guide.

Pause counters never reached the xstats dump. They are not in rte_eth_xstats_get on
mlx5 at all, so the dump could not print them. src/net_pause.{h,cpp} reads them from the
kernel netdev instead (ETHTOOL_GPAUSEPARAM / ETHTOOL_GSTATS), and both raw engines call
it: warn at init when pause is enabled, print the counters with the stats dump. Counters are
per-run deltas against an init baseline, because the raw values are cumulative since boot and
would call a run throttled when pause was off for it.

Test plan

  • 400 GbE loopback, pause on: warns at init, reports the frames exchanged during the run
  • No pause activity: reports 0, no false "throttled" warning despite ~132M lifetime frames
  • Port with no netdev (software loopback): silent no-op
  • --check pause parsing: both-off, both-on, rx-only, tx-only, RX/TX negotiated lines
  • Build clean, clang-format clean, check_doc_refs.py and mkdocs anchors pass

Pause is enabled by default on many mlx5 ports and caps raw-Ethernet
throughput without incrementing any drop counter, so it reads as a slow
transmitter rather than a misconfiguration. On a 400 GbE loopback it cost
21.7% of line rate while rx_discards_phy and rx_out_of_buffer stayed at 0.

--check pause reports the ethtool pause parameters per NIC and, separately,
whether the pause counters are non-zero, since pause that is enabled but
never asserted costs nothing. It deliberately makes no per-direction claim:
ethtool and systemd-networkd document their rx/tx pause naming with opposite
senses, so the advice is to disable both.

This covers the tuning-script half of #250. DAQIRI's own xstats dump still
cannot see these counters, which is tracked in the same issue.

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
The tuning-script check added in the previous commit only helps if it is run
beforehand. It says nothing about whether pause fired during a run, and the
engines could not tell you either: mlx5 does not expose the pause counters
through rte_eth_xstats_get, so a throttled run left no trace in the shutdown
dump even though it prints every non-zero xstat. Across an entire benchmark
campaign not one application log mentioned pause while ethtool showed nine
non-zero pause counters on the same ports.

Read them from the kernel netdev instead, which works alongside a bound PMD
because mlx5 is bifurcated and keeps its netdev. src/net_pause.{h,cpp} wraps
ETHTOOL_GPAUSEPARAM and ETHTOOL_GSTATS with no DPDK or ibverbs dependency, so
it lives in daqiri_common and both raw engines use it: a warning at init when
pause is enabled, and the counters alongside the existing stats dump.

The counters are cumulative since boot, so the totals say nothing about a
single run -- a link with pause disabled still reports millions from earlier
traffic, which would claim a run was throttled when it was not. init records a
baseline and the dump reports the delta, matching the surrounding xstats, which
are also per-run.

No per-direction claim is made anywhere: ethtool and systemd-networkd document
their rx/tx pause naming with opposite senses, so the advice is to disable both.

Verified on a 400 GbE loopback: warns with pause enabled and reports the frames
exchanged during the run, stays silent when the delta is zero, and no-ops on a
port with no netdev (software loopback) rather than printing zeros.

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
@chloecrozier
chloecrozier requested a review from RamyaGuru August 20, 2026 05:36
@greptile-apps

greptile-apps Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR surfaces 802.3x pause configuration and activity so throughput throttling is distinguishable from a slow transmitter.

  • Adds Linux ethtool-based pause-state and per-run counter collection.
  • Integrates pause telemetry into the DPDK and ibverbs raw engines.
  • Adds a tune_system.py --check pause readiness check.
  • Documents pause diagnosis and configuration guidance.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename Overview
src/net_pause.cpp Adds best-effort ethtool ioctl inspection, initialization baselines, and pause-counter delta logging.
src/net_pause.h Defines the internal pause telemetry state and helper interface.
src/engines/dpdk/daqiri_dpdk_engine.cpp Adds pause-state checks during initialization and pause counters to DPDK statistics output.
src/engines/ibverbs/daqiri_ibverbs_engine.cpp Adds pause-state checks and counter reporting to the raw ibverbs engine lifecycle.
python/tune_system.py Adds a focused host check for pause configuration and cumulative pause activity.
src/CMakeLists.txt Compiles the new Linux-only pause telemetry implementation into the common library.
docs/tutorials/system_configuration.md Adds operational guidance for checking and disabling Ethernet pause where appropriate.
docs/benchmarks/raw_benchmarking.md Documents how pause counters explain throughput plateaus without packet drops.

Reviews (7): Last reviewed commit: "#252 - Tighten pause documentation" | Re-trigger Greptile

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
@RamyaGuru
RamyaGuru requested a review from cliffburdick August 20, 2026 14:21
…ration

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
Comment thread docs/tutorials/system_configuration.md Outdated
Comment thread docs/tutorials/system_configuration.md Outdated

@cliffburdick cliffburdick left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple small comments. Good PR overall

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>
@chloecrozier

Copy link
Copy Markdown
Member Author

@cliffburdick thanks for reviewing, I just pushed the changes

…nters

Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com>

# Conflicts:
#	AGENTS.md
Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com>
@cliffburdick
cliffburdick merged commit 37cb59f into main Sep 4, 2026
3 checks passed
@cliffburdick
cliffburdick deleted the 250-surface-pause-counters branch September 4, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Surface 802.3x pause state and counters

3 participants