NVIDIA · cliffburdick · Sep 5, 2026 · RamyaGuru · Sep 9, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -102,7 +102,7 @@ The always-built socket engine implements Linux UDP/TCP streams directly. Applic
 
 The opt-in `dpdk` raw engine (`src/engines/dpdk/`, `DpdkEngine`) programs RX steering, send-to-kernel fallbacks (`flow_isolation: true`), and `tx_eth_src` TX offloads via DPDK RTE Flow during `daqiri_init()`. Standard UDP/IP (group 3), flex-item (group 1) and eCPRI-over-Ethernet (group 2, EtherType `0xAEFE`, via `RTE_FLOW_ITEM_TYPE_ECPRI` matching message type and pc_id/rtc_id) RX flows use separate flow groups; `validate_config()` rejects mixing these flow classes per interface, duplicate `flex_item_id` values, and unknown queue targets / `flex_item_id` values before NIC programming. Queue actions with two or more IDs use mlx5 Toeplitz IPv4/UDP RSS. The mlx5 PMD honors the native eCPRI flow item only under **firmware steering** (`dv_flow_en=1`); under HW steering (`dv_flow_en=2`, the default) the rule installs but silently never matches on ConnectX-class NICs. So `initialize()` auto-switches any interface carrying eCPRI RX flows to `dv_flow_en=1` (with a `WARN`), which means the async/template dynamic-RX-flow path is unavailable on that port. Flex-item parser handles are created per `(port, flex_item_id)` (scoped per interface). All programmed `rte_flow` rules and flex-item handles are tracked and destroyed in order on shutdown, init failure, and engine teardown (programmed flows → flex items → group-0 ETH jump rules). See `docs/benchmarks/raw_benchmarking.md` (Flow programming smoke test) for manual verification steps. It also reports 802.3x pause state at init and per-run pause counter deltas from `src/net_pause.{h,cpp}` when a kernel netdev is available.
 
-The `ibverbs` raw engine (`src/engines/ibverbs/`, `IbverbsEngine`) drives a Mellanox/mlx5 Multi-Packet (striding) Receive Queue via **DevX** (`mlx5dv_devx_obj_create` against vendored PRM structs in `mlx5_prm_min.h`): a DevX CQ + striding RQ + TIR + `mlx5dv_dr` flow steering, with manual WQE/doorbell management and worker-driven cyclic refill. RX packets DMA strided into one pre-posted MR (host or GPU via `ibv_reg_dmabuf_mr`); a queue with >1 memory region instead uses a non-striding DevX *regular* RQ with multi-segment scatter WQEs for **physical** header-data split (header → CPU MR, payload → GPU MR). TX builds mlx5 send WQEs directly on a raw-packet QP's SQ (via `mlx5dv_init_obj`, bypassing `ibv_post_send`) from a slab of registered slots tracked by cyclic index counters, with NIC checksum offload and a `tx_eth_src` offload. `loopback: "hw"` enables single-port unicast self-loopback through a retained mlx5dv activation QP and opts both direct and RSS TIRs into receiving the internally returned packets. It uses the libdpdk-free `daqiri::Ring`/`daqiri::ObjectPool` for the worker→app burst handoff (like the rdma engine — neither links DPDK) and drives the NIC through libibverbs/mlx5dv directly. Feature set: RX (MPRQ), TX, GPUDirect, physical/logical HDS, multi-queue 5-tuple flow steering with per-packet flow IDs, flex-item arbitrary-offset, IPv4-total-length and eCPRI-over-Ethernet (EtherType `0xAEFE`, message type + pc_id/rtc_id) flow matching (mlx5 flex parser / `misc_parameters_4`), per-packet RX hardware timestamps, per-queue hardware packet pacing through the mlx5 QP rate table, accurate TX send scheduling (wait-on-time WAIT WQE), and GPU reorder/quantize. Packet pacing and accurate send are independent: `pacing_mbps` configures the QP's average rate, while `set_packet_tx_time()` emits WAIT WQEs for absolute per-packet times. Because it uses the kernel netdev directly, `ensure_port_mtus` raises the netdev MTU at init to cover the configured frame size in either direction — RX (post-decap) and TX egress (post-encap) — sizing each direction with its own transform wire overhead (jumbo frames silently drop otherwise). It also uses `src/net_pause.{h,cpp}` to warn when pause is enabled and to report per-run pause frame deltas from `print_stats()`. Queues sharing a `cpu_core` are serviced round-robin by one poller thread.
+The `ibverbs` raw engine (`src/engines/ibverbs/`, `IbverbsEngine`) drives a Mellanox/mlx5 Multi-Packet (striding) Receive Queue via **DevX** (`mlx5dv_devx_obj_create` against vendored PRM structs in `mlx5_prm_min.h`): a DevX CQ + striding RQ + TIR + `mlx5dv_dr` flow steering, with manual WQE/doorbell management and worker-driven cyclic refill. RX packets DMA strided into one pre-posted MR (host or GPU via `ibv_reg_dmabuf_mr`); a queue with >1 memory region instead uses a non-striding DevX *regular* RQ with multi-segment scatter WQEs for **physical** header-data split (header → CPU MR, payload → GPU MR). TX builds mlx5 send WQEs directly on a raw-packet QP's SQ (via `mlx5dv_init_obj`, bypassing `ibv_post_send`) from a slab of registered slots tracked by cyclic index counters, with NIC checksum offload and a `tx_eth_src` offload. `loopback: "hw"` enables single-port unicast self-loopback through a retained mlx5dv activation QP and opts both direct and RSS TIRs into receiving the internally returned packets. It uses the libdpdk-free `daqiri::Ring`/`daqiri::ObjectPool` for the worker→app burst handoff (like the rdma engine — neither links DPDK) and drives the NIC through libibverbs/mlx5dv directly. Feature set: RX (MPRQ), TX, GPUDirect, physical/logical HDS, multi-queue 5-tuple flow steering with per-packet flow IDs, flex-item arbitrary-offset, IPv4-total-length and eCPRI-over-Ethernet (EtherType `0xAEFE`, message type + pc_id/rtc_id) flow matching (mlx5 flex parser / `misc_parameters_4`), per-packet RX hardware timestamps, per-queue hardware packet pacing through the mlx5 QP rate table, accurate TX send scheduling (wait-on-time WAIT WQE), GPU/CPU software reorder and quantize, and ConnectX-7+ first-DMA hardware reorder. Hardware reorder is explicit opt-in (`reorder_engine: "hw"`, `cyclic_sequence: true`): the mlx5 flex parser steers finite-ring sequence values to private fixed-slot RQs, the host poller aggregates CQEs, and destinations may be CPU or GPU memory. It does not use DPA. Exact 32-bit parser-sample matching means wide monotonic sequence fields are unsupported; use software reorder for those streams. Each destination exposes one receive credit, and a direct-placed batch is rearmed only when the application frees its burst, preventing DMA into caller-owned output. Packet pacing and accurate send are independent: `pacing_mbps` configures the QP's average rate, while `set_packet_tx_time()` emits WAIT WQEs for absolute per-packet times. Because it uses the kernel netdev directly, `ensure_port_mtus` raises the netdev MTU at init to cover the configured frame size in either direction — RX (post-decap) and TX egress (post-encap) — sizing each direction with its own transform wire overhead (jumbo frames silently drop otherwise). It also uses `src/net_pause.{h,cpp}` to warn when pause is enabled and to report per-run pause frame deltas from `print_stats()`. Queues sharing a `cpu_core` are serviced round-robin by one poller thread.
 
 ### Zero-copy / BurstParams
 All packet data flows through `BurstParams`, a batch of packets. Only pointers are passed between NIC, DAQIRI internals, and the application — the caller reads directly from the buffers the NIC DMA'd into. **The caller must explicitly free bursts**; a missed free drains the pool and produces `NO_FREE_BURST_BUFFERS` / `NO_FREE_PACKET_BUFFERS` errors and NIC drops. See `docs/concepts.md` (Zero-Copy Ownership) and `docs/api-reference/cpp.md` (free-function call patterns).

diff --git a/README.md b/README.md
@@ -32,6 +32,8 @@ DAQIRI provides direct NIC hardware access in userspace, bypassing the Linux ker
 - **GPUDirect** — Receive data directly into GPU memory via two modes:
   - *Header-Data Split*: Headers to CPU, payload to GPU (recommended for most workloads).
   - *Batched GPU*: Entire packets to GPU memory (maximum bandwidth, GPU-side parsing required).
+- **Hardware RX reorder** — On ConnectX-7 or newer NICs, the raw ibverbs engine can use the mlx5
+  flex parser for first-DMA placement into ordered CPU or GPU aggregates while the host polls CQEs.
 - **Burst file writes** — Write received bursts as raw packet files or appendable PCAP
   captures. Host-backed buffers use POSIX writes; CUDA device-backed buffers can use cuFile/GDS.
 - **AI/ML integration** — Optional `daqiri_resnet50_inference` application

diff --git a/docs/api-reference/configuration.md b/docs/api-reference/configuration.md
@@ -405,6 +405,17 @@ v1 batch-size requirement:
 
 - **`name`**: Reorder config name. Must be unique per interface.
   - type: `string`
+- **`reorder_engine`**: Reorder implementation. `sw` preserves the CUDA/CPU copy path;
+  `hw` selects ibverbs first-DMA placement on supported mlx5 NICs.
+  - type: `string`
+  - values: `sw`, `hw`
+  - default: `sw`
+- **`cyclic_sequence`**: Required acknowledgement for `reorder_engine: hw`. Hardware placement
+  uses exact 32-bit programmable-parser samples, so the sampled destination value must cycle over
+  the finite output ring and all non-address bits in each sampled word must remain zero. Wide
+  monotonic sequence values are unsupported; use `reorder_engine: sw` for those streams.
+  - type: `boolean`
+  - default: `false`
 - **`reorder_type`**: Reorder implementation (`gpu` or `cpu`).
   - type: `string`
   - values: `gpu`, `cpu`
@@ -415,6 +426,12 @@ v1 batch-size requirement:
 - **`payload_byte_offset`**: Byte offset in each packet where copied payload starts. Bytes before
   this offset are skipped.
   - type: `integer`
+- **`packet_size`**: Payload bytes placed in each output slot by `reorder_engine: hw`, excluding
+  the bytes skipped by `payload_byte_offset`. Required for hardware reorder and ignored by the
+  software path.
+  - type: `integer`
+  - performance note: values below 4000 bytes are accepted but emit a warning because the current
+    ibverbs direct-placement implementation performs much better with larger packets
 - **`data_types`**: Optional payload data type conversion for GPU reorder. If omitted, payload
   bytes are copied as-is.
   - `input_type`: On-wire input element type. Values: `int4`, `int8`, `int16`, `int32`

diff --git a/docs/api-reference/cpp.md b/docs/api-reference/cpp.md
@@ -357,6 +357,9 @@ Reordered RX bursts can be identified from `burst->hdr.hdr.burst_flags`:
 - `DAQIRI_BURST_FLAG_REORDERED` means the burst contains one aggregated reorder buffer.
 - `DAQIRI_BURST_FLAG_REORDER_TIMEOUT` means that aggregate was emitted by the timeout path
   rather than by filling the configured `packets_per_batch`.
+- `DAQIRI_BURST_FLAG_DIRECT_PLACED` means hardware wrote every payload directly into its final
+  aggregate slot. DAQIRI withholds replacement RX credits for those slots until the caller frees
+  the burst, so holding a direct-placed burst applies backpressure and may drop the next cycle.
 - For reordered bursts, `burst->hdr.hdr.max_pkt` is the logical number of source packets in the
   aggregate, while `burst->hdr.hdr.num_pkts` remains `1` because the consumer receives one
   aggregate buffer.

diff --git a/docs/api-reference/python.md b/docs/api-reference/python.md
@@ -312,6 +312,8 @@ Reordered RX bursts are identified by flags on `burst.hdr.hdr.burst_flags`:
 - `DAQIRI_BURST_FLAG_REORDERED`: burst contains one aggregated reorder buffer.
 - `DAQIRI_BURST_FLAG_REORDER_TIMEOUT`: the aggregate was emitted by the
   timeout path rather than by filling the configured `packets_per_batch`.
+- `DAQIRI_BURST_FLAG_DIRECT_PLACED`: the ibverbs hardware path placed payloads directly in their
+  final aggregate slots. The slots are not rearmed until `free_rx_burst()` releases the burst.
 
 For reordered bursts, `burst.hdr.hdr.max_pkt` is the logical number of source
 packets in the aggregate, while `burst.hdr.hdr.num_pkts` remains `1` because the
@@ -691,6 +693,7 @@ encapsulation/push rules are configured in YAML under `tx.flows`.
 | `DAQIRI_ABI_VERSION` | DAQIRI shared-library ABI version. |
 | `DAQIRI_BURST_FLAG_REORDERED` | Burst flag indicating a reordered aggregate. |
 | `DAQIRI_BURST_FLAG_REORDER_TIMEOUT` | Burst flag indicating a reorder timeout aggregate. |
+| `DAQIRI_BURST_FLAG_DIRECT_PLACED` | Burst flag indicating an ibverbs hardware-placed aggregate. |
 | `MEM_ACCESS_LOCAL` | Local memory access flag. |
 | `MEM_ACCESS_RDMA_WRITE` | RDMA write memory access flag. |
 | `MEM_ACCESS_RDMA_READ` | RDMA read memory access flag. |
@@ -760,7 +763,7 @@ names that mostly omit the trailing underscore from the C++ member name (e.g.
 | `SocketConfig` | Socket client/server endpoint URI, legacy IP/port, and timing settings. |
 | `RoCEConfig` | RoCE transport settings. |
 | `RDMAConfig` | RDMA mode, transport mode, and port. |
-| `ReorderConfig` | Reorder name, type, memory region, payload offset, flows, method, and data type conversion. |
+| `ReorderConfig` | Reorder engine/cyclic-sequence contract, type, memory region, payload offset, flows, method, and data type conversion. |
 | `ReorderBitFieldConfig` | Bit offset and width for extracting reorder fields. |
 | `ReorderSeqBatchNumberConfig` | Sequence-number, batch-number, and packets-per-batch field config. |
 | `ReorderSeqPacketsPerBatchConfig` | Sequence-number and packets-per-batch field config. |

diff --git a/docs/benchmarks/raw_benchmarking.md b/docs/benchmarks/raw_benchmarking.md
@@ -110,6 +110,25 @@ The benchmark executables and example YAML configurations are located at:
 
 The fields in the YAML configs will be explained in more detail in [Understanding the Configuration File](../tutorials/configuration-walkthrough.md). For now, we'll stick to modifying the strict minimum required fields to run the application as-is on your system.
 
+### Hardware reorder benchmark
+
+`daqiri_bench_raw_reorder_seq` can exercise ibverbs hardware direct placement on ConnectX-7 or
+newer NICs. The `*_reorder_seq_*.yaml` hardware examples select `engine: "ibverbs"`, set
+`reorder_engine: "hw"` and `cyclic_sequence: true`, and make the benchmark TX sequence wrap with
+`sequence_number_modulus`. The reorder output memory region may use `kind: device` for GPUDirect
+placement or a CPU-addressable kind.
+
+The current hardware path uses the mlx5 flex parser and private receive queues; a host CPU polls
+CQEs and releases only complete aggregates to the application. It does not use DPA and it does not
+launch the software reorder kernel. Successful output includes a non-zero
+`direct_placed_batches` count. Always free each received burst promptly, because its fixed output
+slots are rearmed only by `free_rx_burst()`.
+
+Use `mlnx_perf -i <rx-netdev> -t 1` during a run of at least 10 seconds and report stable RX
+samples after discarding startup and shutdown. For a cabled test, report the physical receive
+rate. For single-port hardware loopback, report `vport_loopback_bytes` as described below and do
+not add TX and RX for the same returned traffic.
+
 ### Hardware tunnel transform examples
 
 Raw DPDK and raw ibverbs builds can program hardware flow actions that push or

diff --git a/docs/concepts.md b/docs/concepts.md
@@ -430,14 +430,26 @@ memory.
 
 ## RX Packet Aggregation and Reorder
 
-DAQIRI can perform GPU- or CPU-side packet aggregation and reordering
-on RX through `rx.reorder_configs`:
+DAQIRI can perform packet aggregation and reordering on RX through
+`rx.reorder_configs`. Software reorder remains the default:
 
 - **GPU reorder configs** copy selected packet payloads into a configured
   output memory region and deliver the result as one *reordered
   aggregate burst*.
 - **CPU reorder configs** provide the same aggregate-burst model for
   CPU-addressable packet and output memory.
+- **Hardware reorder configs** (`reorder_engine: hw`) use the raw ibverbs engine's mlx5 flex
+  parser and flow steering to make the NIC DMA each payload directly into its final aggregate
+  slot. A host poller consumes CQEs and publishes the burst after the batch is complete; no DPA or
+  reorder-copy kernel is involved. The destination may be CPU or GPU memory.
+
+Hardware reorder is an explicit, hardware-specific optimization for ConnectX-7 or newer NICs. It
+requires a cyclic sequence/address value over the finite output ring because the current mlx5
+rules use exact 32-bit programmable-sample matches. Use software reorder for wide monotonic
+sequence fields, data-type conversion, timeout-flushed partial batches, or unsupported hardware.
+Direct-placed slots remain caller-owned until the aggregate burst is freed: DAQIRI withholds their
+replacement receive credits, so retaining a burst backpressures that batch and can cause drops on
+the next sequence cycle.
 
 This is the path to use when packets arrive out of order (e.g. across
 multiple NIC queues) and need to be reassembled into a single, contiguous

diff --git a/docs/getting-started.md b/docs/getting-started.md
@@ -238,6 +238,15 @@ actions are not part of that template fast path. The dynamic RX-flow example lea
 setting at `0` so it runs on devices whose mlx5 async flow setup is unavailable or resource
 limited.
 
+The raw ibverbs engine also supports opt-in first-DMA hardware reorder on ConnectX-7 or newer
+mlx5 NICs. Set `reorder_engine: "hw"` and acknowledge the finite-ring sequence contract with
+`cyclic_sequence: true`; software reorder remains the default. The NIC flex parser places each
+payload directly into its final CPU- or GPU-memory aggregate slot while a host CPU polls CQEs—no
+DPA is used. Exact 32-bit parser-sample matching requires the sampled destination value to cycle
+over the configured ring with other sampled bits held at zero. Direct-placed slots are rearmed
+only after the application frees the aggregate burst. See [Raw Ethernet Benchmarking](benchmarks/raw_benchmarking.md#hardware-reorder-benchmark)
+and the [configuration reference](api-reference/configuration.md#rx-reorder-configs).
+
 CUDA architectures default to `80;90` (A100, H100), with `121` (GB10) added
 when configuring with CUDA Toolkit 13.0 or newer. Override
 `CMAKE_CUDA_ARCHITECTURES` when targeting other GPUs.