MORI-IO nixl-style cpp bench script

[amirakb89]

bench(io): nixlbench-matching C++ engine benchmark

Summary

This PR extends #444 (the busy-wait numpy-batch-marshaling benchmark work) with a native C++ MORI-IO benchmark,
tests/cpp/io/bench_engine.cpp, whose measurement loop is written to match nixlbench (NVIDIA NIXL's xferbench) exactly. The branch descends directly from #444's head (1bd5dcd3); everything here is on top of that.

Where #444 gave the Python benchmark a busy-wait completion mode and explicit warmup so its small-message numbers stop being dominated by condition-variable wakeup latency, this PR takes the next step: a benchmark with zero Python-interpreter overhead and a measurement methodology identical to nixl, so MORI-IO and NIXL RDMA numbers are truly apples-to-apples on the same fabric.

Relationship to #444

• Built on perf(io): add busy-wait completion mode and numpy batch-transfer mars… #444, not a replacement. The Python benchmark (--busy-wait,
  --post-batch-size, --warmup-iters, --sweep-step from perf(io): add busy-wait completion mode and numpy batch-transfer mars… #444) is unchanged.
• The C++ tool applies the same ideas perf(io): add busy-wait completion mode and numpy batch-transfer mars… #444 introduced — busy-wait completion,
  excluded warmup — but bakes them in as the only path (busy-wait is
  unconditional; there is no cv-blocking Wait() here) and adds the nixl-parity
  timer/metric definitions the Python bench structurally cannot match.

What "nixl-matching" means here

• One whole-loop timer around the entire iteration loop (identical to
  nixlbench's total_timer), not the per-iteration time.time() sum the Python
  bench uses — the per-iter sum silently excludes the inter-iteration gap.
• Latency reported per single transfer: total_us / (iters × batch), matching
  nixl's avg_latency = total_duration / (per_thread_iter × batch_size).
• Bandwidth = msg × batch × iters / 1e9 / (total_us / 1e6) (GB = 10⁹), same
  units as nixl.
• Completion is always an inline busy spin-poll in the benchmark thread
  (mirroring nixl's getXferStatus scan); no cv-blocking wait that would add
  ~5–10 µs wakeup latency and make numbers non-comparable.
• Strict stop-and-wait (one request outstanding at a time, == nixl
  --pipeline_depth 1).
• Warmup iterations excluded from timing.
• Memory is registered once in setup (buffer sized to the largest request in
  the whole sweep) and reused across every size point and iteration — no
  registration or allocation inside the timed loop, matching the Python bench.

Results (batch = 1, msg = 1 MiB)

VRAM→VRAM, WRITE, single device, 1 QP, 1 worker thread, session enabled,
500 timed iters. Broadcom Thor2 bnxt_re RoCE, 2 nodes.

Benchmark	Avg BW (GB/s)	Avg Lat (µs)
C++ bench_engine	27.88	37.61
Python benchmark.py	25.61 (max 26.34)	40.95

The C++ tool reads ~9% higher at this latency-bound point precisely because of the
nixl-parity changes: the whole-loop timer and unconditional busy-wait remove the
inter-iteration gap and the cv-wakeup latency that the Python per-iter timing
still includes. At large batched sizes (e.g. batch 64 × 1 MiB) the two converge to
within <1% (~48 GB/s), since those overheads become negligible next to the
~22 µs/transfer of a 64 MiB request.

Note: at batch = 1 both tools are latency-bound and far below line rate; batching
or pipelining fills the pipe and recovers full bandwidth. This point is chosen to
highlight the measurement-methodology difference, not peak throughput.

Reproduce

Build

cd /path/to/mori
cmake -B build -DBUILD_IO=ON -DBUILD_TESTS=ON -DBUILD_EXAMPLES=OFF
cmake --build build --target bench_engine -j
# binary: build/tests/cpp/bench_engine
# runtime libs: build/src/io/libmori_io.so, build/src/application/libmori_application.so
export LD_LIBRARY_PATH=$PWD/build/src/io:$PWD/build/src/application:$LD_LIBRARY_PATH

C++ bench_engine — TCP rendezvous, start rank 1 (target) FIRST

# TARGET (rank 1) — start FIRST:
MORI_DISABLE_AUTO_XGMI=1 build/tests/cpp/bench_engine \
    --rank 1 --master-ip <RANK0_IP> --self-ip <RANK1_IP> --port 18527 \
    --mem-type gpu --op-type write --enable-sess \
    --num-qp-per-transfer 1 --num-worker-threads 1 \
    --transfer-batch-size 1 --all --sweep-start-size 1048576 --sweep-max-size 1048576 \
    --iters 500 --warmup-iters 50

# INITIATOR (rank 0) — start SECOND (prints the results table):
MORI_DISABLE_AUTO_XGMI=1 build/tests/cpp/bench_engine \
    --rank 0 --master-ip <RANK0_IP> --self-ip <RANK0_IP> --port 18527 \
    --mem-type gpu --op-type write --enable-sess \
    --num-qp-per-transfer 1 --num-worker-threads 1 \
    --transfer-batch-size 1 --all --sweep-start-size 1048576 --sweep-max-size 1048576 \
    --iters 500 --warmup-iters 50

Python benchmark.py — torchrun/gloo, start rank 0 (master) FIRST

# MASTER / INITIATOR (rank 0) — start FIRST (prints the results table):
MORI_DISABLE_AUTO_XGMI=1 bash tools/run_internode_io_benchmark.sh \
    --rank 0 --master-addr <RANK0_IP> --master-port 1234 --ifname <CTRL_NIC> \
    -- --mem-type gpu --op-type write --num-initiator-dev 1 --num-target-dev 1 \
       --num-qp-per-transfer 1 --num-worker-threads 1 --enable-sess \
       --transfer-batch-size 1 --all --sweep-start-size 1048576 --sweep-max-size 1048576 \
       --iters 500

# rank 1 — start SECOND (same args, --rank 1):
MORI_DISABLE_AUTO_XGMI=1 bash tools/run_internode_io_benchmark.sh \
    --rank 1 --master-addr <RANK0_IP> --master-port 1234 --ifname <CTRL_NIC> \
    -- --mem-type gpu --op-type write --num-initiator-dev 1 --num-target-dev 1 \
       --num-qp-per-transfer 1 --num-worker-threads 1 --enable-sess \
       --transfer-batch-size 1 --all --sweep-start-size 1048576 --sweep-max-size 1048576 \
       --iters 500

Flag differences between the two tools (they cannot take an identical command
string): C++ uses its own TCP rendezvous (--master-ip/--self-ip/--port,
rank 1 first) while Python uses torchrun/gloo (--master-addr/--ifname,
rank 0 first); the C++ tool is single-device so it has no
--num-initiator-dev/--num-target-dev. All shared benchmark values above are
identical.

Test plan

• C++ bench_engine builds (-DBUILD_EXAMPLES=OFF to skip the MPI-only
  examples target) and links libmori_io + libmori_application.
• 2-node VRAM WRITE run, batch 1 and batched (64), read back sane BW/latency.
• C++ vs Python agree to <1% at large batched sizes; the batch-1 gap is
  explained by the timer/busy-wait methodology difference.
• Reviewer sanity-check on a second fabric (CX7/mlx5).

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c755fd822e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Deregister memory before freeing its backing buffer

Both this target exit and the initiator exit free buf while localMem remains registered with the RDMA backend; the initiator may also still have a session referencing it. Once a transfer has caused the region to be registered with a NIC, freeing the HIP allocation first leaves the backend with a registration for invalid storage and defers cleanup until engine destruction, which can cause teardown failures or stale DMA mappings. Destroy the session where applicable and call engine.DeregisterMemory(localMem) before hipHostFree/hipFree on both paths.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Declare or remove the new NumPy dependency

In an environment that previously had the documented benchmark prerequisites but not NumPy, the script now fails at startup on this import before it can parse any arguments. NumPy is not declared in pyproject.toml or requirements-build.txt, so users following the existing installation flow are not guaranteed to have it; add it to the applicable runtime/setup requirements or retain an implementation using the previously supported list inputs.

Useful? React with 👍 / 👎.

[maning00]

Could we reuse MORI's existing networking abstractions instead of introducing another raw-socket rendezvous implementation here?
mori::application::TCPContext and SocketBootstrapNetwork already provide connection lifecycle management, send/receive primitives, barriers, timeouts, and cleanup. This benchmark also links mori_application, so duplicating these responsibilities with raw file descriptors creates a second implementation with different retry, timeout, and error-handling behavior.
Please consider using the existing bootstrap layer, or extracting a small reusable rendezvous helper from it. That would also avoid manual close()/std::exit() paths and make resource ownership explicit.

[maning00]

Both are ordinary std::vector types, so this normally introduces two heap allocations and two deallocations per iteration. This harness overhead is included in the reported latency and can be significant for small messages.
Could these vectors be allocated once outside the timed loop and reused by updating sp[0] and ids[0] before each submission? The mode dispatch could likewise be selected before timing, although the allocations are the more significant concern.