#288 - Add UCX GPU egress example

[dleshchev]

Summary

This PR adds an opt-in two-host example that combines DAQIRI raw-Ethernet ingress, batched CUDA processing, and reliable UCP messaging over RoCEv2:

Host A raw source → DAQIRI raw RX on Host B → CUDA placement and processing
                  → UCP Active Messages → GPU-accessible receiver on Host A

DAQIRI and UCX run in the same Host B process while retaining independent progress threads and separate physical NIC ports. The example is application-local: it does not add another DAQIRI engine, change DAQIRI's single-engine model, or add a general-purpose reorder API.

What is implemented

Staged example applications

The example provides three executables:

• daqiri_ucx_raw_source generates deterministic packetized input over raw Ethernet.
• daqiri_ucx_raw_pipeline supports assemble, process, and egress stages so raw RX, CUDA processing, and the composed pipeline can be measured independently.
• daqiri_ucx_gpu_transport_bench exercises the same producer and receiver components with generated payloads, allowing UCX/RDMA behavior to be validated without DAQIRI ingress.

The demonstration workload uses 256×256 uint16 images. Sixteen packet fragments form an image and sixteen images form one 2 MiB CUDA processing batch. A CUDA ordering/placement kernel assembles the input, followed by an in-place scale-and-offset transform with defined saturation and rounding behavior.

Application-local raw assembly

Every source packet carries a small versioned DQRI header containing its source epoch, batch ID, packet sequence, fragment index, payload length, and header checksum. The assembler uses these fields and a fragment bitmap to accept in-batch reordering while rejecting missing, duplicate, malformed, timed-out, or cross-batch input.

This intentionally avoids depending on DAQIRI's existing arrival-count reorder window for application image boundaries. It remains an example protocol rather than a new public DAQIRI API.

Thread and buffer ownership

Host B uses distinct owners for:

• DAQIRI CQ polling.
• Raw packet validation and CUDA placement.
• Batched CUDA processing.
• UCP progress, sends, credits, and slot retirement.

DAQIRI bursts remain RX-owned until CUDA has finished reading their packet data. Processing slots use generation-checked, move-only leases across the RX → CUDA → UCX lifecycle. The UCX producer records a reusable CUDA event on the processing stream and submits a payload only after that event completes. A producer slot is returned to RX only after every UCP send referencing it completes and RX observes its retirement.

On the receiver, successful UCP receive completion makes a move-only ReceivedImage available to application code. release_after(cudaStream_t) returns the associated credit only after downstream GPU work on that stream has completed.

UCP protocol and overload behavior

Data uses two-sided UCP Active Message rendezvous. A CPU-visible serialized header describes each payload, and the receiver selects a registered GPU-accessible destination before calling ucp_am_recv_data_nbx().

Small Active Messages implement HELLO, ACCEPT, REJECT, cumulative CREDIT, EOS, and EOS_ACK. Receiver slots provide bounded message credits. When no credit is available, the producer drops new output before UCP submission; admitted messages are not deliberately dropped. Monotonic sequence numbers expose those overload drops as receiver gaps.

The implementation keeps these completion boundaries distinct:

• CUDA completion permits UCX/RNIC reads of the producer slot.
• UCP send completion permits producer-slot reuse but does not prove remote delivery.
• UCP receive completion delivers the complete payload to the receiver application.
• Receiver release permits receive-slot reuse; downstream CUDA completion is otherwise independent of delivery.

Build and documentation

The new ucx container target builds UCX 1.20 with UCP, verbs, mlx5, RDMA-CM, and CUDA-memory support. DAQIRI_BUILD_UCX_GPU_EGRESS enables the example beneath the existing DAQIRI_BUILD_APPLICATIONS gate.

The application includes portable placeholder configurations, an application README, and a website tutorial modeled after the ResNet50 inference example.

Two-Spark validation

The complete pipeline was validated using two DGX Spark systems and two separate 100-Gbit/s links:

• Link 1: det1 for raw-Ethernet source and DAQIRI ingress.
• Link 2: det4 for UCP/RoCEv2 egress and receive.

det1 and det4 have different PCIe prefixes on this setup. Both UCX_NET_DEVICES and UCX_IB_GID_INDEX were explicitly pinned. UCX protocol logging selected rc_mlx5 rendezvous on the intended Link 2 ports.

UCX-only result

• 1,000,000 generated and GPU-validated payloads.
• 97.87 Gbit/s application payload throughput.
• Zero drops, send failures, sequence gaps, or validation failures.

Complete pipeline result

• 25.6 million raw frames received through DAQIRI.
• 100,000 batches containing 1.6 million processed payloads.
• 94.94 Gbit/s application payload throughput.
• Stable one-second NIC samples around 96.8–98.0 Gbit/s across the two links.
• Zero missing or rejected batches, no-credit drops, sequence gaps, send failures, or GPU validation failures.

These Spark results use mapped pinned host memory. cuda_device remains supported for validation on another platform, but this PR does not claim GPUDirect RDMA for CUDA device memory merely because UCX accepts the pointer.

Validation performed

• Built the final UCX-enabled image in the project container.
• Passed all 7 CTest tests covering protocol serialization, overload policy, SPSC publication, DQRI headers, CUDA fragment placement, batch assembly, and the scale/offset transform.
• Passed git diff --check and project formatting checks.
• Passed the documentation reference checker.
• Passed a strict MkDocs build and generated-site link check.
• Ran the UCX-only and complete two-host hardware tests for more than ten stable seconds while sampling NIC counters with mlnx_perf.

Scope and limitations

• One producer, one receiver, and one logical stream.
• Fixed image count for the current benchmark runs.
• Drop-newest overload handling before UCP submission; no replay or persistence.
• Fail-stop behavior on connection loss; restart both peers to establish a new epoch.
• Mapped pinned host memory is the verified DGX Spark mode.
• CUDA-device GPUDirect RDMA and injected link-failure/reconnect behavior remain follow-up validation work.

Closes #288