Skip to content

perf(cuda): make GPU kernel launches asynchronous (drop per-launch device sync) - #59

Closed
isPANN wants to merge 1 commit into
feat/cuda-andor-gpufrom
perf/cuda-async-kernel-launch
Closed

perf(cuda): make GPU kernel launches asynchronous (drop per-launch device sync)#59
isPANN wants to merge 1 commit into
feat/cuda-andor-gpufrom
perf/cuda-async-kernel-launch

Conversation

@isPANN

@isPANN isPANN commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Problem

Every tropical-GEMM kernel launch helper in tropical-gemm-cuda issued a blocking stream.synchronize() immediately after enqueuing the kernel:

unsafe { builder.launch(cfg)?; }
stream.synchronize()?;   // <-- full host<->device round-trip, every launch

For a single large matmul this is invisible. But for a tropical tensor-network contraction made of thousands of tiny nodes (e.g. max-plus weighted-MIS), it serializes the entire pipeline — the host blocks on each ~few-µs kernel while the GPU sits idle between launches.

Evidence

An nsys CUDA trace of a 9431-node max-plus contraction on an A40:

  • 42,442 cuStreamSynchronize calls — exactly equal to the 42,442 GEMM kernel instances (one blocking sync per launch).
  • Actual GPU kernel time was only ~36% of the wall; the rest was the host stalled in these per-launch syncs.
  • A network with the same total arithmetic packed into a few large nodes was GPU-bound and fast — confirming the gap is per-launch sync overhead, not kernel throughput.

Fix

Remove the per-launch stream.synchronize() from every kernel-launch helper (forward GEMM, argmax, external/DLPack, backward, and K-packed AndOr), making launches asynchronous on the context's stream.

This is correctness-preserving:

  • All operand uploads, kernels, and device-to-device copies run on the context's single stream (CudaContext::stream() = the device default stream), so stream ordering guarantees each launch observes its predecessors' results.
  • Every host read goes through GpuMatrix::to_host, which synchronizes — those two barriers are kept.
  • Callers that want an explicit barrier (e.g. before timing) can call ctx.stream().synchronize().

A module-level doc comment in kernels.rs documents this asynchronous-launch contract.

Notes

  • Base is feat/cuda-andor-gpu (the branch downstream omeinsum-rs / miso pin); retarget to main if preferred.
  • No API changes; +34 / -10, comments + doc only beyond the removed syncs.
  • CUDA build/tests validated downstream on A40 (can't compile the cuda feature on non-CUDA CI hosts).

🤖 Generated with Claude Code

…vice sync)

Every tropical-GEMM kernel launch helper issued a blocking
stream.synchronize() right after enqueuing the kernel, forcing a full
host<->device round-trip per launch. On contractions built from many small
matrices this serialized the whole pipeline: the host blocked on each tiny
kernel while the GPU sat idle between launches.

An nsys profile of a 9431-node tropical (max-plus) contraction showed
42,442 cuStreamSynchronize calls -- exactly one per GEMM kernel instance --
dominating the wall, while actual kernel execution was only ~36% of it. The
same total arithmetic packed into a few large nodes was GPU-bound and fast;
the gap was purely per-launch sync overhead.

All launches run on the context's single CUDA stream, so stream ordering
already guarantees each launch observes its predecessors, and host reads go
through GpuMatrix::to_host, which synchronizes. The per-launch sync was
redundant for correctness and is removed from every launch helper (forward
GEMM, argmax, external/DLPack, backward, K-packed AndOr). The two to_host
barriers are kept. Callers needing an explicit barrier can call
ctx.stream().synchronize().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Jun 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.80%. Comparing base (82d2edb) to head (7b713cc).

Additional details and impacted files
@@                 Coverage Diff                  @@
##           feat/cuda-andor-gpu      #59   +/-   ##
====================================================
  Coverage                94.80%   94.80%           
====================================================
  Files                       20       20           
  Lines                      944      944           
====================================================
  Hits                       895      895           
  Misses                      49       49           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@isPANN

isPANN commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by the main-targeted PR above: the per-launch sync removal belongs on main (the launch helpers are shared; main already carries the device-resident API that downstream omeinsum/miso use). The K-packed AndOr launchers on this branch get the identical one-line change when #44 lands on main. Closing in favor of the main PR.

@isPANN isPANN closed this Jun 11, 2026
@isPANN
isPANN deleted the perf/cuda-async-kernel-launch branch June 14, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant