perf(cuda): make GPU kernel launches asynchronous (drop per-launch device sync) - #59
Closed
isPANN wants to merge 1 commit into
Closed
perf(cuda): make GPU kernel launches asynchronous (drop per-launch device sync)#59isPANN wants to merge 1 commit into
isPANN wants to merge 1 commit into
Conversation
…vice sync) Every tropical-GEMM kernel launch helper issued a blocking stream.synchronize() right after enqueuing the kernel, forcing a full host<->device round-trip per launch. On contractions built from many small matrices this serialized the whole pipeline: the host blocked on each tiny kernel while the GPU sat idle between launches. An nsys profile of a 9431-node tropical (max-plus) contraction showed 42,442 cuStreamSynchronize calls -- exactly one per GEMM kernel instance -- dominating the wall, while actual kernel execution was only ~36% of it. The same total arithmetic packed into a few large nodes was GPU-bound and fast; the gap was purely per-launch sync overhead. All launches run on the context's single CUDA stream, so stream ordering already guarantees each launch observes its predecessors, and host reads go through GpuMatrix::to_host, which synchronizes. The per-launch sync was redundant for correctness and is removed from every launch helper (forward GEMM, argmax, external/DLPack, backward, K-packed AndOr). The two to_host barriers are kept. Callers needing an explicit barrier can call ctx.stream().synchronize(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feat/cuda-andor-gpu #59 +/- ##
====================================================
Coverage 94.80% 94.80%
====================================================
Files 20 20
Lines 944 944
====================================================
Hits 895 895
Misses 49 49 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Collaborator
Author
|
Superseded by the main-targeted PR above: the per-launch sync removal belongs on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every tropical-GEMM kernel launch helper in
tropical-gemm-cudaissued a blockingstream.synchronize()immediately after enqueuing the kernel:For a single large matmul this is invisible. But for a tropical tensor-network contraction made of thousands of tiny nodes (e.g. max-plus weighted-MIS), it serializes the entire pipeline — the host blocks on each ~few-µs kernel while the GPU sits idle between launches.
Evidence
An
nsysCUDA trace of a 9431-node max-plus contraction on an A40:cuStreamSynchronizecalls — exactly equal to the 42,442 GEMM kernel instances (one blocking sync per launch).Fix
Remove the per-launch
stream.synchronize()from every kernel-launch helper (forward GEMM, argmax, external/DLPack, backward, and K-packed AndOr), making launches asynchronous on the context's stream.This is correctness-preserving:
CudaContext::stream()= the device default stream), so stream ordering guarantees each launch observes its predecessors' results.GpuMatrix::to_host, which synchronizes — those two barriers are kept.ctx.stream().synchronize().A module-level doc comment in
kernels.rsdocuments this asynchronous-launch contract.Notes
feat/cuda-andor-gpu(the branch downstreamomeinsum-rs/ miso pin); retarget tomainif preferred.+34 / -10, comments + doc only beyond the removed syncs.cudafeature on non-CUDA CI hosts).🤖 Generated with Claude Code