CUDA strided-batched tropical GEMM rejects stride ≥ i32::MAX (kernel indexing is 32-bit) — fails on large matrices that fit in VRAM

[isPANN]

Symptom

The strided-batched CUDA tropical GEMM panics on large operands with:

batched tropical GEMM kernel: DimensionMismatch(
  "batched GEMM output C stride 2147483648 exceeds i32::MAX; matrix too large for the strided-batched kernel")

Raised by the guard in crates/tropical-gemm-cuda/src/kernels.rs (~L27–43): the kernel's stride/indexing parameter is i32, so any stride > i32::MAX (2³¹−1) is rejected before launch.

This is a kernel limitation, not OOM

• The offending stride is 2147483648 = 2³¹, i.e. exactly one over i32::MAX.
• It's the output C stride = m·n (distance between consecutive batch matrices), so a single output matrix here is 2³¹ elements ≈ 8 GB at f32; the full tensor ≈ 16 GB.
• That fits comfortably in the 80 GB A800 — nothing is out of memory. The kernel simply cannot address an element/stride offset ≥ 2³¹ with a signed 32-bit index.
• Same family as the gridDim.z = 65535 batch-count limit fixed in feat(cuda): forward strided-batched tropical GEMM (one launch over the batch) #62 (a kernel dimension-parameter ceiling, not a memory ceiling).

Repro context

Hit by miso (via omeinsum-rs's cuda-tropical backend) doing GPU tensor-network contraction:

• miso solve --backend gpu --driver dfs-lp --refiner noop --sc-target 32 on a reg5→KSG weighted-MIS instance (KSG graph V≈9721).
• The Phase-1 MaxPlus contraction at sc_target=32 produces a batched GEMM whose output C stride m·n = 2³¹.
• Deterministic: same instance fails every run; 4 sibling instances (different seeds) at the same size/sc pass (their contraction trees don't cross 2³¹).
• Workaround: --sc-target 31 halves the peak tensor → stride 2³⁰, within range; the contracted value is identical (sc-independent), so this is purely the kernel ceiling, not a correctness/precision issue.

caller (omeinsum-rs): batched_tropical_gemm_dev → c_stride = m*n; K::launch_gemm_batched(...).expect("batched tropical GEMM kernel").

Suggested fixes (in rough order of effort)

1. Widen to 64-bit indexing in the kernel: stride and the within-matrix element offset must both be int64/size_t (note: when a single matrix m·n ≥ 2³¹, it's not enough to widen only the batch stride — the per-matrix element index also overflows int). This removes the ceiling entirely.
2. Loop fallback: when any stride ≥ i32::MAX, fall back to per-batch (non-strided) GEMM calls. Doesn't help a single matrix whose m·n ≥ 2³¹ unless that GEMM also tiles internally.
3. Internal tiling: tile the M/N dimensions so each tile's offsets fit in i32 and accumulate — handles the single-large-matrix case without a full 64-bit rewrite.

Impact

Blocks GPU tropical contraction on large matrices (≈ sc ≥ 32, i.e. peak tensor ≳ 2³²) even though they fit in VRAM — exactly the large-instance regime where the GPU backend matters most. Currently censors those benchmark points or forces a lower sc_target.