GPU Gesvd_truncate: cusolverDnDgesvdj returns CUSOLVER_STATUS_EXECUTION_FAILED (CUDA 13)

[yingjerkao]

Summary

On CUDA 13, linalg::Gesvd_truncate on a GPU tensor makes cusolverDnDgesvdj return CUSOLVER_STATUS_EXECUTION_FAILED, which checkCudaErrors turns into a fatal abort. Because it aborts the whole gpu_test_main process, it also prevents any test registered after it from running.

CUDA error at src/backend/linalg_internal_gpu/cuGeSvd_internal.cu:244 code=6(CUSOLVER_STATUS_EXECUTION_FAILED)
  "cusolverDnDgesvdj(cusolverH, jobz, econ, N, M, (data_type *)Mij, ldA, (data_type *)S->data(),
   (data_type *)vTMem, ldu, (data_type *)UMem, ldvT, (data_type *)d_work, lwork, devinfo, gesvdj_params)"

Where

src/backend/linalg_internal_gpu/cuGeSvd_internal.cu:244 — the cusolverDnDgesvdj (Jacobi SVD) call in the Double GPU Gesvd path.

Failing test

GesvdTruncate.GpuFlagCombinations (tests/gpu/linalg_test/truncate_test.cpp). It runs Gesvd_truncate on a random 6×5 tensor for Double and ComplexDouble; the Double case aborts on the gesvdj call. Minimal shape of the repro:

Tensor T = Tensor({6, 5}, Type.Double).to(Device.cuda);
InitTensorUniform(T);                       // values in [-1000, 1000]
linalg::Gesvd_truncate(T, /*keepdim=*/3, 0., true, true);   // -> CUSOLVER_STATUS_EXECUTION_FAILED

Environment

• CUDA 13.0 (nvcc V13.0.88), cuSOLVER bundled with CUDA 13
• NVIDIA RTX 4070 Ti SUPER, sm_89
• -DUSE_CUDA=ON -DUSE_CUTENSOR=OFF -DUSE_CUQUANTUM=OFF, Debug + ASan

Notes / relationship to other issues

• Distinct from GPU cuSvd crashes with is_UvT=false (no U/vT buffers) #831. GPU cuSvd crashes with is_UvT=false (no U/vT buffers) #831 is the other SVD path — cuSvd_internal → cusolverDnXgesvdp, which passes null U/V buffers when is_UvT=false → cudaErrorIllegalAddress. GPU cuSvd crashes with is_UvT=false (no U/vT buffers) #831 explicitly states the Gesvd/gesvdj path "works"; this report shows that under CUDA 13 the gesvdj path also fails, here with EXECUTION_FAILED (a cuSOLVER convergence/parameter failure, not a null-pointer deref). Likely a CUDA-13 cuSOLVER gesvdj compatibility problem (cf. the CUDA-13 cuKron breakage in GPU build broken: cuKron fails to compile (CUDA 13) after the #982 type_promote change #999).

• Surfaced while running the full gpu_test_main suite (which has no CI runner). For completeness, the other failures seen in that run are already tracked: zero-extent cudaFree → GPU zero-extent Storage: cudaFree(cudaErrorInvalidValue) on destruction #1089, rank-0 BlockUniTensor::Trace → GPU BlockUniTensor Trace test uses wrong locator on rank-0 result (tests/gpu/BlockUniTensor_test.cpp) #1090, slow large-shape elementwise → Slow GPU tests: element-wise *AllShapes all_types/mixed_types cases dominate ctest runtime #851. linalg_Test.GpuBkFUtQr fails only because GPU QR requires cuQuantum (expected in a USE_CUQUANTUM=OFF build; the test should be guarded/skipped rather than fail).

Suggested follow-up

Investigate the gesvdj_params / jobz / econ setup and devinfo handling for cusolverDnDgesvdj under CUDA 13; when EXECUTION_FAILED is returned, devinfo should be read to report which superdiagonal failed to converge rather than aborting. A guard to skip the GPU SVD/QR tests when the required cuSOLVER path is unavailable would also keep the rest of gpu_test_main runnable.

[ianmccul]

i've seen that before, it works if -DUSE_CUQUANTUM=ON

[IvanaGyro]

-DUSE_CUDA=ON -DUSE_CUTENSOR=OFF -DUSE_CUQUANTUM=OFF, Debug + ASan This configuration is not supported per #584 (comment)

[yingjerkao]

Narrowed this down while verifying #993 on a CUDA 13 box. The failure is confined to non-cuQuantum builds, because cuQuantum builds never reach the failing call.

The mechanism

src/linalg/Gesvd_truncate.cpp has a #ifdef UNI_CUQUANTUM branch (lines 19 and 59) that routes the GPU path through cuQuantumGeSvd_internal (cuTensorNet) instead of the cuSOLVER cuGeSvd_internal path. So cusolverDnDgesvdj at cuGeSvd_internal.cu:244 is only ever called when cuQuantum is off.

Same test, same source tree (master @ 2318026), same single-test filter, two builds:

build	GesvdTruncate.GpuFlagCombinations
USE_CUQUANTUM=OFF	CUSOLVER_STATUS_EXECUTION_FAILED at cuGeSvd_internal.cu:244
USE_CUQUANTUM=ON (+ USE_CUTENSOR=ON)	[ OK ] (1715 ms)

Rsvd.GpuFlagCombinations behaves identically — it aborts at the same call site without cuQuantum and passes with it.

Hypotheses ruled out (with evidence)

I checked the obvious alternative explanations before concluding the above; all three are wrong:

1. Not a different cuSOLVER version. The cuQuantum build additionally loads conda's libcusolver.so.11/libcublas.so.12 (cuTensorNet needs CUDA 12), which looked like an obvious suspect. But symbol versioning keeps them separate — LD_DEBUG=bindings shows Cytnx binding to the same CUDA 13 library in both builds:

   gpu_test_main -> /usr/local/cuda/lib64/libcusolver.so.12: `cusolverDnDgesvdj`
   libcutensornet.so.2 -> miniforge3/lib/libcusolver.so.11: `cusolverDnDgesvdj`
   

2. Not symbol interposition. LD_PRELOAD-ing conda's libcusolver.so.11 + libcublas.so.12 onto the failing binary does not change the outcome — still fails.

3. Not lazy CUDA module loading. CUDA_MODULE_LOADING=EAGER on the failing binary — still fails.

Why this matters

• The bug is real in the cuSOLVER gesvdj path and is not a general CUDA 13 problem, so the investigation can focus on cuGeSvd_internal.cu rather than the toolkit.
• It is invisible to anyone building with cuQuantum, which is easy to mistake for "it works now".
• Conversely it is exactly what the new self-hosted GPU CI (ci(gpu): native self-hosted GPU build + test workflow #1044) will hit, since that path builds without cuQuantum.

Environment

• RTX 4070 Ti SUPER (sm_89), CUDA 13.0 (nvcc V13.0.88), GCC 13.3, CMake 4.0
• Both builds RelWithDebInfo, RUN_TESTS=ON, CMAKE_CUDA_ARCHITECTURES=89
• cuQuantum build used conda cuTENSOR 2.2.0 / cuTensorNet 2.7.0

[yingjerkao]

Correction to my previous comment. I claimed this is "exactly what the new self-hosted GPU CI (#1044) will hit, since that path builds without cuQuantum." That is wrong, and it overstates the severity.

.github/workflows/ci-gpu_tests.yml:80 configures with both optional deps on:

-DUSE_CUDA=ON -DUSE_CUTENSOR=ON -DUSE_CUQUANTUM=ON

and so does every CUDA build the project ships:

configuration	cuTENSOR	cuQuantum
openblas-cuda preset (and mkl-cuda, debug-*-cuda, which inherit it)	ON	ON
GPU CI (ci-gpu_tests.yml)	ON	ON
CUDA release wheels (prepare_cuda_release.py switches to --preset=openblas-cuda, provisioning cutensor-cu13 / cutensornet-cu13 / custatevec)	ON	ON

Since Gesvd_truncate.cpp diverts to cuTensorNet under UNI_CUQUANTUM, neither the GPU CI nor the published cytnx-cuda wheels execute the failing cusolverDnDgesvdj call. Confirmed on the cuQuantum build: GesvdTruncate.*, SvdTruncate.*, GeSvd.*, Svd.* and Rsvd.* all pass.

So the accurate scope is narrower than I said: this reproduces only in a CUDA build that deliberately sets USE_CUQUANTUM=OFF, which is not one of the supported presets. My non-cuQuantum build_gpu was hand-rolled, not a configuration the project actually ships.

That does not make it a non-issue — it means cuGeSvd_internal.cu is an untested fallback path: nothing in CI or in the shipped wheels covers it, so it can rot silently (the same class of problem as #1127). But it should be prioritised as "broken fallback nobody currently builds" rather than "breaks GPU CI".

Apologies for the noise — the underlying mechanism in my previous comment (the #ifdef UNI_CUQUANTUM divert, and the three ruled-out hypotheses) still stands.