PR #1329 removed a hack to avoid segfaults on exit from rocprofiler. We had to remove it because it was causing more harm than good: it was swallowing all failures. That is, no matter what happened, the process would return 0. That meant that if we hit errors in production, the process returned 0. If we hit errors in test, the process returned 0. As a consequence, across many systems, both in production and test, real failures were being misclassified as success.
An example of the segfault on exit:
INFO:2026-03-25 12:10:48 3903892:3904171 RocmActivityProfiler.cpp:68] HIP versions. Rocprofiler-sdk: 1.0; Runtime: 70051831; Driver: 70051831
*** Aborted at 1774465848 (Unix time, try 'date -d @1774465848') ***
*** Signal 11 (SIGSEGV) (0x7f5920dfccd0) received by PID 3903892 (pthread TID 0x7f59391db940) (linux TID 3903892) (code: address not mapped to object), stack trace: ***
@ 0000000014132a1e folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
./fbcode/folly/debugging/symbolizer/SignalHandler.cpp:551
@ 000000000004455f (unknown)
/home/engshare/third-party2/glibc/2.34/src/glibc-2.34/signal/../sysdeps/unix/sysv/linux/libc_sigaction.c:8
-> /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c
@ 00000000000652da (unknown)
@ 0000000000056352 (unknown)
@ 00000000000567e8 (unknown)
@ 0000000000046f74 (unknown)
@ 00000000000471d8 (unknown)
@ 000000000008765d (unknown)
@ 0000000000090389 (unknown)
@ 0000000000090603 (unknown)
@ 0000000000062222 (unknown)
@ 0000000000380359 (unknown)
@ 00000000003805b4 (unknown)
@ 00000000000478b7 __run_exit_handlers
/home/engshare/third-party2/glibc/2.34/src/glibc-2.34/stdlib/exit.c:113
@ 00000000000479c9 exit
/home/engshare/third-party2/glibc/2.34/src/glibc-2.34/stdlib/exit.c:143
@ 000000000002c65d __libc_start_call_main
/home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/nptl/libc_start_call_main.h:74
-> /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86/libc-start.c
@ 000000000002c717 __libc_start_main
/home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../csu/libc-start.c:409
-> /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86/libc-start.c
@ 00000000140ac320 _start
/home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86_64/start.S:116
Luckily, the test that encounter this segfault is also open source. It lives at: https://github.com/meta-pytorch/MSLK/blob/d7e793a616f7b12fa0dd843713c67bef14658e68/test/utils/kernel_launcher_test.cu#L239. Something that makes it different from most of our other PyTorch tests is that it's entirely orchestrated (I think) from the GPU. At least two other tests in that file also encounter the segfault on exit; I can figure out exactly which ones if that would be helpful.
PR #1329 removed a hack to avoid segfaults on exit from rocprofiler. We had to remove it because it was causing more harm than good: it was swallowing all failures. That is, no matter what happened, the process would return 0. That meant that if we hit errors in production, the process returned 0. If we hit errors in test, the process returned 0. As a consequence, across many systems, both in production and test, real failures were being misclassified as success.
An example of the segfault on exit:
Luckily, the test that encounter this segfault is also open source. It lives at: https://github.com/meta-pytorch/MSLK/blob/d7e793a616f7b12fa0dd843713c67bef14658e68/test/utils/kernel_launcher_test.cu#L239. Something that makes it different from most of our other PyTorch tests is that it's entirely orchestrated (I think) from the GPU. At least two other tests in that file also encounter the segfault on exit; I can figure out exactly which ones if that would be helpful.