SIGSEGV with pyroscope-io 0.8.11 on Python 3.11 (suspected ABI mismatch in native extension)

[neves-0]

Description

Hello,

I’m reporting a reproducible segmentation fault (SIGSEGV) observed when using pyroscope-io 0.8.11 from pypi on Python 3.11 under Linux.

The issue was investigated using a core dump and GDB, and the crash consistently occurs inside the native extension shipped with the package.

Environment

Python: 3.11.14
OS: Linux x86_64 (Docker container)
pyroscope-io version: 0.8.11, 0.8.14
Workload: Celery worker (prefork, high concurrency)

Observed behavior

Randomly, the Python process crashes with a SIGSEGV.
The crash does not originate from Python code but from native code.

Core dump analysis shows that the failing code comes from the pyroscope native extension:

pyroscope/_native__lib.cpython-39-x86_64-linux-gnu.so

This extension is loaded while running Python 3.11.

Crash evidence (excerpt)
Program terminated with signal SIGSEGV, Segmentation fault.

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=11, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
grafana/pyroscope-rs#1  0x00007fedc017e9ff in __pthread_kill_internal (threadid=<optimized out>, signo=11) at ./nptl/pthread_kill.c:89
grafana/pyroscope-rs#2  0x00007fedc0129cc2 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
grafana/pyroscope-rs#3  <signal handler called>
grafana/pyroscope-rs#4  0x00007fedc018d262 in __GI___libc_free (mem=0x18) at ./malloc/malloc.c:3453
grafana/pyroscope-rs#5  0x00007fed6f7f432c in anyhow::error::object_drop () from /usr/local/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-39-x86_64-linux-gnu.so
grafana/pyroscope-rs#6  0x00007fed6f85b887 in std::sys::backtrace::__rust_begin_short_backtrace () from /usr/local/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-39-x86_64-linux-gnu.so
grafana/pyroscope-rs#7  0x00007fed6f8645e4 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
   from /usr/local/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-39-x86_64-linux-gnu.so
grafana/pyroscope-rs#8  0x00007fed6fa1b67b in std::sys::pal::unix::thread::Thread::new::thread_start ()

The crash happens during object destruction (Drop) inside the Rust native code.

Suspected cause

The native extension shipped in pyroscope-io 0.8.11 is named:

_native__lib.cpython-39-x86_64-linux-gnu.so

When used with Python 3.11, this suggests a potential ABI mismatch (3.9/3.11) between the interpreter and the compiled extension.

The wheel published for Linux is tagged as: py2.py3-none-manylinux
which allows it to be installed on Python 3.11 without warning, even though it contains a CPython-version-specific .so.

Based on the crash pattern, this looks consistent with an ABI incompatibility.

Additional note

I also checked pyroscope-io 0.8.14, and the Linux wheel still contains the same native extension:

_native__lib.cpython-39-x86_64-linux-gnu.so

I am currently testing version 0.8.14, but the packaging seems unchanged in this regard.

EDIT: Same crash with version 0.8.14

Summary

pyroscope-io 0.8.11 and pyroscope-io 0.8.14 crashes with SIGSEGV on Python 3.11.
The crash occurs in the native Rust extension.
The extension is built for CPython 3.9 but loaded on Python 3.11.
An ABI mismatch seems a plausible root cause.
The Linux wheel metadata allows this situation silently.

I hope this report helps; I can provide more details (core dump, full backtrace) if needed.

Thanks.

[korniltsev-grafanista]

I dont think ABI is a problem. The shared lib does not use any python API at all.

Please share a coredump or a way to reproduce this

[korniltsev-grafanista]

We've also switched to abi3 recently grafana/pyroscope-rs#346 but as I said above - I dont think it will help.

free(0x18);

is clearly a memory corruption or some other bug

[neves-0]

hi @korniltsev-grafanista

Can you confirm that you have successfully retrieved the coredump that I made available for analysis? Thank you!

[korniltsev-grafanista]

Yes, I've received it
But did not have time to look into it yet unfortunately

[kaminek]

We are facing the same issue here pyroscope-io 0.8.x ships py2.py3-none wheels bundling a native .so compiled for cpython-39. When loaded into our Python 3.11 runtime, the ABI mismatch causes the profiler to segfault via tgkill() (SIGSEGV, si_code=SI_TKILL). All our coredumps point to pyroscope/_native__lib.cpython-39-x86_64-linux-gnu.so.

[korniltsev-grafanista]

I don't believe this is true, the profiler does not link to python. I have a coredump from the topic starter, I will take a look when I have time.
@kaminek If you are able to provide a reproducer or a coredump publicly, please do, that would help

[korniltsev-grafanista]

If you are brave enough and can upgrade to cpython3.14 you may want to try an experimental signal based profiler grafana/pyroscope-rs#417

Not ready for production.

[korniltsev-grafanista]

This is why I do not believe this is true

~/pyroscope-rs (main) > ldd dist/hui/pyroscope_python_extension/pyroscope_python_extension.abi3.so
        linux-vdso.so.1 (0x00007ff996eb8000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff996e67000)
        librt.so.1 => /lib64/librt.so.1 (0x00007ff9963fc000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff9963f8000)
        libm.so.6 => /lib64/libm.so.6 (0x00007ff996303000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007ff9962ff000)
        libc.so.6 => /lib64/libc.so.6 (0x00007ff99610a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff996eba000)

[korniltsev-grafanista]

There's definetly a problem, just not this one.

[kaminek]

Hey, here is the coredump we saw

Thread 1 (Thread 0x7a06ce9576c0 (LWP 21)):
#0  0x00007a06fa1c495c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
grafana/pyroscope-rs#1  0x00007a06fa16fcc2 in raise () from /lib/x86_64-linux-gnu/libc.so.6
grafana/pyroscope-rs#2  <signal handler called>
grafana/pyroscope-rs#3  0x00007a06fa1d3262 in free () from /lib/x86_64-linux-gnu/libc.so.6
grafana/pyroscope-rs#4  0x00007a06ceefb7bc in anyhow::error::object_drop () from /www/.venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-39-x86_64-linux-gnu.so
grafana/pyroscope-rs#5  0x00007a06ceddce17 in std::sys::backtrace::__rust_begin_short_backtrace () from /www/.venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-39-x86_64-linux-gnu.so
grafana/pyroscope-rs#6  0x00007a06cede2304 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /www/.venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-39-x86_64-linux-gnu.so
grafana/pyroscope-rs#7  0x00007a06cf112beb in std::sys::pal::unix::thread::Thread::new::thread_start () from /www/.venv/lib/python3.11/site-packages/pyroscope/_native__lib.cpython-39-x86_64-linux-gnu.so
grafana/pyroscope-rs#8  0x00007a06fa1c2b7b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
grafana/pyroscope-rs#9  0x00007a06fa2407b8 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

[korniltsev-grafanista]

This looks like a memory corruption to me. Does it only crash in pyroscope threads or other threads as well?
Can you reproduce it locally or only on remote machcines? Is it easy to crash or need a load on the interpreter and wait time?
Any tips how to reproduce? libraries version, python versions/or ways to build/install the python, base dockker images, maybe a docker image reproducer that crashes semi-reliably?
Will you be open to try a version with ASAN if I mannage to build one?

[korniltsev-grafanista]

Is anyone keen to run the profiler with asan?

Here's an example of how someone can do this, requires building cpython from source with clang and asan and building the profiler from source and clang, there is a dockerfile example here
grafana/pyroscope-rs#465

btw I am not sure why you use 0.8.* versions, please try the python-1.0.4 as well

Please provide a reproducer - that would accelerate everything quite a lot

[Eclips4]

FWIW, we've also hit this error but haven't tried it with ASan to get more information. From what we can see, the problem seems to be in the version, which is currently 0.8.11. We'll try to bump it to the latest and let you know whether this helps.