Repository navigation
Description
What are you trying to do?
Run short-lived Python processes using the Temporal Python SDK in integration tests, including namespace registration and workflow/worker observations, and let the parent process wait for normal exit and closed stdout/stderr pipes.
Describe the bug
With the official PyPI temporalio==1.35.0 wheel, a child occasionally finishes its asyncio work, prints its last phase marker and reaches its atexit marker, yet never exits. The parent eventually terminates the child with its unchanged watchdog (roughly 29–30 seconds). This is intermittent: other runs of the same tests exit normally.
We independently observed the same native stack categories described in #2046, which was closed as not planned by its reporter. We commented there with our initial findings, but are filing this separately because the defect still reproduces for us, and we have additional independent evidence.
Environment
- OS: GitHub-hosted Ubuntu 24.04, Linux x86_64
- SDK: official
temporalio==1.35.0, not our experimental wheel - CPython 3.13.17 in frozen diagnostic runs; normal CI also reproduced post-
atexitsurvivors with 3.13.18 - Temporal Server 1.32.0 in the relevant integration tests
- Short-lived subprocesses invoked by integration/browser test harnesses
Native evidence
In real hung CI children, GDB classified:
- Main thread:
CoreRuntimeHandle::drop/ Tokio runtime shutdown and synchronization - Another native thread: Python GIL acquisition, including
PyThread_hang_thread, in a Temporal/Tokio context
One 3.13.18 CI child reached ASYNCIO_RETURN and ATEXIT around 0.8 s but exited only after the watchdog sent SIGTERM about 29 s later. Other naturally completed children exited normally.
These observations strongly match #2046's described finalization deadlock. We have not independently established the full pyo3_async_runtimes::set_result callback/owner/wait-for cycle in our optimized Linux x86_64 SDK build. Not every unrelated test timeout has been classified as this bug.
Attempted candidate fix (isolated only)
We built a separate, hash-verified test-only Python wheel from SDK 1.35.0 sources with the one-file finalization patch proposed in #2047. No application dependency or normal CI used this custom build.
- Bounded ordinary runtime create/drop tests exited normally for both official and patched wheels; worker threads disappeared after each normal drop.
- We observed two stable process-global Unix socket descriptors after the first runtime initialization, with no additional growth over three cycles in either wheel; exact initialization ownership is not conclusively established.
- A deliberately injected exception from
sys.is_finalizing()revealed a concern in the candidate patch: three consecutive runtime drops retained 2, 4, 6 native threads in the patched wheel, versus 0, 0, 0 in the official wheel. - This injection does not establish that such an exception occurs in production, but it makes an unconditional retain-on-check-error path unsuitable for us without a narrowly justified lifetime policy.
- Unmodified PostgreSQL/Temporal registration integration controls passed once per arm, but we did not reproduce the native wait cycle in a paired baseline-versus-patched experiment. We therefore do not claim the candidate fixes the intermittent hang.
We have not rolled out a patched SDK or modified existing test timeouts.
Expected behavior
After normal completion of the asyncio work and cleanup, short-lived subprocesses should exit and close their pipes without hanging in native runtime shutdown.
Questions for maintainers
- Is this recognized as an SDK interpreter-finalization / native-callback lifecycle bug, and is an upstream correction planned?
- Was there a correctness concern behind the closure of #2047, or is another approach preferred?
- What is the recommended abi3-compatible strategy for finalization-safe Core runtime destruction without retaining threads if the Python finalization check fails outside interpreter shutdown?
- Would a minimal, sanitized reproducer and a focused regression-tested PR be welcome?
We can provide further sanitized stack categories, reproducibility details and build/test evidence. We would prefer an upstream fix over maintaining a private SDK variant.
Activity
AppSolves commentedon Nov 16, 2026
Hi @VegetarianOrc and @Sushisource, a polite but urgent follow-up from the original reporter. Could we please get a maintainer triage update on this issue?
The problem is still affecting our release/integration CI, and we've since confirmed the same post-
atexitprocess-exit hang with the officialtemporalio==1.36.0wheel, not just 1.35.0. In recent Linux x86_64 / CPython 3.13.18 runs, a short-lived Temporal observer/registration subprocess completed its operations and emittedASYNCIO_RETURNandATEXIT(in about 0.5–1.5 seconds), but remained alive with 3 native threads, 9 FDs and a FUTEX wait until the unchanged parent watchdog sent SIGTERM at about 29 seconds. Other identical-path children exited normally. We have multiple separately recorded CI occurrences.We are not claiming to have proven the exact native wait-for cycle for these latest processes, nor that every Temporal test timeout shares this cause. However, the recurring, specifically instrumented post-
ATEXIThangs are making an otherwise green integration suite intermittently fail and consuming significant engineering/review time. We have intentionally not masked the issue with timeout increases, disabled checks,os._exit, or a private SDK patch.I noticed the related discussions in #300 and the proposed approaches in #2047 and #2062, which are both closed without a merged fix. Could you clarify:
Even a brief status and a recommended next step would help us plan responsibly. We're happy to contribute further sanitized evidence and testing, and we appreciate the maintainers' work on this SDK. Thank you!