Skip to content

Fix runtime shutdown before interpreter finalization - #2120

Open
brianstrauch wants to merge 3 commits into
mainfrom
fix/runtime-interpreter-shutdown
Open

brianstrauch wants to merge 3 commits into
mainfrom
fix/runtime-interpreter-shutdown

Conversation

@brianstrauch

@brianstrauch brianstrauch commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Short-lived clients can complete an awaited RPC while the Rust callback delivering its result is still active. If Python finalizes before that callback releases its thread state, the process can crash or hang.

Register an atexit hook that shuts down live native runtimes before interpreter finalization, releasing the GIL while joining their threads. Guard access to stopped runtimes and avoid joining runtimes inherited across a fork. Add subprocess regression tests for pending log callbacks with one or multiple runtimes and for an RPC completion callback that remains active after Python receives its result.

Addresses #300.

Related CI job: ai-integrations.

Validation on CPython 3.13.13, macOS ARM64:

  • poe format
  • poe build-develop
  • poe bridge-lint
  • poe test -s tests/bridge/test_runtime.py tests/test_runtime.py -n 2 — 18 passed, including server-dependent tests.
  • Pyright, basedpyright, and SDK-source docstring checks passed.

Full poe lint does not pass locally: mypy reports four errors in unchanged generated gRPC bindings, reproduced on a clean main checkout. The full mypy and docstring commands also inspect unrelated local benchmark files under build/ and report errors there.

@brianstrauch
brianstrauch requested a review from a team as a code owner November 12, 2026 23:51
@tconley1428

Copy link
Copy Markdown
Contributor

Cloud doesn't seem to allow whatever RPC you are using for the test and fails.

@VegetarianOrc VegetarianOrc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think let's also add some warnings to the docstring on the Python side Runtime class. We've already got some notes about runtimes not working across forks, but we should expand it to warn against this leak. Something akin to

Creating a runtime before forking can retain inherited native memory and file
descriptors in the child until it exits, even after the inherited runtime is
garbage collected.

Comment thread temporalio/bridge/src/runtime.rs Resolved
Comment thread temporalio/bridge/src/runtime.rs Resolved
Comment thread temporalio/bridge/src/runtime.rs Resolved
Comment thread temporalio/bridge/src/runtime.rs Resolved
Comment thread tests/bridge/runtime_shutdown.py Resolved
@VegetarianOrc
VegetarianOrc force-pushed the fix/runtime-interpreter-shutdown branch from dce72e2 to 8820f0d Compare November 19, 2026 03:33

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants