Skip to content

Fix stuck workflow eviction after deadlocks - #2142

Merged
VegetarianOrc merged 1 commit into
mainfrom
amazzeo/fix-deadlock-eviction
Nov 19, 2026
Merged

VegetarianOrc merged 1 commit into
mainfrom
amazzeo/fix-deadlock-eviction

Conversation

@VegetarianOrc

@VegetarianOrc VegetarianOrc commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

During cancellation, asyncio.wait_for marks the asyncio.Future as cancelled even if the underlying concurrent.futures.Future declines cancellation b/c it's running or finished. If cancellation is declined, eviction could start too early and retry indefinitiely.

This PR preserves waiting w/ a deadline and when that deadline is reached, the concurrent.futures.Future is cancelled directly. This cancels queued work, but running work leaves the asyncio.Future pending until the work is actually completed so eviction can await it.

  • Separate the workflow activation deadline from cancellation. Cancel queued activations through the executor's future, but keep running activations awaitable so safe eviction can wait for them to exit.
  • Keep a timed-out eviction attempt in flight and consume its outcome before retrying. A late success is acknowledged without starting a second attempt.
  • Remove already-finished tasks from workflow bookkeeping during eviction even if their done callbacks were lost. Pending tasks still have to unwind normally.

Why?

A deadlock interrupt can land while a finished query task schedules its done callbacks, leaving the task in the SDK's tracking set and causing eviction to retry forever. The previous timeout handling also cancelled the asyncio wrapper without stopping the running executor call, so eviction could race with the activation and strand a pending task.

Checklist

  1. Closes Eviction retries forever when a deadlock interrupt lands while a finished task schedules its done callbacks #2067.

  2. How was this tested:

    • Added a regression test that simulates a lost query-task done callback and verifies that both sandboxed and unsandboxed workflows can be evicted and then complete after a signal.
    • Added tests that verify a timed-out activation still reports TMPRL1101, that queued work never starts after the deadline, and that eviction waits for running work to exit whether it returns or raises.
    • Added tests that control when an eviction attempt finishes, verifying that retries do not overlap, a success during backoff is not repeated, and a failed attempt is retried.
  3. Any docs updates needed?

    • No public API or documentation changes are needed. A changelog fragment describes the fix.

@VegetarianOrc
VegetarianOrc force-pushed the amazzeo/fix-deadlock-eviction branch from 8145bb8 to 4ccc32f Compare November 18, 2026 19:20
@VegetarianOrc
VegetarianOrc marked this pull request as ready for review November 18, 2026 19:43
@VegetarianOrc
VegetarianOrc requested a review from a team as a code owner November 18, 2026 19:43
@VegetarianOrc
VegetarianOrc merged commit e64f9dd into main Nov 19, 2026
34 of 36 checks passed
@VegetarianOrc
VegetarianOrc deleted the amazzeo/fix-deadlock-eviction branch November 19, 2026 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Eviction retries forever when a deadlock interrupt lands while a finished task schedules its done callbacks

2 participants