Repository navigation
Fix stuck workflow eviction after deadlocks - #2142
Merged
Merged
Conversation
VegetarianOrc
force-pushed
the
amazzeo/fix-deadlock-eviction
branch
from
November 18, 2026 19:20
8145bb8 to
4ccc32f
Compare
VegetarianOrc
marked this pull request as ready for review
November 18, 2026 19:43
tconley1428
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
During cancellation,
asyncio.wait_formarks theasyncio.Futureas cancelled even if the underlyingconcurrent.futures.Futuredeclines cancellation b/c it's running or finished. If cancellation is declined, eviction could start too early and retry indefinitiely.This PR preserves waiting w/ a deadline and when that deadline is reached, the
concurrent.futures.Futureis cancelled directly. This cancels queued work, but running work leaves theasyncio.Futurepending until the work is actually completed so eviction can await it.Why?
A deadlock interrupt can land while a finished query task schedules its done callbacks, leaving the task in the SDK's tracking set and causing eviction to retry forever. The previous timeout handling also cancelled the asyncio wrapper without stopping the running executor call, so eviction could race with the activation and strand a pending task.
Checklist
Closes Eviction retries forever when a deadlock interrupt lands while a finished task schedules its done callbacks #2067.
How was this tested:
TMPRL1101, that queued work never starts after the deadline, and that eviction waits for running work to exit whether it returns or raises.Any docs updates needed?