Repository navigation
Don't claim jobs for a process that has been pruned - #837
Open
cole-robertson wants to merge 1 commit into
Open
cole-robertson wants to merge 1 commit into
cole-robertson wants to merge 1 commit into
Conversation
A worker normally finds out its process was pruned at its next heartbeat. Until then it keeps claiming jobs with the process id it remembers. Nothing releases those claimed executions while supervisors keep running, so a job that doesn't finish before the worker exits stays claimed until a supervisor boots and fails it with ProcessMissingError. Lock the process row before selecting candidates, and claim nothing if the process is gone. Deregistering deletes that row and releases its claimed executions in one transaction, so a claim either commits first and is released with the rest, or claims nothing. The ReadyExecution tests now claim with a registered process instead of the literal id 42. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A worker normally finds out its process was pruned at its next heartbeat, up to
process_heartbeat_interval(60 seconds by default) later. Until then its poll loop keeps claiming jobs with the process id it remembers. Nothing releases those claimed executions while supervisors keep running: pruning and deregistering go through the process record, which is gone, and the fork supervisor looks a terminated worker up by name. A job claimed in that window that doesn't finish before the worker exits stays claimed until some supervisor boots and fails it withProcessMissingError.#244 described this race ("a worker gets deregistered while still working normally ... those executions would get claimed by a process that's getting deleted") and released orphaned executions in every prune. eb026d3 moved that to supervisor boot only, so the gap is open again.
One way a live worker gets pruned is two job hosts whose clocks disagree by more than
process_alive_threshold - process_heartbeat_interval(4 minutes by default):last_heartbeat_atis stamped with each host's own clock, so the host with the right clock can prune the other's workers.This makes
ReadyExecution.claimlock the process row before selecting candidates, and claim nothing if the process is gone. Deregistering deletes that row and releases its claimed executions in one transaction, so the two can't interleave: a claim either commits before the process is deleted and is released with the rest, or sees the process gone and claims nothing. It costs one locking primary-key lookup per queue it tries to claim from, including empty ones.The
ReadyExecutiontests claimed with the literal process id 42, which has no record, so they now register a process. New tests check that:use_skip_locked(PostgreSQL and MySQL). This fails on main, and also with an unlocked existence check;Reproduction
Two supervisors share one SQLite queue database, at the default settings. Supervisor A's process tree runs with its wall clock 6 minutes behind (libfaketime), and a 30-second job is enqueued every 2 seconds. B prunes A's worker as soon as it registers. Each trial waits for that worker to exit, then for up to 330 seconds, and counts the jobs still claimed by a process id that isn't registered.
ProcessMissingError.The same happens in a new Rails 8.1 app with Solid Queue's generated configuration and its queue database on PostgreSQL, running
bin/jobsas the two hosts. In 3 of 3 runs on 1.7.2, the pruned worker claimed 3 jobs that had started and not finished, and they were still claimed 6 minutes later. On this branch, 0 of 3 runs left a job claimed.The test suite passes on SQLite, PostgreSQL and MySQL.