Skip to content

Fail a run as lost when its worker stops - #674

Open
yorzi wants to merge 5 commits into
activeagents:mainfrom
yorzi:fix/agent-run-recovery
Open

yorzi wants to merge 5 commits into
activeagents:mainfrom
yorzi:fix/agent-run-recovery

Conversation

@yorzi

@yorzi yorzi commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

What this does

Since #664 a run is claimed once, so a redelivered job never runs it twice. The other side of that, which #664 left out of scope: a worker that dies after its claim (a kill, an out-of-memory error, a deploy that does not wait for jobs) leaves the run running for good. The redelivery sees a running run and leaves it, and nothing else ever looks at it. The same happens after AgentResumeJob claims a paused run, and for a web process that dies inside Agent#test_execute. Nothing in the run says whether anyone is still working on it: started_at is a start, and append_event writes with update_column, so updated_at does not move while the run works.

Ending such a run is only safe by cancelling it. unless_cancelled discards a late result over a cancel, but any other ending is overwritten. A run failed by anything other than a cancel while its worker was only slow came back complete, with the late output and the other ending's error message side by side.

This follows two things already in the engine: the scoring lease from #614 (a token, an expiry, a fenced write, column detection for hosts that have not migrated) and InputRequestExpiryJob (expired when read, plus an optional recurring job). It is five commits:

  1. Give each run segment a lease that its progress renews. Migration 033 adds execution_token and execution_expires_at to agent_runs, indexed with status. Every claim of a run for an execution segment writes a new token and an expiry config.run_lease_duration ahead (30 minutes by default). Those claims are claim_pending, claim_pause and the creation in test_execute. Each progress event (one per LLM or tool call) renews the expiry while the run is running under that token. Without the columns, or with the duration set to nil, no lease is taken.
  2. Discard a result from a worker whose run left its lease. unless_cancelled takes the worker's token and yields only while the run still holds it. Every write that ends a segment releases the lease (result, pause, failure), so whatever ended the run first is kept, and a late result is a logged no-op. test_execute now records through it as the jobs do, which also stops it overwriting a cancel.
  3. Fail a run as lost once its lease runs out. AgentRun#expire_if_lost! is one conditional update on id, status, token and expiry. It records failed with error_class ActionAgent::AgentRun::WorkerLost and a message that says the run was not executed again. The run's page and the run list expire what they reach. ActionAgent::AgentRunExpiryJob reaches the rest and is not scheduled by default. A run with no lease, such as one running when the migration is installed, is never expired.
  4. Document run leases and lost workers. A section in docs/framework/dashboard.md, the upgrade note and a Fixed entry in the CHANGELOG's [Unreleased] section, and the option in the generated initializer.
  5. Read a run's lease token through the column guard (review of 2026-11-17). The three workers read run.execution_token directly, which raised NoMethodError on every run on a host without migration 033. AgentRun#lease_token returns the token only when the lease columns are installed and nil otherwise; the workers, renew_lease! and the fence read it.

Nothing is executed again, by design. The worker may have died after the provider billed the call or after a tool acted, so the run's message and the docs say so, and running the prompt again is a new run made by a person.

Choices made here that are yours to change:

  1. Terminal state. failed with error_class WorkerLost (what record_failure! already writes), so lists, filters and scorecards need no change. The alternative is a new interrupted status.
  2. Trigger. Expire on read plus the unscheduled job, mirroring InputRequestExpiryJob. Without a scheduler, a lost run nobody opens stays running, as today.
  3. Lease length. 30 minutes, as JUDGE_LEASE_DURATION, renewed only by progress events. A single provider call longer than that looks lost, and its result is then discarded. A heartbeat thread would avoid that but would also keep a wedged process alive. Raising the setting is the documented answer.

Tests, all in actionagent/test/run_lease_test.rb plus one in install_generator_test.rb, 39 in all:

  • Lease: the execution job's claim, a resumed segment (new token), and test_execute each take a lease; two claims never share a token; an event renews; a renewal leaves a run that finished or changed token alone; no lease without a duration or without the columns; the migration installs under a custom prefix and reverses.
  • Fence: a worker whose run was ended under it records neither its result nor its own failure, in the execution job, the resume job and test_execute. A cancel during test_execute stays. A worker whose run was taken under another token records nothing. A completed, paused or failed segment releases its lease. Plus one real-connection test: the worker held in its generation on its own connection while the run is failed as lost on another.
  • No columns (RunLeaseWithoutColumnsTest): the two columns and their index are dropped inside the test transaction and the column information reset, so the attribute methods are really gone. The execution job (result and failure), the resume job, test_execute (result, and a cancel during it), a progress event, the expiry paths and the run's page and list all work as before leases. Reverting any of the three callers, or the guard in lease_token, lease_attributes or lease_release_attributes, fails it.
  • Expiry: failed once with WorkerLost, its existing metadata kept, announced once, one log line. Not executed again. A live run, a renewed one or one re-taken since it was read is left running, and a live run costs no query. Only running runs expire, never one without a lease, and nothing without the columns. A cancel that lands first wins. The job fails every lost run and only those. The run's page and the run list show it failed. A late worker's result is discarded after expiry. A run that keeps reporting progress outlasts the lease. Plus a real-connection race of two sweeps (one transition, one broadcast).

Each part of each commit reverted in turn fails at least one test (7 parts in the lease commit, 8 in the fence, 13 in the expiry, 6 in the guard). Each commit passes its own tests when checked out alone.

Backends: SQLite on Rails 8.1.4 and 7.2.3.2. PostgreSQL 17 through a standalone Active Record probe of the same method bodies, since the dummy schema does not load on PostgreSQL (an index name over 63 characters). One claim in 50 races. The expiry writes the JSON metadata through update_all. Two sweeps make one transition in 50 races. A renewal racing a sweep was consistent in 50 races. A late worker is fenced whichever commits first. The engine suite on PostgreSQL itself is not run.

Not in this PR: stopping a generation early when its renewal fails (it would save spend, but changes no guarantee); a "run again" action; a task for runs that predate the migration (cancel them).

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the contributing guide
  • This fits activeagent's stated focus
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my use case

Related issue

None filed. Follow-up to #664, which names this gap as out of scope. Precedents: #614.

Quality check

  • The project's hooks and linters pass (bin/rubocop on the changed files; the repo has no commit hooks installed)
  • The new tests fail before this change and pass after it
  • The full test suite passes: bin/test with the CI placeholder keys, rebased on main after 1.9.9 (Rails 8.1.4): 4683 runs, 0 failures, 11 errors, all CliContractTest (rack-session undefined method 'id' for Hash). Those 11 fail identically on an untouched main in the same checkout and pass in a fresh worktree of main, so they come from local state, not this change; CI is the check. Before the rebase, Rails 7.2 with the database built from scratch (db:create db:migrate): 4426 runs, 0 failures, 0 errors
  • I updated documentation where needed (docs/framework/dashboard.md, CHANGELOG with the upgrade note, the generated initializer)
  • I didn't edit generated files by hand (the dummy migration and two schema lines follow the scoring lease's commit; a fresh db:create db:migrate builds the same columns)

Ownership

  • I understand every line of this change and can answer questions about it

API changes

  • Breaking change
  • New public methods or classes (AgentRun#expire_if_lost!, #renew_lease!, .expire_lost!, .lease_attributes, .execution_leases?, .lost, AgentRun::WorkerLost, AgentRunExpiryJob, config.run_lease_duration)
  • Changed method signatures (AgentRun#unless_cancelled takes an optional token:; callers without it behave as before)
  • No API changes

@TonsOfFun TonsOfFun left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — thanks! It conflicts with main now (#672/#673 just landed, CHANGELOG and agent_run.rb). Could you rebase? I'll merge once it's green.

@yorzi

yorzi commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

@TonsOfFun ready to go!

@TonsOfFun TonsOfFun left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, and for rebasing it after 1.9.8. The design reads right: the lease per segment, the fenced unless_cancelled(token:), expire-on-read plus the unscheduled job, and the execution_leases? guard for hosts that have not run the generator. One thing blocks it.

A host without migration 033 fails every run. renew_lease!, expire_if_lost!, lease_attributes and lease_release_attributes all check execution_leases? first, but three reads of the token do not:

token = run.execution_token   # AgentExecutionJob#perform, before the begin
token = run.execution_token   # AgentResumeJob#perform
token = run.execution_token   # Agent#test_execute

Without the column there is no execution_token attribute, so each of those raises NoMethodError, and the job's rescue never sees it because the read sits before the begin. That is the opposite of what the PR (rightly) promises for such hosts: "takes no leases and runs as before".

The test "nothing is expired without the lease columns" stubs execution_leases? to false, but the dummy schema still has the columns, so the attribute reader exists and the test cannot catch this.

Suggested fix: read the token through the guard, for example an instance method on AgentRun

def lease_token
  self.class.execution_leases? ? execution_token : nil
end

used in the three places; unless_cancelled(token: nil) already means "check for a cancel only". Plus a test that runs AgentExecutionJob (and ideally the resume path and test_execute) with execution_leases? false and the reader undefined, for example by stubbing AgentRun.column_names and undef-ing the attribute methods on a subclass, or by running the job against a schema without the columns.

Everything else I would leave as it is: the 30-minute default with the documented knob, failed + WorkerLost rather than a new status, and expire-on-read on the run page and list. Once the guard is in, this is in for 1.9.9 (the next host repin copies 033 with it).

@TonsOfFun

Copy link
Copy Markdown
Contributor

@yorzi a nudge on this one: the 1.9.9 release went out today without it, and I would like it in the next. Two things are left:

  1. The blocker from the 2026-11-17 review: run.execution_token is read unguarded in AgentExecutionJob#perform, AgentResumeJob#perform and Agent#test_execute, so a host that has not run migration 033 raises NoMethodError on every run. Reading it through something like AgentRun#lease_token behind execution_leases?, plus a test with the reader undefined, would cover it.
  2. It conflicts with main again after 1.9.9 (CHANGELOG at least). The ## [Unreleased] section is new as of Land the community provider fixes #712, #720, #721, #722 #724, so the entry belongs there.

If you are short on time, say so and I can push both to your branch.

A worker that dies after claiming a run leaves it running, and nothing
in the run says whether anyone is still working on it: started_at is a
start, and the progress events a generation emits do not move
updated_at. This is the first of three steps toward recovering such
runs, and changes no outcome on its own.

Migration 033 adds execution_token and execution_expires_at to
agent_runs, with an index on status and expiry. Each claim of a run for
an execution segment (AgentExecutionJob#claim_pending,
AgentResumeJob#claim_pause, Agent#test_execute) writes a new token and
an expiry ActionAgent.run_lease_duration ahead, 30 minutes by default.
Each progress event renews the expiry while the run is running under
that token. The token is per segment, so a resumed run is a new owner.

A host that has not run the migration, or sets the duration to nil,
takes no leases, detected from the columns as the scoring lease does.
unless_cancelled kept a cancel from being overwritten by the result of
a worker that was still running, but nothing else: a run ended any
other way while its worker was slow, failed as lost say, was turned
complete by the late result, with the other ending's error message
left beside the output. Agent#test_execute had no fence at all and
overwrote a cancel too.

unless_cancelled now takes the worker's lease token, and yields only
while the run still holds it. Every write that ends a segment (the
result, the pause, the failure) releases the lease, so anything that
ends the run, or a later segment that claims it under a new token,
makes a late result a no-op that is logged. AgentExecutionJob,
AgentResumeJob and test_execute pass the token their claim took;
test_execute now records through unless_cancelled like the jobs.

Without a lease (no migration 033, or no lease duration) the token is
nil and the check is the cancel alone, as before.
A run whose worker died after claiming it stayed running forever: a
redelivered job leaves a running run alone, and nothing else looked.

AgentRun#expire_if_lost! fails a running run whose lease ran out in one
conditional update on its id, status, token and expiry, so a renewal,
a result, a cancel or another sweep that lands first wins. The run is
recorded as failed with error_class ActionAgent::AgentRun::WorkerLost,
no new status, so lists, filters and scorecards read it as they read
any failure. Its message says the run was not executed again and that
its provider and tool calls may already have taken effect; nothing is
retried, and a worker that was only slow has its result discarded by
the fence.

The run's page and the run list expire the lost runs they reach, and
AgentRunExpiryJob, not scheduled by default, reaches the rest, as
input requests already expire. A run with no lease, such as one
started before migration 033, is never expired.
The dashboard guide gains a section on what happens when a run's worker
is lost: the lease each claim takes and what renews it, when a lost run
is failed and how to schedule AgentRunExpiryJob, that nothing is
executed again and what that leaves to check, the cost of a provider
call longer than the lease, and what happens to runs that predate the
migration. The changelog lists migration 033 among the upgrade steps
and describes the change under Fixed.
AgentExecutionJob, AgentResumeJob and Agent#test_execute read
run.execution_token directly. On a host that has not run migration 033
there is no such attribute, so each raised NoMethodError, and in the
jobs the read sat before the begin, so the job's rescue never recorded
it: every run failed, where the lease was meant to be simply absent.
The test for that case stubbed execution_leases? to false while the
dummy schema still had the columns, so the reader existed and the test
could not see it.

AgentRun#lease_token reads the token only when the lease columns are
installed and is nil otherwise, which unless_cancelled already takes to
mean "check for a cancel only". The three callers, renew_lease! and the
fence's own comparison read it.

RunLeaseWithoutColumnsTest drops the two columns and their index from
the table inside the test transaction and resets AgentRun's column
information, so the attribute methods are really gone, then runs the
execution job (result and failure), the resume job, test_execute
(result and a cancel during it), a progress event, the expiry paths
and the run's page and list. Reverting any of the three callers, the
guard in lease_token, lease_attributes or lease_release_attributes
fails it.
@yorzi
yorzi force-pushed the fix/agent-run-recovery branch from 1ddf840 to 1d44f56 Compare November 19, 2026 06:56
@yorzi

yorzi commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

@TonsOfFun Pushed both. Rebased on main after 1.9.9 (CHANGELOG entry now under [Unreleased]), and a fifth commit reads the token through AgentRun#lease_token, which is nil without the lease columns, in the two jobs, test_execute, renew_lease! and the fence. RunLeaseWithoutColumnsTest drops the columns and their index inside the test transaction, so the readers are really undefined, and runs both jobs, test_execute, a progress event, the expiry paths and the run page/list. Reverting any of the three callers or the column guards fails it. CI is green.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants