Repository navigation
Conversation
|
@TonsOfFun ready to go! |
096b278 to
1ddf840
Compare
TonsOfFun
left a comment
There was a problem hiding this comment.
Thanks for this, and for rebasing it after 1.9.8. The design reads right: the lease per segment, the fenced unless_cancelled(token:), expire-on-read plus the unscheduled job, and the execution_leases? guard for hosts that have not run the generator. One thing blocks it.
A host without migration 033 fails every run. renew_lease!, expire_if_lost!, lease_attributes and lease_release_attributes all check execution_leases? first, but three reads of the token do not:
token = run.execution_token # AgentExecutionJob#perform, before the begin
token = run.execution_token # AgentResumeJob#perform
token = run.execution_token # Agent#test_executeWithout the column there is no execution_token attribute, so each of those raises NoMethodError, and the job's rescue never sees it because the read sits before the begin. That is the opposite of what the PR (rightly) promises for such hosts: "takes no leases and runs as before".
The test "nothing is expired without the lease columns" stubs execution_leases? to false, but the dummy schema still has the columns, so the attribute reader exists and the test cannot catch this.
Suggested fix: read the token through the guard, for example an instance method on AgentRun
def lease_token
self.class.execution_leases? ? execution_token : nil
endused in the three places; unless_cancelled(token: nil) already means "check for a cancel only". Plus a test that runs AgentExecutionJob (and ideally the resume path and test_execute) with execution_leases? false and the reader undefined, for example by stubbing AgentRun.column_names and undef-ing the attribute methods on a subclass, or by running the job against a schema without the columns.
Everything else I would leave as it is: the 30-minute default with the documented knob, failed + WorkerLost rather than a new status, and expire-on-read on the run page and list. Once the guard is in, this is in for 1.9.9 (the next host repin copies 033 with it).
|
@yorzi a nudge on this one: the 1.9.9 release went out today without it, and I would like it in the next. Two things are left:
If you are short on time, say so and I can push both to your branch. |
A worker that dies after claiming a run leaves it running, and nothing in the run says whether anyone is still working on it: started_at is a start, and the progress events a generation emits do not move updated_at. This is the first of three steps toward recovering such runs, and changes no outcome on its own. Migration 033 adds execution_token and execution_expires_at to agent_runs, with an index on status and expiry. Each claim of a run for an execution segment (AgentExecutionJob#claim_pending, AgentResumeJob#claim_pause, Agent#test_execute) writes a new token and an expiry ActionAgent.run_lease_duration ahead, 30 minutes by default. Each progress event renews the expiry while the run is running under that token. The token is per segment, so a resumed run is a new owner. A host that has not run the migration, or sets the duration to nil, takes no leases, detected from the columns as the scoring lease does.
unless_cancelled kept a cancel from being overwritten by the result of a worker that was still running, but nothing else: a run ended any other way while its worker was slow, failed as lost say, was turned complete by the late result, with the other ending's error message left beside the output. Agent#test_execute had no fence at all and overwrote a cancel too. unless_cancelled now takes the worker's lease token, and yields only while the run still holds it. Every write that ends a segment (the result, the pause, the failure) releases the lease, so anything that ends the run, or a later segment that claims it under a new token, makes a late result a no-op that is logged. AgentExecutionJob, AgentResumeJob and test_execute pass the token their claim took; test_execute now records through unless_cancelled like the jobs. Without a lease (no migration 033, or no lease duration) the token is nil and the check is the cancel alone, as before.
A run whose worker died after claiming it stayed running forever: a redelivered job leaves a running run alone, and nothing else looked. AgentRun#expire_if_lost! fails a running run whose lease ran out in one conditional update on its id, status, token and expiry, so a renewal, a result, a cancel or another sweep that lands first wins. The run is recorded as failed with error_class ActionAgent::AgentRun::WorkerLost, no new status, so lists, filters and scorecards read it as they read any failure. Its message says the run was not executed again and that its provider and tool calls may already have taken effect; nothing is retried, and a worker that was only slow has its result discarded by the fence. The run's page and the run list expire the lost runs they reach, and AgentRunExpiryJob, not scheduled by default, reaches the rest, as input requests already expire. A run with no lease, such as one started before migration 033, is never expired.
The dashboard guide gains a section on what happens when a run's worker is lost: the lease each claim takes and what renews it, when a lost run is failed and how to schedule AgentRunExpiryJob, that nothing is executed again and what that leaves to check, the cost of a provider call longer than the lease, and what happens to runs that predate the migration. The changelog lists migration 033 among the upgrade steps and describes the change under Fixed.
AgentExecutionJob, AgentResumeJob and Agent#test_execute read run.execution_token directly. On a host that has not run migration 033 there is no such attribute, so each raised NoMethodError, and in the jobs the read sat before the begin, so the job's rescue never recorded it: every run failed, where the lease was meant to be simply absent. The test for that case stubbed execution_leases? to false while the dummy schema still had the columns, so the reader existed and the test could not see it. AgentRun#lease_token reads the token only when the lease columns are installed and is nil otherwise, which unless_cancelled already takes to mean "check for a cancel only". The three callers, renew_lease! and the fence's own comparison read it. RunLeaseWithoutColumnsTest drops the two columns and their index from the table inside the test transaction and resets AgentRun's column information, so the attribute methods are really gone, then runs the execution job (result and failure), the resume job, test_execute (result and a cancel during it), a progress event, the expiry paths and the run's page and list. Reverting any of the three callers, the guard in lease_token, lease_attributes or lease_release_attributes fails it.
1ddf840 to
1d44f56
Compare
|
@TonsOfFun Pushed both. Rebased on main after 1.9.9 (CHANGELOG entry now under [Unreleased]), and a fifth commit reads the token through AgentRun#lease_token, which is nil without the lease columns, in the two jobs, test_execute, renew_lease! and the fence. RunLeaseWithoutColumnsTest drops the columns and their index inside the test transaction, so the readers are really undefined, and runs both jobs, test_execute, a progress event, the expiry paths and the run page/list. Reverting any of the three callers or the column guards fails it. CI is green. |
What this does
Since #664 a run is claimed once, so a redelivered job never runs it twice. The other side of that, which #664 left out of scope: a worker that dies after its claim (a kill, an out-of-memory error, a deploy that does not wait for jobs) leaves the run
runningfor good. The redelivery sees a running run and leaves it, and nothing else ever looks at it. The same happens afterAgentResumeJobclaims a paused run, and for a web process that dies insideAgent#test_execute. Nothing in the run says whether anyone is still working on it:started_atis a start, andappend_eventwrites withupdate_column, soupdated_atdoes not move while the run works.Ending such a run is only safe by cancelling it.
unless_cancelleddiscards a late result over a cancel, but any other ending is overwritten. A run failed by anything other than a cancel while its worker was only slow came backcomplete, with the late output and the other ending's error message side by side.This follows two things already in the engine: the scoring lease from #614 (a token, an expiry, a fenced write, column detection for hosts that have not migrated) and
InputRequestExpiryJob(expired when read, plus an optional recurring job). It is five commits:execution_tokenandexecution_expires_attoagent_runs, indexed withstatus. Every claim of a run for an execution segment writes a new token and an expiryconfig.run_lease_durationahead (30 minutes by default). Those claims areclaim_pending,claim_pauseand the creation intest_execute. Each progress event (one per LLM or tool call) renews the expiry while the run is running under that token. Without the columns, or with the duration set tonil, no lease is taken.unless_cancelledtakes the worker's token and yields only while the run still holds it. Every write that ends a segment releases the lease (result, pause, failure), so whatever ended the run first is kept, and a late result is a logged no-op.test_executenow records through it as the jobs do, which also stops it overwriting a cancel.AgentRun#expire_if_lost!is one conditional update on id, status, token and expiry. It recordsfailedwitherror_classActionAgent::AgentRun::WorkerLostand a message that says the run was not executed again. The run's page and the run list expire what they reach.ActionAgent::AgentRunExpiryJobreaches the rest and is not scheduled by default. A run with no lease, such as one running when the migration is installed, is never expired.docs/framework/dashboard.md, the upgrade note and a Fixed entry in the CHANGELOG's[Unreleased]section, and the option in the generated initializer.run.execution_tokendirectly, which raisedNoMethodErroron every run on a host without migration 033.AgentRun#lease_tokenreturns the token only when the lease columns are installed andnilotherwise; the workers,renew_lease!and the fence read it.Nothing is executed again, by design. The worker may have died after the provider billed the call or after a tool acted, so the run's message and the docs say so, and running the prompt again is a new run made by a person.
Choices made here that are yours to change:
failedwitherror_classWorkerLost(whatrecord_failure!already writes), so lists, filters and scorecards need no change. The alternative is a newinterruptedstatus.InputRequestExpiryJob. Without a scheduler, a lost run nobody opens staysrunning, as today.JUDGE_LEASE_DURATION, renewed only by progress events. A single provider call longer than that looks lost, and its result is then discarded. A heartbeat thread would avoid that but would also keep a wedged process alive. Raising the setting is the documented answer.Tests, all in
actionagent/test/run_lease_test.rbplus one ininstall_generator_test.rb, 39 in all:test_executeeach take a lease; two claims never share a token; an event renews; a renewal leaves a run that finished or changed token alone; no lease without a duration or without the columns; the migration installs under a custom prefix and reverses.test_execute. A cancel duringtest_executestays. A worker whose run was taken under another token records nothing. A completed, paused or failed segment releases its lease. Plus one real-connection test: the worker held in its generation on its own connection while the run is failed as lost on another.RunLeaseWithoutColumnsTest): the two columns and their index are dropped inside the test transaction and the column information reset, so the attribute methods are really gone. The execution job (result and failure), the resume job,test_execute(result, and a cancel during it), a progress event, the expiry paths and the run's page and list all work as before leases. Reverting any of the three callers, or the guard inlease_token,lease_attributesorlease_release_attributes, fails it.WorkerLost, its existing metadata kept, announced once, one log line. Not executed again. A live run, a renewed one or one re-taken since it was read is left running, and a live run costs no query. Only running runs expire, never one without a lease, and nothing without the columns. A cancel that lands first wins. The job fails every lost run and only those. The run's page and the run list show it failed. A late worker's result is discarded after expiry. A run that keeps reporting progress outlasts the lease. Plus a real-connection race of two sweeps (one transition, one broadcast).Each part of each commit reverted in turn fails at least one test (7 parts in the lease commit, 8 in the fence, 13 in the expiry, 6 in the guard). Each commit passes its own tests when checked out alone.
Backends: SQLite on Rails 8.1.4 and 7.2.3.2. PostgreSQL 17 through a standalone Active Record probe of the same method bodies, since the dummy schema does not load on PostgreSQL (an index name over 63 characters). One claim in 50 races. The expiry writes the JSON metadata through
update_all. Two sweeps make one transition in 50 races. A renewal racing a sweep was consistent in 50 races. A late worker is fenced whichever commits first. The engine suite on PostgreSQL itself is not run.Not in this PR: stopping a generation early when its renewal fails (it would save spend, but changes no guarantee); a "run again" action; a task for runs that predate the migration (cancel them).
Type of change
Scope check
Related issue
None filed. Follow-up to #664, which names this gap as out of scope. Precedents: #614.
Quality check
bin/rubocopon the changed files; the repo has no commit hooks installed)bin/testwith the CI placeholder keys, rebased on main after 1.9.9 (Rails 8.1.4): 4683 runs, 0 failures, 11 errors, allCliContractTest(rack-sessionundefined method 'id' for Hash). Those 11 fail identically on an untouchedmainin the same checkout and pass in a fresh worktree ofmain, so they come from local state, not this change; CI is the check. Before the rebase, Rails 7.2 with the database built from scratch (db:create db:migrate): 4426 runs, 0 failures, 0 errorsdocs/framework/dashboard.md, CHANGELOG with the upgrade note, the generated initializer)db:create db:migratebuilds the same columns)Ownership
API changes
AgentRun#expire_if_lost!,#renew_lease!,.expire_lost!,.lease_attributes,.execution_leases?,.lost,AgentRun::WorkerLost,AgentRunExpiryJob,config.run_lease_duration)AgentRun#unless_cancelledtakes an optionaltoken:; callers without it behave as before)