Repository navigation
Sessions: attempts on the explicit recovery VM, without the selector - #314
slinkydeveloper wants to merge 1 commit into
Conversation
A session object's attempts no longer run on the SDK's journaled context with a selector patched in to fake one await point over do_progress: they run on shared-core's ExplicitRecoveryVM (restatedev/sdk-shared-core#118), which never replays and has no await point at all. - src/explicit_recovery.rs: PyExplicitRecoveryVM. Commands return handles, results come back as notifications; those of the database's own handles go straight to the Rust store. - sessions/_driver.py: one attempt. One input pump, the only reader of the body; a tick after every call into the VM writes the output, settles the futures, and notices the end of the attempt (a shutdown when Restate closes the input, a failure, or the end). Cancellation: the VM's is_cancelled(), checked at every tick. An open annotation sends heartbeats, so a long model request is not shut down for inactivity. - The store over the VM syncs a commit with one acked write of the database's head (KvStore::put_and_sync), instead of an empty run. - Handler.attempt_runner: the server hands a session object's request to it. - The selector and its working run (#312) are gone: nothing suspends on the cancel signal. - shared-core is patched to the PR's branch until it is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Important
Implicit cancellation (explicit_cancellation=False, the default for session handlers) can now be silently dropped. If no future is pending when the cancellation arrives, nothing fails, and the handler runs to a normal answer. Details inline.
Reviewed changes
The whole PR at c5e1f7d: session object attempts move from the SDK's journaled context and selector onto shared-core's ExplicitRecoveryVM.
- New VM binding —
src/explicit_recovery.rsaddsPyExplicitRecoveryVM. Actions return handles, and notifications are routed either to Rust waiters (wait_in_rust) or to Python. - Attempt driver —
sessions/_driver.pyaddsAttemptDriver(input pump, the tick, cancellation, heartbeats) andVmHost.server.pyhands off toHandler.attempt_runner. - Store sync by ack —
VmStorereads, writes and lists keys directly on the VM, and syncs with an acked write of the head (or ofv22/sync).SQL_WAITbecomesSQL_COMMIT, which runs the outbox callbacks between the write and the wait for its ack. - Selector removed —
_selector.pyand its tests are deleted, along with #312's working run. - Nested annotations —
annotate(..., parent=)adds trace parents.
ℹ️ shared-core is pinned to a git branch
Cargo.toml patches restate-sdk-shared-core to the explicit-recovery-vm branch of restatedev/sdk-shared-core#118. A release built from this would depend on a moving branch. Switch to a published shared-core version before merging.
claude-opus-5-5 | 𝕏
| self.act(operator.methodcaller("cancel_invocation", call.known_invocation_id())) | ||
| self._cancelling = _quiet(asyncio.ensure_future(self.cancel_calls())) | ||
| cancelled = TerminalError("cancelled", 409) | ||
| for handle, future in list(self._futures.items()): |
There was a problem hiding this comment.
With implicit cancellation, _cancelled runs once: on the tick that first sees vm.is_cancelled(). It fails only the futures pending at that moment. If nothing is pending then, the cancellation is lost:
- Local work. The cancellation arrives while the handler awaits a model request. The later
ctx.call(...)completes normally, and its child is never cancelled. - Earlier attempt. The cancellation arrived in an earlier attempt.
start()schedules the tick beforerun()creates the handler's task, so_cancelled()sees no futures and no calls. The handler then reruns to a normal answer.
The old path delivered the cancellation at the handler's next await: the regular VM's do_progress returns CancelSignalReceived and only then flips it. I reproduced the drop with a fake VM: cancellation.done() is True, yet a call made afterwards returns its result.
Technical details
# Implicit cancellation dropped when nothing is pending at its arrival
## Affected sites
- python/restate/sessions/_driver.py:303 — `_tick` calls `_cancelled()` once, when `vm.is_cancelled()` first turns true ("acted on once")
- python/restate/sessions/_driver.py:369-387 — `_cancelled` fails only `self._futures` present now, and cancels only `self.calls` present now
- python/restate/sessions/_driver.py:146-167 — `start()` schedules the first tick before `run()` creates the handler task, so a cancellation from an earlier attempt is handled before the handler has any futures
## Required outcome
- With `explicit_cancellation=False`, a cancellation that finds nothing pending is still delivered: the handler's next Restate await fails with `TerminalError("cancelled", 409)`, and calls made from then on are cancelled. This matches the old selector and the regular VM.
- This holds both for a cancellation arriving during local work and for one received by an earlier attempt.
- Add an in-process test of each case, e.g. a fake VM whose `is_cancelled()` is true before the handler's first action. The new Docker test only covers a cancellation while a call is pending.
## Suggested approach (optional)
- Keep a "cancellation not yet delivered" flag. Clear it once `_cancelled` actually fails at least one future. While it is set, `future()` (or `act()`) fails the new future with 409 and cancels calls as they are created.|
|
||
| T = typing.TypeVar("T") | ||
|
|
||
| HEARTBEAT_INTERVAL = timedelta(seconds=20) |
There was a problem hiding this comment.
The heartbeat interval is fixed at 20s. SessionObject and its handlers accept inactivity_timeout, though. Any object configured below 20s gets its annotated work shut down and restarted. Your own daydreams test shows those restarts don't count against the retry policy, so a model request longer than the timeout restarts forever. Consider deriving the interval from the handler's or service's inactivity_timeout when one is set, e.g. a third of it, capped at 20s.
| self.calls: list[_CallFuture] = [] | ||
| # The cancellation of the calls whose invocation id a cancellation waits for. | ||
| self._cancelling: asyncio.Future[Any] | None = None | ||
| """The attempt's calls, for ``cancel_calls``.""" |
There was a problem hiding this comment.
This attribute docstring is attached to _cancelling, but it describes calls.
| self.calls: list[_CallFuture] = [] | |
| # The cancellation of the calls whose invocation id a cancellation waits for. | |
| self._cancelling: asyncio.Future[Any] | None = None | |
| """The attempt's calls, for ``cancel_calls``.""" | |
| self.calls: list[_CallFuture] = [] | |
| """The attempt's calls, for ``cancel_calls``.""" | |
| # The cancellation of the calls whose invocation id a cancellation waits for. | |
| self._cancelling: asyncio.Future[Any] | None = None |

Session objects now run their attempts on shared-core's new
ExplicitRecoveryVM(restatedev/sdk-shared-core#118), instead of the SDK's journaled context with a selector patched in.Changes
PyExplicitRecoveryVM(src/explicit_recovery.rs). Commands return handles, and results come back as notifications. The database's own handles are settled in Rust.sessions/_driver.py):is_cancelled();syncrun.Testing
pytest tests/*: 691 passed. ruff, mypy, pyright and clippy are clean.basic/notes.py.restate.envhas expired.🤖 Generated with Claude Code