Skip to content

fix(temporal): stop a failing signal handler from wedging its workflow - #569

Closed
aringuyen3 wants to merge 3 commits into
mainfrom
aringuyen/temporal-signal-handler-guards
Closed

aringuyen3 wants to merge 3 commits into
mainfrom
aringuyen/temporal-signal-handler-guards

Conversation

@aringuyen3

@aringuyen3 aringuyen3 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When a Temporal signal handler raises an exception that Temporal does not treat as a workflow failure, the workflow task fails. Temporal retries that task indefinitely and replays the same signal each time, so the workflow stops making progress and ignores every later signal. The tutorial agents, and projects created from the base temporal template, did this for any event whose content they did not expect: content=None, a DataContent, a ToolResponseContent holding a dict, and so on.

This PR fixes it in two layers.

SDK: SignalHandlerErrorInterceptor

agentex.lib.core.temporal.workers.signal_errors.SignalHandlerErrorInterceptor is new, and AgentexWorker installs it by default (drop_signals_on_handler_error=False opts out).

  • An unexpected exception from a signal handler is logged once (replay-safe, with the signal name and the workflow and run IDs, but not the signal's arguments) and the signal is dropped. Temporal already applies the same policy to a signal whose arguments fail to deserialize.
  • Re-raised unchanged:
    • anything workflow.is_failure_exception accepts: ApplicationError and other FailureErrors, asyncio.TimeoutError, declared failure types;
    • the SDK's own errors that are not declared failure types (NondeterminismError, the sandbox's RestrictedWorkflowAccessError, ReadOnlyContextError, ...), also inside an ExceptionGroup;
    • BaseExceptions: cancellation, continue-as-new, eviction.
  • Any other exception group is dropped, such as one from an asyncio.TaskGroup whose child raised an ApplicationError or a declared failure type (the OpenAI Agents plugin's AgentsWorkflowError, for example). Temporal does not look inside groups, so re-raising it would fail the task forever rather than fail the workflow.
  • It is the first of the worker's own interceptors, so user interceptors still see the exception and can turn it into an ApplicationError to fail the workflow on purpose. Traces are unchanged: the OpenTelemetry interceptor only records FailureErrors on signal spans.
  • The drop is logged with the sandbox's restrictions lifted, so handlers that import lazily while emitting (Rich's, for example) still emit it under the default sandboxed runner. If logging itself fails, one line with the signal name and the workflow and run IDs goes to stderr instead.

Tutorials, the agentex init template, the procurement demo

  • Every Temporal tutorial's on_task_event_send (000 to 150), the base temporal template and the procurement demo check the event before doing anything, and log and return for content they cannot process. That means anything other than TextContent in most handlers. 010 and 050 also skip events the user did not write, 030 also takes its DataContent queue controls, and the handlers that echo every event (000, the template, the procurement demo) skip only missing content.
  • raise ValueError("State is not initialized") becomes a wait for on_task_create, since a signal can be handled before the run method starts (with signal-with-start, for example).
  • The procurement demo's send_event signal drops malformed payloads instead of raising, now also for integer literals beyond Python's digit limit and for deeply nested JSON.
  • span.output is only set when tracing returned a span (060, 070, 080, 100).
  • Every tutorial's tests/test_agent.py gets offline signal-handler tests. Their setup (import path and env defaults) is a module-scoped fixture in the tutorial's conftest.py that applies only to that tutorial's tests, so one pytest session can run several tutorials. 150_codex's conftest moves to the same pattern, and 090's project/ gets the empty __init__.py every other tutorial has.

Behavior changes

  • A genuine bug in a signal handler (an AttributeError, say) now drops that signal with an ERROR log (Dropped signal ...) instead of failing the workflow task until a fix ships. Consider alerting on that log line.
  • A drop is not a rollback: whatever the handler did before it raised is kept.
  • Once a signal has been dropped, replaying that history without the interceptor fails the task again. That happens after opting out, after rolling back to an earlier SDK, or with a Replayer built without it. Replay tests should pass interceptors=[SignalHandlerErrorInterceptor()].
  • Tutorial workflows that processed a non-text event under the previous code may hit a nondeterminism error when replayed with the new handlers. Reset or terminate those.

Testing

  • ./scripts/lint: ruff and pyright are clean. ./scripts/check-slim-deps passes. ./scripts/test: 1855 passed on Python 3.12 and on 3.13.
  • New SDK tests:
    • unit tests against a fake inbound chain;
    • Replayer tests on hand-built histories, including exception groups from an asyncio.TaskGroup and a worker-declared TemporalError (these run in CI, which has no Temporal server);
    • end-to-end tests on a local Temporal dev server, including asyncio.TaskGroup handlers whose child raises an ApplicationError or a declared TemporalError, skipped when the temporal CLI is not on PATH, as in CI;
    • worker-wiring tests.
  • Each tutorial's offline tests pass on their own (pytest, python -m pytest, and CI's uv run command with the built wheels) and all together in one session; the new ones fail against the previous handlers. 100_gemini_litellm has no manifest.yaml, so CI does not run its tests, and nothing in CI runs the procurement demo's handler; both were checked locally. The live tutorial tests need an AgentEx server and were not run locally.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; both previous findings are fixed, and no new actionable issues were found.

Summary

Adds SignalHandlerErrorInterceptor to keep unexpected signal-handler errors from stopping later signals.

  • Unexpected signal-handler errors are logged and dropped so workflows can continue.
  • Tutorial workflows skip unusable events and wait for setup when needed.
  • The procurement workflow waits for setup and ignores events with no content.
  • Malformed procurement signals are logged and dropped instead of stopping the workflow.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Signal arrives] --> B[Run caller interceptors and handler]
    B --> C{Handler raises?}
    C -->|No| D[Return normally]
    C -->|Yes| E{Must Temporal receive the error?}
    E -->|Yes| F[Re-raise unchanged]
    E -->|No| G[Log and drop the signal]
    G --> H[Workflow can handle later signals]
Loading

Reviews (2) · Last reviewed commit: "fix(examples): address review notes on t..." · Reviewed by Greptile

aringuyen3 and others added 2 commits November 13, 2026 03:26
When a signal handler raises an exception that Temporal does not treat as
a workflow failure, the workflow task fails and Temporal retries it
forever, replaying the same signal each time: the workflow stops making
progress and ignores every later signal.

AgentexWorker now installs SignalHandlerErrorInterceptor by default
(drop_signals_on_handler_error=False opts out). It logs such an exception
once, replay-safely and without the signal's arguments, and drops the
signal. Failure errors, declared failure types, the SDK's own
rule-violation errors (also inside exception groups), cancellation and
continue-as-new keep their usual outcome. Any other exception group is
dropped as well, such as one from an asyncio.TaskGroup whose child raised
an ApplicationError or a declared failure type: Temporal does not look
inside groups, so it would fail the task forever rather than fail the
workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Temporal tutorial (000-150), the base temporal template used by
`agentex init` and the procurement demo now check an event before acting
on it, and log and return for content they cannot process instead of
raising on it or dereferencing it: anything but TextContent in most
handlers (010 and 050 also skip events the user did not write, and 030
also takes its DataContent queue controls), and only missing content in
the handlers that echo every event. A signal that arrives before
on_task_create waits for the state instead of raising, the procurement
demo drops malformed send_event payloads, and span.output is only set
when tracing returned a span.

Each tutorial gets offline signal-handler tests. Their setup is a
module-scoped fixture in the tutorial's conftest.py that applies only to
that tutorial's tests, so one pytest session can run several tutorials;
150_codex's conftest moves to the same pattern, and 090's project/ gets
the empty __init__.py every other tutorial has.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Each tutorial's conftest fixture also clears the SDK's cached
  EnvironmentVariables, so in a session that runs several tutorials each
  workflow module reads its own settings.
- The new offline tests import SendEventParams from agentex.protocol.acp,
  the canonical path.
- Sort the imports in 090's run_worker.py now that its project/ is a
  regular package (ruff I001, which a warm ruff cache had hidden locally).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aringuyen3 aringuyen3 closed this Nov 13, 2026
@aringuyen3
aringuyen3 deleted the aringuyen/temporal-signal-handler-guards branch November 13, 2026 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants