Repository navigation
fix(temporal): stop a failing signal handler from wedging its workflow - #569
Closed
aringuyen3 wants to merge 3 commits into
Closed
aringuyen3 wants to merge 3 commits into
aringuyen3 wants to merge 3 commits into
Conversation
When a signal handler raises an exception that Temporal does not treat as a workflow failure, the workflow task fails and Temporal retries it forever, replaying the same signal each time: the workflow stops making progress and ignores every later signal. AgentexWorker now installs SignalHandlerErrorInterceptor by default (drop_signals_on_handler_error=False opts out). It logs such an exception once, replay-safely and without the signal's arguments, and drops the signal. Failure errors, declared failure types, the SDK's own rule-violation errors (also inside exception groups), cancellation and continue-as-new keep their usual outcome. Any other exception group is dropped as well, such as one from an asyncio.TaskGroup whose child raised an ApplicationError or a declared failure type: Temporal does not look inside groups, so it would fail the task forever rather than fail the workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Temporal tutorial (000-150), the base temporal template used by `agentex init` and the procurement demo now check an event before acting on it, and log and return for content they cannot process instead of raising on it or dereferencing it: anything but TextContent in most handlers (010 and 050 also skip events the user did not write, and 030 also takes its DataContent queue controls), and only missing content in the handlers that echo every event. A signal that arrives before on_task_create waits for the state instead of raising, the procurement demo drops malformed send_event payloads, and span.output is only set when tracing returned a span. Each tutorial gets offline signal-handler tests. Their setup is a module-scoped fixture in the tutorial's conftest.py that applies only to that tutorial's tests, so one pytest session can run several tutorials; 150_codex's conftest moves to the same pattern, and 090's project/ gets the empty __init__.py every other tutorial has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Each tutorial's conftest fixture also clears the SDK's cached EnvironmentVariables, so in a session that runs several tutorials each workflow module reads its own settings. - The new offline tests import SendEventParams from agentex.protocol.acp, the canonical path. - Sort the imports in 090's run_worker.py now that its project/ is a regular package (ruff I001, which a warm ruff cache had hidden locally). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
danielmillerp
approved these changes
Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a Temporal signal handler raises an exception that Temporal does not treat as a workflow failure, the workflow task fails. Temporal retries that task indefinitely and replays the same signal each time, so the workflow stops making progress and ignores every later signal. The tutorial agents, and projects created from the base
temporaltemplate, did this for any event whose content they did not expect:content=None, aDataContent, aToolResponseContentholding a dict, and so on.This PR fixes it in two layers.
SDK:
SignalHandlerErrorInterceptoragentex.lib.core.temporal.workers.signal_errors.SignalHandlerErrorInterceptoris new, andAgentexWorkerinstalls it by default (drop_signals_on_handler_error=Falseopts out).workflow.is_failure_exceptionaccepts:ApplicationErrorand otherFailureErrors,asyncio.TimeoutError, declared failure types;NondeterminismError, the sandbox'sRestrictedWorkflowAccessError,ReadOnlyContextError, ...), also inside anExceptionGroup;BaseExceptions: cancellation, continue-as-new, eviction.asyncio.TaskGroupwhose child raised anApplicationErroror a declared failure type (the OpenAI Agents plugin'sAgentsWorkflowError, for example). Temporal does not look inside groups, so re-raising it would fail the task forever rather than fail the workflow.ApplicationErrorto fail the workflow on purpose. Traces are unchanged: the OpenTelemetry interceptor only recordsFailureErrors on signal spans.Tutorials, the
agentex inittemplate, the procurement demoon_task_event_send(000 to 150), the basetemporaltemplate and the procurement demo check the event before doing anything, and log and return for content they cannot process. That means anything other thanTextContentin most handlers. 010 and 050 also skip events the user did not write, 030 also takes itsDataContentqueue controls, and the handlers that echo every event (000, the template, the procurement demo) skip only missing content.raise ValueError("State is not initialized")becomes a wait foron_task_create, since a signal can be handled before the run method starts (with signal-with-start, for example).send_eventsignal drops malformed payloads instead of raising, now also for integer literals beyond Python's digit limit and for deeply nested JSON.span.outputis only set when tracing returned a span (060, 070, 080, 100).tests/test_agent.pygets offline signal-handler tests. Their setup (import path and env defaults) is a module-scoped fixture in the tutorial'sconftest.pythat applies only to that tutorial's tests, so one pytest session can run several tutorials.150_codex's conftest moves to the same pattern, and090'sproject/gets the empty__init__.pyevery other tutorial has.Behavior changes
AttributeError, say) now drops that signal with an ERROR log (Dropped signal ...) instead of failing the workflow task until a fix ships. Consider alerting on that log line.Replayerbuilt without it. Replay tests should passinterceptors=[SignalHandlerErrorInterceptor()].Testing
./scripts/lint: ruff and pyright are clean../scripts/check-slim-depspasses../scripts/test: 1855 passed on Python 3.12 and on 3.13.Replayertests on hand-built histories, including exception groups from anasyncio.TaskGroupand a worker-declaredTemporalError(these run in CI, which has no Temporal server);asyncio.TaskGrouphandlers whose child raises anApplicationErroror a declaredTemporalError, skipped when thetemporalCLI is not onPATH, as in CI;pytest,python -m pytest, and CI'suv runcommand with the built wheels) and all together in one session; the new ones fail against the previous handlers.100_gemini_litellmhas nomanifest.yaml, so CI does not run its tests, and nothing in CI runs the procurement demo's handler; both were checked locally. The live tutorial tests need an AgentEx server and were not run locally.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; both previous findings are fixed, and no new actionable issues were found.Summary
Adds
SignalHandlerErrorInterceptorto keep unexpected signal-handler errors from stopping later signals.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Signal arrives] --> B[Run caller interceptors and handler] B --> C{Handler raises?} C -->|No| D[Return normally] C -->|Yes| E{Must Temporal receive the error?} E -->|Yes| F[Re-raise unchanged] E -->|No| G[Log and drop the signal] G --> H[Workflow can handle later signals]Reviews (2) · Last reviewed commit: "fix(examples): address review notes on t..." · Reviewed by Greptile