Skip to content

refactor(pydantic-ai)!: Require entering PydanticAiSelectorExtractor before extract() - #2448

Open
Mantisus wants to merge 1 commit into
apify:v2from
Mantisus:selector-extractor-lifecycle
Open

Mantisus wants to merge 1 commit into
apify:v2from
Mantisus:selector-extractor-lifecycle

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • PydanticAiSelectorExtractor loads its selector cache when it's entered with async with and saves it when it's exited. extract() raises a RuntimeError on an extractor that isn't entered.
  • It enters and exits its fallback together with itself. A failed or cancelled entry exits whatever it already entered.
  • A fallback that several selector extractors use one after another counts its usage into the extractor that is using it.

Issues

Testing

  • Unit tests for entering and exiting the selector extractor and its fallback chain, failures at each step, and a fallback used by several extractors.

✍️ Drafted by Claude Code

@Mantisus
Mantisus marked this pull request as draft November 15, 2026 00:40
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 15, 2026
@Mantisus Mantisus self-assigned this Nov 15, 2026
@Mantisus
Mantisus force-pushed the selector-extractor-lifecycle branch from f788375 to 4a18870 Compare November 15, 2026 22:35
@Mantisus
Mantisus force-pushed the selector-extractor-lifecycle branch from 4a18870 to d5c7fe4 Compare November 18, 2026 14:31
@Mantisus
Mantisus marked this pull request as ready for review November 18, 2026 14:31
@Mantisus
Mantisus requested a review from vdusek November 18, 2026 14:31
@vdusek vdusek changed the title refactor(pydantic-ai)!: Require entering the selector extractor before extract() refactor(pydantic-ai)!: Require entering PydanticAiSelectorExtractor before extract() Nov 18, 2026
self._init_lock = asyncio.Lock()
self._active = False

def set_ai_usage(self, value: PydanticAiUsageStats) -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: set_ai_usage re-shares with the fallback even when this extractor isn't entered. If first holds a shared fallback and someone calls second.set_ai_usage(x), the fallback calls that first makes count into x. That breaks "counts into whichever extractor holds it". Re-share only when self.active, since __aenter__ shares it again anyway.

✍️ Drafted by Claude Code

extractor = PydanticAiDirectExtractor(model=model)
product = await extractor.extract('<html>...</html>', Product)

async with extractor:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: The example now enters PydanticAiDirectExtractor, but its extract() has no @ensure_context and works without entering. Should both extractors follow one rule, e.g. @ensure_context on the direct extractor too? Or is the difference on purpose?

✍️ Drafted by Claude Code


- `SessionPool`, `BrowserPool`, an HTTP client and the `rendering_type_predictor` of `AdaptivePlaywrightCrawler`: enter it with `async with` before you run the crawler, or `run()` raises a `RuntimeError`. Keep it open until every crawler using it has finished.
- The `extractor` of `PydanticAiCrawler`: enter it the same way. `PydanticAiDirectExtractor` is now an async context manager, like `PydanticAiSelectorExtractor`. A selector extractor enters and exits its `fallback` itself, so don't enter the fallback yourself, and give each selector extractor its own. If your subclass of `BasePydanticAiHtmlExtractor` overrides `__aenter__` or `__aexit__`, call `super()`. A custom extractor that implements the `PydanticAiHtmlExtractor` protocol directly needs the new `active` property, `__aenter__` and `__aexit__`.
- The `extractor` of `PydanticAiCrawler`: enter it the same way. Used on its own, `PydanticAiSelectorExtractor` has to be entered before `extract()` too, or it raises a `RuntimeError`. `PydanticAiDirectExtractor` is now an async context manager, like `PydanticAiSelectorExtractor`. A selector extractor enters and exits its `fallback` itself, so don't enter the fallback yourself, and give each selector extractor its own. If your subclass of `BasePydanticAiHtmlExtractor` overrides `__aenter__` or `__aexit__`, call `super()`. A custom extractor that implements the `PydanticAiHtmlExtractor` protocol directly needs the new `active` property, `__aenter__` and `__aexit__`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: "give each selector extractor its own" is stricter than what this PR supports. test_fallback_used_in_turn shares one fallback between extractors entered one after another, and only entering them at the same time fails. Maybe: "don't share one fallback between selector extractors that are entered at the same time".

✍️ Drafted by Claude Code

with pytest.raises(RuntimeError, match='already active'):
await second.__aenter__()

assert second.active is False

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: The main point of this test is that a failed entry doesn't exit a fallback that another extractor holds, but nothing asserts it.

Suggested change
assert second.active is False
assert second.active is False
assert fallback.active is True

✍️ Drafted by Claude Code

Comment on lines +258 to +259
The cache is a `RecoverableState` persisted to a `KeyValueStore`. It loads when the extractor is entered with
`async with` and is saved when it's exited, so `extract` needs an entered extractor.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: RecoverableState.initialize() doesn't reload the record after teardown(), so the cache loads only on the first entry. A second entry keeps the in-memory cache.

Suggested change
The cache is a `RecoverableState` persisted to a `KeyValueStore`. It loads when the extractor is entered with
`async with` and is saved when it's exited, so `extract` needs an entered extractor.
The cache is a `RecoverableState` persisted to a `KeyValueStore`. It loads when the extractor is first entered
with `async with` and is saved when it's exited, so `extract` needs an entered extractor.

✍️ Drafted by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let crawlers manage only the collaborators they create and extract per-run state

2 participants