Repository navigation
Conversation
f788375 to
4a18870
Compare
4a18870 to
d5c7fe4
Compare
extract()PydanticAiSelectorExtractor before extract()
| self._init_lock = asyncio.Lock() | ||
| self._active = False | ||
|
|
||
| def set_ai_usage(self, value: PydanticAiUsageStats) -> None: |
There was a problem hiding this comment.
Suggestion: set_ai_usage re-shares with the fallback even when this extractor isn't entered. If first holds a shared fallback and someone calls second.set_ai_usage(x), the fallback calls that first makes count into x. That breaks "counts into whichever extractor holds it". Re-share only when self.active, since __aenter__ shares it again anyway.
✍️ Drafted by Claude Code
| extractor = PydanticAiDirectExtractor(model=model) | ||
| product = await extractor.extract('<html>...</html>', Product) | ||
|
|
||
| async with extractor: |
There was a problem hiding this comment.
Question: The example now enters PydanticAiDirectExtractor, but its extract() has no @ensure_context and works without entering. Should both extractors follow one rule, e.g. @ensure_context on the direct extractor too? Or is the difference on purpose?
✍️ Drafted by Claude Code
|
|
||
| - `SessionPool`, `BrowserPool`, an HTTP client and the `rendering_type_predictor` of `AdaptivePlaywrightCrawler`: enter it with `async with` before you run the crawler, or `run()` raises a `RuntimeError`. Keep it open until every crawler using it has finished. | ||
| - The `extractor` of `PydanticAiCrawler`: enter it the same way. `PydanticAiDirectExtractor` is now an async context manager, like `PydanticAiSelectorExtractor`. A selector extractor enters and exits its `fallback` itself, so don't enter the fallback yourself, and give each selector extractor its own. If your subclass of `BasePydanticAiHtmlExtractor` overrides `__aenter__` or `__aexit__`, call `super()`. A custom extractor that implements the `PydanticAiHtmlExtractor` protocol directly needs the new `active` property, `__aenter__` and `__aexit__`. | ||
| - The `extractor` of `PydanticAiCrawler`: enter it the same way. Used on its own, `PydanticAiSelectorExtractor` has to be entered before `extract()` too, or it raises a `RuntimeError`. `PydanticAiDirectExtractor` is now an async context manager, like `PydanticAiSelectorExtractor`. A selector extractor enters and exits its `fallback` itself, so don't enter the fallback yourself, and give each selector extractor its own. If your subclass of `BasePydanticAiHtmlExtractor` overrides `__aenter__` or `__aexit__`, call `super()`. A custom extractor that implements the `PydanticAiHtmlExtractor` protocol directly needs the new `active` property, `__aenter__` and `__aexit__`. |
There was a problem hiding this comment.
Note: "give each selector extractor its own" is stricter than what this PR supports. test_fallback_used_in_turn shares one fallback between extractors entered one after another, and only entering them at the same time fails. Maybe: "don't share one fallback between selector extractors that are entered at the same time".
✍️ Drafted by Claude Code
| with pytest.raises(RuntimeError, match='already active'): | ||
| await second.__aenter__() | ||
|
|
||
| assert second.active is False |
There was a problem hiding this comment.
Suggestion: The main point of this test is that a failed entry doesn't exit a fallback that another extractor holds, but nothing asserts it.
| assert second.active is False | |
| assert second.active is False | |
| assert fallback.active is True |
✍️ Drafted by Claude Code
| The cache is a `RecoverableState` persisted to a `KeyValueStore`. It loads when the extractor is entered with | ||
| `async with` and is saved when it's exited, so `extract` needs an entered extractor. |
There was a problem hiding this comment.
Nit: RecoverableState.initialize() doesn't reload the record after teardown(), so the cache loads only on the first entry. A second entry keeps the in-memory cache.
| The cache is a `RecoverableState` persisted to a `KeyValueStore`. It loads when the extractor is entered with | |
| `async with` and is saved when it's exited, so `extract` needs an entered extractor. | |
| The cache is a `RecoverableState` persisted to a `KeyValueStore`. It loads when the extractor is first entered | |
| with `async with` and is saved when it's exited, so `extract` needs an entered extractor. |
✍️ Drafted by Claude Code
Description
PydanticAiSelectorExtractorloads its selector cache when it's entered withasync withand saves it when it's exited.extract()raises aRuntimeErroron an extractor that isn't entered.fallbacktogether with itself. A failed or cancelled entry exits whatever it already entered.Issues
Testing
✍️ Drafted by Claude Code