Repository navigation
fix(crawlers)!: Count requests in the statistics passed to AdaptivePlaywrightCrawler factory methods - #2447
fix(crawlers)!: Count requests in the statistics passed to AdaptivePlaywrightCrawler factory methods#2447Mantisus wants to merge 1 commit into
AdaptivePlaywrightCrawler factory methods#2447Conversation
bd176e9 to
fa2fa4b
Compare
… require the adaptive state model
fa2fa4b to
a6a9c44
Compare
AdaptivePlaywrightCrawler factory methods
vdusek
left a comment
There was a problem hiding this comment.
Note: docs/03_concepts/22_statistics.mdx:65 (outside the diff) still says to use Statistics.with_default_state to get an instance. With this PR, such an instance makes AdaptivePlaywrightCrawler raise a ValueError. Can we add a sentence that the adaptive crawler needs Statistics(state_model=AdaptivePlaywrightCrawlerStatisticState)?
✍️ Drafted by Claude Code
|
|
||
| When a crawler runs again, it resets only a session pool it created. A `SessionPool` you pass keeps its sessions. In v1, a repeated `run()` reset a passed pool too. | ||
|
|
||
| The `with_beautifulsoup_static_parser` and `with_parsel_static_parser` factory methods of `AdaptivePlaywrightCrawler` pass `statistics` to the crawler as they are, instead of a copy built with `Statistics.replace_state_model`, so the instance you pass is the one that counts. The instance must use `AdaptivePlaywrightCrawlerStatisticState` as its state model: |
There was a problem hiding this comment.
Nit: Can we name the error here, e.g. "...as its state model, or the crawler raises a ValueError"? The constructor raises it too, not only the factory methods. Then users who search the guide for the error find this section.
✍️ Drafted by Claude Code
| playwright_crawler_specific_kwargs: _PlaywrightCrawlerAdditionalOptions | None = None, | ||
| save_response_cookies: bool = True, | ||
| statistics: Statistics[StatisticsState] | None = None, | ||
| statistics: Statistics[AdaptivePlaywrightCrawlerStatisticState] | None = None, |
There was a problem hiding this comment.
Suggestion: Statistics is invariant, so a variable of type Statistics[MyState], where MyState subclasses AdaptivePlaywrightCrawlerStatisticState, fails type checking here (ty: invalid-argument-type). The issubclass check accepts it at runtime. A bounded TypeVar would allow it, the same as StatisticStateExtension in JS. Fine as a follow-up.
✍️ Drafted by Claude Code
| statistics: Statistics[AdaptivePlaywrightCrawlerStatisticState] | None = None, | ||
| **kwargs: Unpack[_BasicCrawlerOptions], | ||
| ) -> AdaptivePlaywrightCrawler[ParsedHttpCrawlingContext[Selector], Selector, Selector]: | ||
| """Create `AdaptivePlaywrightCrawler` that uses `Parcel` for parsing static content.""" |
There was a problem hiding this comment.
Nit: Parcel -> Parsel. The typo is older, but the line is next to the change.
| """Create `AdaptivePlaywrightCrawler` that uses `Parcel` for parsing static content.""" | |
| """Create `AdaptivePlaywrightCrawler` that uses `Parsel` for parsing static content.""" |
✍️ Drafted by Claude Code
| self.error_tracker_retry = ErrorTracker(save_error_snapshots=False) | ||
|
|
||
| self._requests_in_progress = dict[str, RequestProcessingRecord]() | ||
| self._state_model = state_model |
There was a problem hiding this comment.
Suggestion: RecoverableState._state_type already holds this class. A read-only state_model property would avoid the second field and the noqa: SLF001 in the adaptive crawler.
✍️ Drafted by Claude Code
| await crawler.run(['https://a.placeholder.com']) | ||
|
|
||
|
|
||
| def test_factory_passes_stats_as_is() -> None: |
There was a problem hiding this comment.
Nit: This test and test_rejects_plain_statistics cover only with_parsel_static_parser. Can we parametrize both over with_parsel_static_parser and with_beautifulsoup_static_parser?
✍️ Drafted by Claude Code
| from crawlee.crawlers import AdaptivePlaywrightCrawler | ||
| from crawlee.crawlers._adaptive_playwright._adaptive_playwright_crawler_statistics import ( | ||
| AdaptivePlaywrightCrawlerStatisticState, | ||
| ) |
There was a problem hiding this comment.
Nit: The state model is exported from crawlee.crawlers, so the private module import isn't needed.
| from crawlee.crawlers import AdaptivePlaywrightCrawler | |
| from crawlee.crawlers._adaptive_playwright._adaptive_playwright_crawler_statistics import ( | |
| AdaptivePlaywrightCrawlerStatisticState, | |
| ) | |
| from crawlee.crawlers import AdaptivePlaywrightCrawler, AdaptivePlaywrightCrawlerStatisticState |
✍️ Drafted by Claude Code
Description
AdaptivePlaywrightCrawler.with_beautifulsoup_static_parserandwith_parsel_static_parserpassstatisticsto the crawler as is, so the instance you pass is the one that counts, with all its settings.AdaptivePlaywrightCrawlerraises aValueErrorwhenstatisticsdon't useAdaptivePlaywrightCrawlerStatisticStateas their state model.Statistics.replace_state_modelis removed.For how to update your code, see the upgrade guide.
Issues
Testing
ValueErrorfor another state model.✍️ Drafted by Claude Code