Skip to content

fix(crawlers)!: Count requests in the statistics passed to AdaptivePlaywrightCrawler factory methods - #2447

Open
Mantisus wants to merge 1 commit into
apify:v2from
Mantisus:adaptive-factory-statistics
Open

Mantisus wants to merge 1 commit into
apify:v2from
Mantisus:adaptive-factory-statistics

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser and with_parsel_static_parser pass statistics to the crawler as is, so the instance you pass is the one that counts, with all its settings.
  • AdaptivePlaywrightCrawler raises a ValueError when statistics don't use AdaptivePlaywrightCrawlerStatisticState as their state model.
  • Statistics.replace_state_model is removed.

For how to update your code, see the upgrade guide.

Issues

Testing

  • Unit and integration tests for statistics passed to the factory methods, including the ValueError for another state model.

✍️ Drafted by Claude Code

@Mantisus
Mantisus marked this pull request as draft November 14, 2026 22:37
@Mantisus Mantisus self-assigned this Nov 14, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 14, 2026
@Mantisus
Mantisus force-pushed the adaptive-factory-statistics branch from bd176e9 to fa2fa4b Compare November 15, 2026 22:30
@Mantisus
Mantisus force-pushed the adaptive-factory-statistics branch from fa2fa4b to a6a9c44 Compare November 18, 2026 14:27
@Mantisus
Mantisus marked this pull request as ready for review November 18, 2026 14:28
@Mantisus
Mantisus requested a review from vdusek November 18, 2026 14:28
@vdusek vdusek changed the title fix(statistics)!: Count requests in statistics passed to adaptive crawler factories and require the adaptive state model fix(crawlers)!: Count requests in the statistics passed to AdaptivePlaywrightCrawler factory methods Nov 18, 2026

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: docs/03_concepts/22_statistics.mdx:65 (outside the diff) still says to use Statistics.with_default_state to get an instance. With this PR, such an instance makes AdaptivePlaywrightCrawler raise a ValueError. Can we add a sentence that the adaptive crawler needs Statistics(state_model=AdaptivePlaywrightCrawlerStatisticState)?

✍️ Drafted by Claude Code


When a crawler runs again, it resets only a session pool it created. A `SessionPool` you pass keeps its sessions. In v1, a repeated `run()` reset a passed pool too.

The `with_beautifulsoup_static_parser` and `with_parsel_static_parser` factory methods of `AdaptivePlaywrightCrawler` pass `statistics` to the crawler as they are, instead of a copy built with `Statistics.replace_state_model`, so the instance you pass is the one that counts. The instance must use `AdaptivePlaywrightCrawlerStatisticState` as its state model:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Can we name the error here, e.g. "...as its state model, or the crawler raises a ValueError"? The constructor raises it too, not only the factory methods. Then users who search the guide for the error find this section.

✍️ Drafted by Claude Code

playwright_crawler_specific_kwargs: _PlaywrightCrawlerAdditionalOptions | None = None,
save_response_cookies: bool = True,
statistics: Statistics[StatisticsState] | None = None,
statistics: Statistics[AdaptivePlaywrightCrawlerStatisticState] | None = None,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Statistics is invariant, so a variable of type Statistics[MyState], where MyState subclasses AdaptivePlaywrightCrawlerStatisticState, fails type checking here (ty: invalid-argument-type). The issubclass check accepts it at runtime. A bounded TypeVar would allow it, the same as StatisticStateExtension in JS. Fine as a follow-up.

✍️ Drafted by Claude Code

statistics: Statistics[AdaptivePlaywrightCrawlerStatisticState] | None = None,
**kwargs: Unpack[_BasicCrawlerOptions],
) -> AdaptivePlaywrightCrawler[ParsedHttpCrawlingContext[Selector], Selector, Selector]:
"""Create `AdaptivePlaywrightCrawler` that uses `Parcel` for parsing static content."""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Parcel -> Parsel. The typo is older, but the line is next to the change.

Suggested change
"""Create `AdaptivePlaywrightCrawler` that uses `Parcel` for parsing static content."""
"""Create `AdaptivePlaywrightCrawler` that uses `Parsel` for parsing static content."""

✍️ Drafted by Claude Code

self.error_tracker_retry = ErrorTracker(save_error_snapshots=False)

self._requests_in_progress = dict[str, RequestProcessingRecord]()
self._state_model = state_model

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: RecoverableState._state_type already holds this class. A read-only state_model property would avoid the second field and the noqa: SLF001 in the adaptive crawler.

✍️ Drafted by Claude Code

await crawler.run(['https://a.placeholder.com'])


def test_factory_passes_stats_as_is() -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This test and test_rejects_plain_statistics cover only with_parsel_static_parser. Can we parametrize both over with_parsel_static_parser and with_beautifulsoup_static_parser?

✍️ Drafted by Claude Code

Comment on lines 13 to +16
from crawlee.crawlers import AdaptivePlaywrightCrawler
from crawlee.crawlers._adaptive_playwright._adaptive_playwright_crawler_statistics import (
AdaptivePlaywrightCrawlerStatisticState,
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The state model is exported from crawlee.crawlers, so the private module import isn't needed.

Suggested change
from crawlee.crawlers import AdaptivePlaywrightCrawler
from crawlee.crawlers._adaptive_playwright._adaptive_playwright_crawler_statistics import (
AdaptivePlaywrightCrawlerStatisticState,
)
from crawlee.crawlers import AdaptivePlaywrightCrawler, AdaptivePlaywrightCrawlerStatisticState

✍️ Drafted by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let crawlers manage only the collaborators they create and extract per-run state

2 participants