Repository navigation
fix(crawlers)!: Keep statistics and sessions between run() calls
#2449
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: v2
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
|
|
@@ -267,11 +267,13 @@ crawler_1 = BasicCrawler(request_manager=queue) | |||||||
| crawler_2 = BasicCrawler(request_manager=queue) | ||||||||
| ``` | ||||||||
|
|
||||||||
| ## Repeated `run()` calls no longer purge the request queue | ||||||||
| ## Repeated `run()` calls continue the crawl | ||||||||
|
|
||||||||
| In v1, calling `crawler.run()` again on the same instance purged the request queue, unless the queue was named or the previous run had failed. In v2, nothing is purged between runs, the same as Crawlee JS. A repeated `run()` continues with the same request manager and skips the requests an earlier run handled, failed ones included. | ||||||||
|
|
||||||||
| The `purge_request_queue` argument of `run()` is removed. If you passed `purge_request_queue=False`, drop the argument. To crawl the same requests again, purge the request manager yourself: | ||||||||
| The statistics and the session pool carry over too. A repeated `run()` adds to the statistics of the earlier runs, error trackers included, and keeps the sessions. In v1, it started both afresh. The `max_requests_per_crawl` limit counts the requests in the statistics, so it includes the earlier runs. A run that starts with the limit already reached stops right away. To count from zero, call `crawler.statistics.reset()` before the next `run()`. To start with fresh sessions, pass your own `SessionPool` and call its `reset()` between runs. | ||||||||
|
|
||||||||
| The `purge_request_queue` argument of `run()` is removed. If you passed `purge_request_queue=False`, drop the argument. To crawl the same requests again, purge the request manager yourself. To also count from zero, as v1 did, reset the statistics: | ||||||||
|
|
||||||||
| ```python | ||||||||
| # v1 | ||||||||
|
|
@@ -282,6 +284,7 @@ await crawler.run(urls) | |||||||
| await crawler.run(urls) | ||||||||
| request_manager = await crawler.get_request_manager() | ||||||||
| await request_manager.purge() | ||||||||
| crawler.statistics.reset() | ||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Suggestion: v1's
Suggested change
✍️ Drafted by Claude Code |
||||||||
| await crawler.run(urls) | ||||||||
| ``` | ||||||||
|
|
||||||||
|
|
@@ -501,7 +504,7 @@ What you do with an object you pass depends on its type: | |||||||
|
|
||||||||
| - `SessionPool`, `BrowserPool`, an HTTP client and the `rendering_type_predictor` of `AdaptivePlaywrightCrawler`: enter it with `async with` before you run the crawler, or `run()` raises a `RuntimeError`. Keep it open until every crawler using it has finished. | ||||||||
| - The `extractor` of `PydanticAiCrawler`: enter it the same way. `PydanticAiDirectExtractor` is now an async context manager, like `PydanticAiSelectorExtractor`. A selector extractor enters and exits its `fallback` itself, so don't enter the fallback yourself, and give each selector extractor its own. If your subclass of `BasePydanticAiHtmlExtractor` overrides `__aenter__` or `__aexit__`, call `super()`. A custom extractor that implements the `PydanticAiHtmlExtractor` protocol directly needs the new `active` property, `__aenter__` and `__aexit__`. | ||||||||
| - `Statistics`: don't enter it yourself, or `run()` raises a `RuntimeError`. In v1, the crawler left entered statistics alone. Crawlers that run one after another can share an instance, including another crawler's statistics. The instance adds up their runs, so the `max_requests_per_crawl` of the second crawler counts the requests of the first one too. When a crawler runs again, it resets the instance, including what the other crawlers counted. Two crawlers can't run with the same instance at the same time. | ||||||||
| - `Statistics`: don't enter it yourself, or `run()` raises a `RuntimeError`. In v1, the crawler left entered statistics alone. Crawlers that run one after another can share an instance, including another crawler's statistics. The instance adds up their runs, so the `max_requests_per_crawl` of the second crawler counts the requests of the first one too. Two crawlers can't run with the same instance at the same time. | ||||||||
|
|
||||||||
| For example: | ||||||||
|
|
||||||||
|
|
@@ -518,8 +521,6 @@ async with session_pool: | |||||||
| await crawler.run(urls) | ||||||||
| ``` | ||||||||
|
|
||||||||
| When a crawler runs again, it resets only a session pool it created. A `SessionPool` you pass keeps its sessions. In v1, a repeated `run()` reset a passed pool too. | ||||||||
|
|
||||||||
| `RenderingTypePredictor` has a new `active` property, which `initialize()` and `clear()` of the base class keep up to date. If your predictor overrides them, call `super()`. | ||||||||
|
|
||||||||
| `AdaptivePlaywrightCrawler.rendering_type_predictor` is now read-only. Pass your predictor to the constructor instead. | ||||||||
|
|
||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -174,6 +174,9 @@ async def __aenter__(self) -> Self: | |
|
|
||
| self._active = True | ||
|
|
||
| # An attempt that an earlier run left in progress never finishes, so it must not count as a retry in this run. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: A request whose attempt failed and that was reclaimed for a retry does get processed in the next run, so "never finishes" isn't exact. Clearing its record makes that retry count as 0, the same as after a restore. Maybe: "Records of requests that an earlier run left unfinished, including ones waiting for a retry, are dropped, the same as after a restore." ✍️ Drafted by Claude Code |
||
| self._requests_in_progress.clear() | ||
|
|
||
| try: | ||
| await self._state.initialize() | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: "In v1, it started both afresh." -> "In v1, it reset both."
✍️ Drafted by Claude Code