Skip to content

fix(crawlers)!: Keep statistics and sessions between run() calls - #2449

Open
Mantisus wants to merge 1 commit into
apify:v2from
Mantisus:keep-state-across-runs
Open

Mantisus wants to merge 1 commit into
apify:v2from
Mantisus:keep-state-across-runs

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • A repeated run() continues the crawl: it adds to the statistics of the earlier runs, error trackers included, and keeps the sessions of the session pool. max_requests_per_crawl counts the requests of the earlier runs too, so a run that starts with the limit already reached stops right away. To count from zero, call crawler.statistics.reset() before the next run().
  • The same change for Crawlee JS is proposed in v4: A repeated run() continues the queue but resets the statistics the crawler built crawlee#4397.

Issues

Testing

  • Unit and integration tests for repeated runs with own and passed statistics and session pools.

✍️ Drafted by Claude Code

@Mantisus Mantisus self-assigned this Nov 15, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 15, 2026
@Mantisus
Mantisus force-pushed the keep-state-across-runs branch from e3dc508 to 41411cc Compare November 15, 2026 22:39
@Mantisus
Mantisus force-pushed the keep-state-across-runs branch from 41411cc to 8dbf734 Compare November 18, 2026 14:50
@Mantisus
Mantisus marked this pull request as ready for review November 18, 2026 14:50
@Mantisus
Mantisus requested a review from vdusek November 18, 2026 14:50
@Mantisus
Mantisus force-pushed the keep-state-across-runs branch from 8dbf734 to 0316453 Compare November 18, 2026 15:10

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Statistics.reset() (_statistics.py:313, outside the diff) replaces both error trackers with a bare ErrorTracker(), which drops save_error_snapshots and snapshot_kvs_name. With Statistics.with_default_state(save_error_snapshots=True), error_tracker.error_snapshotter is None after reset(). The bug is older than this PR, but the docs now tell users to call crawler.statistics.reset() between runs, so following them turns off error snapshots. Could reset() rebuild the trackers with the original options, like errorTracker.reset() in JS? A follow-up PR is fine.

Question: For the statistics and session pool the crawler builds, this goes beyond current JS v4, which still resets them on a repeated run(). It matches what apify/crawlee#4397 proposes. Is the JS side agreed, so the two don't diverge in the releases?

✍️ Drafted by Claude Code


self._active = True

# An attempt that an earlier run left in progress never finishes, so it must not count as a retry in this run.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: A request whose attempt failed and that was reclaimed for a retry does get processed in the next run, so "never finishes" isn't exact. Clearing its record makes that retry count as 0, the same as after a restore. Maybe: "Records of requests that an earlier run left unfinished, including ones waiting for a retry, are dropped, the same as after a restore."

✍️ Drafted by Claude Code

await crawler.run(urls)
request_manager = await crawler.get_request_manager()
await request_manager.purge()
crawler.statistics.reset()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: v1's reset() also cleared the persisted record. To match "count from zero, as v1 did", reset the store too. Otherwise the stored record keeps the earlier totals until the next persist, and a resumed process can restore them.

Suggested change
crawler.statistics.reset()
crawler.statistics.reset()
await crawler.statistics.reset_store()

✍️ Drafted by Claude Code

In v1, calling `crawler.run()` again on the same instance purged the request queue, unless the queue was named or the previous run had failed. In v2, nothing is purged between runs, the same as Crawlee JS. A repeated `run()` continues with the same request manager and skips the requests an earlier run handled, failed ones included.

The `purge_request_queue` argument of `run()` is removed. If you passed `purge_request_queue=False`, drop the argument. To crawl the same requests again, purge the request manager yourself:
The statistics and the session pool carry over too. A repeated `run()` adds to the statistics of the earlier runs, error trackers included, and keeps the sessions. In v1, it started both afresh. The `max_requests_per_crawl` limit counts the requests in the statistics, so it includes the earlier runs. A run that starts with the limit already reached stops right away. To count from zero, call `crawler.statistics.reset()` before the next `run()`. To start with fresh sessions, pass your own `SessionPool` and call its `reset()` between runs.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: "In v1, it started both afresh." -> "In v1, it reset both."

✍️ Drafted by Claude Code


async def test_repeated_run_starts_with_fresh_statistics() -> None:
"""A repeated `run()` starts from fresh statistics instead of adding to the previous run's."""
async def test_repeated_run_keeps_own_stats() -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This test and test_repeated_run_keeps_stats differ only in statistics=. One test with @pytest.mark.parametrize and pytest.param(None, id='own') / pytest.param(Statistics.with_default_state(), id='passed') would cover both.

✍️ Drafted by Claude Code

@Mantisus

Copy link
Copy Markdown
Collaborator Author

Suggestion: Statistics.reset() (_statistics.py:313, outside the diff) replaces both error trackers with a bare ErrorTracker(), which drops save_error_snapshots and snapshot_kvs_name. With Statistics.with_default_state(save_error_snapshots=True), error_tracker.error_snapshotter is None after reset(). The bug is older than this PR, but the docs now tell users to call crawler.statistics.reset() between runs, so following them turns off error snapshots. Could reset() rebuild the trackers with the original options, like errorTracker.reset() in JS? A follow-up PR is fine.

It was fixed in master in #2450 🙂

Question: For the statistics and session pool the crawler builds, this goes beyond current JS v4, which still resets them on a repeated run(). It matches what apify/crawlee#4397 proposes. Is the JS side agreed, so the two don't diverge in the releases?

This task is included in scope v4. However, I don't yet know what the final agreed-upon behavior will be. Therefore, after the review, I suggest we wait before merging it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let crawlers manage only the collaborators they create and extract per-run state

2 participants