Repository navigation
Conversation
e3dc508 to
41411cc
Compare
41411cc to
8dbf734
Compare
8dbf734 to
0316453
Compare
0316453 to
dfe7541
Compare
vdusek
left a comment
There was a problem hiding this comment.
Suggestion: Statistics.reset() (_statistics.py:313, outside the diff) replaces both error trackers with a bare ErrorTracker(), which drops save_error_snapshots and snapshot_kvs_name. With Statistics.with_default_state(save_error_snapshots=True), error_tracker.error_snapshotter is None after reset(). The bug is older than this PR, but the docs now tell users to call crawler.statistics.reset() between runs, so following them turns off error snapshots. Could reset() rebuild the trackers with the original options, like errorTracker.reset() in JS? A follow-up PR is fine.
Question: For the statistics and session pool the crawler builds, this goes beyond current JS v4, which still resets them on a repeated run(). It matches what apify/crawlee#4397 proposes. Is the JS side agreed, so the two don't diverge in the releases?
✍️ Drafted by Claude Code
|
|
||
| self._active = True | ||
|
|
||
| # An attempt that an earlier run left in progress never finishes, so it must not count as a retry in this run. |
There was a problem hiding this comment.
Nit: A request whose attempt failed and that was reclaimed for a retry does get processed in the next run, so "never finishes" isn't exact. Clearing its record makes that retry count as 0, the same as after a restore. Maybe: "Records of requests that an earlier run left unfinished, including ones waiting for a retry, are dropped, the same as after a restore."
✍️ Drafted by Claude Code
| await crawler.run(urls) | ||
| request_manager = await crawler.get_request_manager() | ||
| await request_manager.purge() | ||
| crawler.statistics.reset() |
There was a problem hiding this comment.
Suggestion: v1's reset() also cleared the persisted record. To match "count from zero, as v1 did", reset the store too. Otherwise the stored record keeps the earlier totals until the next persist, and a resumed process can restore them.
| crawler.statistics.reset() | |
| crawler.statistics.reset() | |
| await crawler.statistics.reset_store() |
✍️ Drafted by Claude Code
| In v1, calling `crawler.run()` again on the same instance purged the request queue, unless the queue was named or the previous run had failed. In v2, nothing is purged between runs, the same as Crawlee JS. A repeated `run()` continues with the same request manager and skips the requests an earlier run handled, failed ones included. | ||
|
|
||
| The `purge_request_queue` argument of `run()` is removed. If you passed `purge_request_queue=False`, drop the argument. To crawl the same requests again, purge the request manager yourself: | ||
| The statistics and the session pool carry over too. A repeated `run()` adds to the statistics of the earlier runs, error trackers included, and keeps the sessions. In v1, it started both afresh. The `max_requests_per_crawl` limit counts the requests in the statistics, so it includes the earlier runs. A run that starts with the limit already reached stops right away. To count from zero, call `crawler.statistics.reset()` before the next `run()`. To start with fresh sessions, pass your own `SessionPool` and call its `reset()` between runs. |
There was a problem hiding this comment.
Nit: "In v1, it started both afresh." -> "In v1, it reset both."
✍️ Drafted by Claude Code
|
|
||
| async def test_repeated_run_starts_with_fresh_statistics() -> None: | ||
| """A repeated `run()` starts from fresh statistics instead of adding to the previous run's.""" | ||
| async def test_repeated_run_keeps_own_stats() -> None: |
There was a problem hiding this comment.
Nit: This test and test_repeated_run_keeps_stats differ only in statistics=. One test with @pytest.mark.parametrize and pytest.param(None, id='own') / pytest.param(Statistics.with_default_state(), id='passed') would cover both.
✍️ Drafted by Claude Code
It was fixed in
This task is included in scope v4. However, I don't yet know what the final agreed-upon behavior will be. Therefore, after the review, I suggest we wait before merging it. |
Description
run()continues the crawl: it adds to the statistics of the earlier runs, error trackers included, and keeps the sessions of the session pool.max_requests_per_crawlcounts the requests of the earlier runs too, so a run that starts with the limit already reached stops right away. To count from zero, callcrawler.statistics.reset()before the nextrun().run()continues the queue but resets the statistics the crawler built crawlee#4397.Issues
Testing
✍️ Drafted by Claude Code