Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/03_concepts/22_statistics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,15 @@ The raw data lives in a <ApiLink to="class/StatisticsState">`StatisticsState`</A
- `requests_retries` counts requests that needed at least one retry, and `request_retry_histogram` shows how many requests finished or failed after 0, 1, 2, and more retries.
- `request_total_finished_duration`, `request_total_failed_duration`, `request_min_duration`, and `request_max_duration` measure request processing time.
- `requests_with_status_code` maps HTTP status codes to the number of responses that returned them.
- `crawler_started_at`, `crawler_finished_at`, and the computed `crawler_runtime` describe the run itself.
- `crawler_started_at`, `crawler_finished_at`, and the computed `crawler_runtime` describe the crawl itself.

From the state, <ApiLink to="class/Statistics#calculate">`Statistics.calculate`</ApiLink> derives a <ApiLink to="class/FinalStatistics">`FinalStatistics`</ApiLink> object. It's a frozen summary with the request counts, the retry histogram as a list, average finished and failed durations, total duration, per-minute rates for finished and failed requests, and the crawler runtime. The same summary appears in the periodic and final statistics logs.

## Reading statistics

There are two ways to get the statistics of a crawl:

- <ApiLink to="class/BasicCrawler#run">`crawler.run()`</ApiLink> returns the `FinalStatistics` of the run once it ends.
- <ApiLink to="class/BasicCrawler#run">`crawler.run()`</ApiLink> returns the `FinalStatistics` once the run ends.
- <ApiLink to="class/BasicCrawler#statistics">`crawler.statistics`</ApiLink> gives you the `Statistics` instance, so you can read the full `state` or call `calculate()` at any time, including from a request handler while the crawl is running.

The following example reads the summary returned by `run()`, then looks at state fields that `FinalStatistics` doesn't include, and the errors collected during the run.
Expand All @@ -37,7 +37,7 @@ The following example reads the summary returned by `run()`, then looks at state
{ReadFinalStatisticsExample}
</RunnableCodeBlock>

A crawler that you run again with `run()` after a completed run resets its statistics at the start of the new run. To keep the numbers from a previous run, store the returned `FinalStatistics` before calling `run()` again.
A crawler that you run again with `run()` continues the crawl, so its statistics add up the runs. The runtime counts only the time the runs took, not the time between them. The `FinalStatistics` that a later `run()` returns include the earlier runs too. To count from zero again, call `crawler.statistics.reset()` before the next `run()`.

## Periodic logging

Expand All @@ -50,7 +50,7 @@ The crawler's constructor also accepts `statistics_log_format`. When the crawler

## Persistence

`Statistics` can save its state to a key-value store, so a crawl that gets migrated, aborted, or restarted continues counting from where it left off. When persistence is on, the state is loaded from the store when the crawl starts, saved on every `PERSIST_STATE` event, and saved once more when the crawl ends. For details about the event, see [Events](./events).
`Statistics` can save its state to a key-value store, so a crawl that gets migrated, aborted, or restarted continues counting from where it left off. When persistence is on, the state is loaded from the store when the crawl starts, saved on every `PERSIST_STATE` event, and saved once more when a run ends. For details about the event, see [Events](./events).

The following options control persistence:

Expand All @@ -74,6 +74,6 @@ A custom `Statistics` logs its periodic messages through the `crawlee.statistics

## Error tracking

Each `Statistics` instance holds an `error_tracker`, which groups errors from failed request attempts by their type, the place in the code where they were raised, and a similar message. Its `total` and `unique_error_count` properties count all errors and the distinct groups, and `get_most_common_errors()` returns the most frequent groups with their counts. At the end of a run that had errors, the crawler logs both counts. A second tracker, `error_tracker_retry`, collects session errors that caused a retry with a new session.
Each `Statistics` instance holds an `error_tracker`, which groups errors from failed request attempts by their type, the place in the code where they were raised, and a similar message. Its `total` and `unique_error_count` properties count all errors and the distinct groups, and `get_most_common_errors()` returns the most frequent groups with their counts. At the end of a run, the crawler logs both counts if there are any errors. A repeated `run()` keeps the errors of the earlier runs in the tracker, so the counts include them too. A second tracker, `error_tracker_retry`, collects session errors that caused a retry with a new session.

With `save_error_snapshots=True`, the error tracker also saves a snapshot of the page on the first occurrence of each unique error. For details and examples, see [Capturing page snapshots on errors](./error-handling#capturing-page-snapshots-on-errors).
2 changes: 1 addition & 1 deletion docs/04_guides/11_secure_scraping.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ If only your crawler's storage sees the response, the same request is a *blind*

A hostile site can try to use up your CPU, memory, or queue rather than read anything:

- A *crawler trap* is a maze of randomly generated URLs linking to more generated pages. It's built to keep a crawler running forever and grow its queue without bound. Bound how far link-following goes with <ApiLink to="class/BasicCrawlerOptions#max_crawl_depth">`max_crawl_depth`</ApiLink>, and cap a whole run with <ApiLink to="class/BasicCrawlerOptions#max_requests_per_crawl">`max_requests_per_crawl`</ApiLink>. Both are unlimited by default, so set them yourself.
- A *crawler trap* is a maze of randomly generated URLs linking to more generated pages. It's built to keep a crawler running forever and grow its queue without bound. Bound how far link-following goes with <ApiLink to="class/BasicCrawlerOptions#max_crawl_depth">`max_crawl_depth`</ApiLink>, and cap a whole crawl with <ApiLink to="class/BasicCrawlerOptions#max_requests_per_crawl">`max_requests_per_crawl`</ApiLink>. Both are unlimited by default, so set them yourself.
- A single *oversized or slow response* can tie up a worker or blow up memory. Both the fetch and the handler are capped at 60 seconds by default. <ApiLink to="class/HttpCrawlerOptions#navigation_timeout">`navigation_timeout`</ApiLink> covers the fetch, including reading the body. <ApiLink to="class/BasicCrawlerOptions#request_handler_timeout">`request_handler_timeout`</ApiLink> covers the handler. With <ApiLink to="class/FileDownloadCrawler">`FileDownloadCrawler`</ApiLink> in stream mode, reading the body counts towards the handler timeout instead. Raise them if a legitimate target needs longer, but keep the cap as tight as your workload allows.
- A *decompression bomb* is an archive that expands to gigabytes when unpacked. If your crawler downloads and extracts archives, cap the output stream during extraction. The declared size is part of the attack and easy to forge.

Expand Down
11 changes: 6 additions & 5 deletions docs/06_upgrading/upgrading_to_v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,11 +267,13 @@ crawler_1 = BasicCrawler(request_manager=queue)
crawler_2 = BasicCrawler(request_manager=queue)
```

## Repeated `run()` calls no longer purge the request queue
## Repeated `run()` calls continue the crawl

In v1, calling `crawler.run()` again on the same instance purged the request queue, unless the queue was named or the previous run had failed. In v2, nothing is purged between runs, the same as Crawlee JS. A repeated `run()` continues with the same request manager and skips the requests an earlier run handled, failed ones included.

The `purge_request_queue` argument of `run()` is removed. If you passed `purge_request_queue=False`, drop the argument. To crawl the same requests again, purge the request manager yourself:
The statistics and the session pool carry over too. A repeated `run()` adds to the statistics of the earlier runs, error trackers included, and keeps the sessions. In v1, it started both afresh. The `max_requests_per_crawl` limit counts the requests in the statistics, so it includes the earlier runs. A run that starts with the limit already reached stops right away. To count from zero, call `crawler.statistics.reset()` before the next `run()`. To start with fresh sessions, pass your own `SessionPool` and call its `reset()` between runs.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: "In v1, it started both afresh." -> "In v1, it reset both."

✍️ Drafted by Claude Code


The `purge_request_queue` argument of `run()` is removed. If you passed `purge_request_queue=False`, drop the argument. To crawl the same requests again, purge the request manager yourself. To also count from zero, as v1 did, reset the statistics:

```python
# v1
Expand All @@ -282,6 +284,7 @@ await crawler.run(urls)
await crawler.run(urls)
request_manager = await crawler.get_request_manager()
await request_manager.purge()
crawler.statistics.reset()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: v1's reset() also cleared the persisted record. To match "count from zero, as v1 did", reset the store too. Otherwise the stored record keeps the earlier totals until the next persist, and a resumed process can restore them.

Suggested change
crawler.statistics.reset()
crawler.statistics.reset()
await crawler.statistics.reset_store()

✍️ Drafted by Claude Code

await crawler.run(urls)
```

Expand Down Expand Up @@ -501,7 +504,7 @@ What you do with an object you pass depends on its type:

- `SessionPool`, `BrowserPool`, an HTTP client and the `rendering_type_predictor` of `AdaptivePlaywrightCrawler`: enter it with `async with` before you run the crawler, or `run()` raises a `RuntimeError`. Keep it open until every crawler using it has finished.
- The `extractor` of `PydanticAiCrawler`: enter it the same way. `PydanticAiDirectExtractor` is now an async context manager, like `PydanticAiSelectorExtractor`. A selector extractor enters and exits its `fallback` itself, so don't enter the fallback yourself, and give each selector extractor its own. If your subclass of `BasePydanticAiHtmlExtractor` overrides `__aenter__` or `__aexit__`, call `super()`. A custom extractor that implements the `PydanticAiHtmlExtractor` protocol directly needs the new `active` property, `__aenter__` and `__aexit__`.
- `Statistics`: don't enter it yourself, or `run()` raises a `RuntimeError`. In v1, the crawler left entered statistics alone. Crawlers that run one after another can share an instance, including another crawler's statistics. The instance adds up their runs, so the `max_requests_per_crawl` of the second crawler counts the requests of the first one too. When a crawler runs again, it resets the instance, including what the other crawlers counted. Two crawlers can't run with the same instance at the same time.
- `Statistics`: don't enter it yourself, or `run()` raises a `RuntimeError`. In v1, the crawler left entered statistics alone. Crawlers that run one after another can share an instance, including another crawler's statistics. The instance adds up their runs, so the `max_requests_per_crawl` of the second crawler counts the requests of the first one too. Two crawlers can't run with the same instance at the same time.

For example:

Expand All @@ -518,8 +521,6 @@ async with session_pool:
await crawler.run(urls)
```

When a crawler runs again, it resets only a session pool it created. A `SessionPool` you pass keeps its sessions. In v1, a repeated `run()` reset a passed pool too.

`RenderingTypePredictor` has a new `active` property, which `initialize()` and `clear()` of the base class keep up to date. If your predictor overrides them, call `super()`.

`AdaptivePlaywrightCrawler.rendering_type_predictor` is now read-only. Pass your predictor to the constructor instead.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

@docs_group('Statistics')
class AdaptivePlaywrightCrawlerStatisticState(StatisticsState):
"""Statistic data about a crawler run with additional information related to adaptive crawling."""
"""Statistic data about a crawl with additional information related to adaptive crawling."""

model_config = ConfigDict(
alias_generator=to_camel, validate_by_name=True, validate_by_alias=True, ser_json_inf_nan='constants'
Expand Down
23 changes: 7 additions & 16 deletions src/crawlee/crawlers/_basic/_basic_crawler.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,8 @@ class _BasicCrawlerOptions(TypedDict):
max_requests_per_crawl: NotRequired[int | None]
"""Maximum number of pages to open during a crawl. The crawl stops upon reaching this limit.
Setting this value can help avoid infinite loops in misconfigured crawlers. `None` means no limit.
Due to concurrency settings, the actual number of pages visited may slightly exceed this value."""
Due to concurrency settings, the actual number of pages visited may slightly exceed this value. The limit counts
the requests in the statistics, so it includes earlier runs and other crawlers that share the statistics."""

max_session_rotations: NotRequired[int]
"""Maximum number of session rotations per request. The crawler rotates the session if a proxy error occurs
Expand Down Expand Up @@ -348,7 +349,8 @@ def __init__(
this limit. Setting this value can help avoid infinite loops in misconfigured crawlers. `None` means
no limit. Due to concurrency settings, the actual number of pages visited may slightly exceed
this value. If used together with `keep_alive`, then the crawler will be kept alive only until
`max_requests_per_crawl` is achieved.
`max_requests_per_crawl` is achieved. The limit counts the requests in the statistics, so it includes
earlier runs and other crawlers that share the statistics.
max_session_rotations: Maximum number of session rotations per request. The crawler rotates the session
if a proxy error occurs or if the website blocks the request.
The session rotations are not counted towards the `max_request_retries` limit.
Expand Down Expand Up @@ -534,7 +536,6 @@ async def persist_state_factory() -> KeyValueStore:

# State flags
self._keep_alive = keep_alive
self._has_finished_before = False
self._stop_before_run = False
self._logger_once = LoggerOnce(self._logger)

Expand Down Expand Up @@ -598,7 +599,7 @@ def concurrency_system(self) -> ConcurrencySystemInterface | None:

@property
def statistics(self) -> Statistics[TStatisticsState]:
"""Statistics about the current (or last) crawler run."""
"""Statistics of the crawler's runs, which add up from one run to the next."""
return self._statistics

@property
Expand Down Expand Up @@ -837,7 +838,8 @@ async def run(self, requests: Sequence[str | Request] | None = None) -> FinalSta

Calling `run()` again on the same instance keeps crawling the same request manager. Requests that a previous
run handled, failed ones included, aren't processed again. To crawl them again, purge the request manager, or
create a new crawler with a freshly opened request queue.
create a new crawler with a freshly opened request queue. The statistics carry over too, so
`max_requests_per_crawl` includes the requests of the earlier runs.

Args:
requests: The requests to be enqueued before the crawler starts.
Expand Down Expand Up @@ -869,15 +871,6 @@ async def run(self, requests: Sequence[str | Request] | None = None) -> FinalSta
self._last_run = self._create_run()

try:
if self._has_finished_before:
self._statistics.reset()
await self._statistics.reset_store()

# A session pool passed in belongs to the caller, who may still use it in another crawler.
if self._session_pool in self._owned_collaborators:
self._session_pool.reset()
await self._session_pool.reset_store()

if requests is not None:
await self.add_requests(requests)

Expand Down Expand Up @@ -937,8 +930,6 @@ def sigint_handler() -> None:
'The crawl was interrupted. To resume, do: CRAWLEE_PURGE_ON_START=0 python %s', sys.argv[0]
)

self._has_finished_before = True

await self._save_crawler_state()

final_statistics = self._statistics.calculate()
Expand Down
4 changes: 2 additions & 2 deletions src/crawlee/statistics/_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
@dataclass(frozen=True)
@docs_group('Statistics')
class FinalStatistics:
"""Statistics about a crawler run."""
"""Statistics about a crawl."""

requests_finished: int
requests_failed: int
Expand Down Expand Up @@ -56,7 +56,7 @@ def __str__(self) -> str:

@docs_group('Statistics')
class StatisticsState(BaseModel):
"""Statistic data about a crawler run."""
"""Statistic data about a crawl."""

model_config = ConfigDict(
alias_generator=to_camel, validate_by_name=True, validate_by_alias=True, ser_json_inf_nan='constants'
Expand Down
3 changes: 3 additions & 0 deletions src/crawlee/statistics/_statistics.py
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,9 @@ async def __aenter__(self) -> Self:

self._active = True

# An attempt that an earlier run left in progress never finishes, so it must not count as a retry in this run.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: A request whose attempt failed and that was reclaimed for a retry does get processed in the next run, so "never finishes" isn't exact. Clearing its record makes that retry count as 0, the same as after a restore. Maybe: "Records of requests that an earlier run left unfinished, including ones waiting for a retry, are dropped, the same as after a restore."

✍️ Drafted by Claude Code

self._requests_in_progress.clear()

try:
await self._state.initialize()

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -736,6 +736,19 @@ async def request_handler(context: AdaptivePlaywrightCrawlingContext) -> None:
mocked_browser_handler.assert_called_once_with()


async def test_repeated_run_keeps_own_stats(test_urls: list[str]) -> None:
"""A repeated `run()` adds to the default statistics of the crawler, since it continues the crawl."""
crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
rendering_type_predictor=_SimpleRenderingTypePredictor(detection_probability_recommendation=cycle([0])),
)
crawler.router.default_handler(AsyncMock())

await crawler.run(test_urls[:1])
stats = await crawler.run(test_urls[1:])

assert stats.requests_finished == len(test_urls)


async def test_adaptive_context_query_selector_beautiful_soup(test_urls: list[str]) -> None:
"""Test that `context.query_selector_one` works regardless of the crawl type for BeautifulSoup variant.

Expand Down
14 changes: 14 additions & 0 deletions tests/unit/_statistics/test_enter_exit.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,20 @@ async def wait_for_release() -> None:
await exit_task


async def test_enter_drops_unfinished_requests() -> None:
"""A request left in progress by an earlier run doesn't count as retried in the next one."""
statistics = Statistics.with_default_state()

async with statistics:
statistics.record_request_processing_start('request')

async with statistics:
statistics.record_request_processing_start('request')
statistics.record_request_processing_finish('request')

assert statistics.state.request_retry_histogram == {0: 1}


async def test_failed_start_tears_down_stats(monkeypatch: pytest.MonkeyPatch) -> None:
"""Statistics whose entry fails after loading the state tear the state down and can be entered again."""
statistics = Statistics.with_default_state()
Expand Down
Loading
Loading