Skip to content

fix(crawlers): Use the HTTP client and session pool of AdaptivePlaywrightCrawler in its sub crawlers - #2451

Open
Mantisus wants to merge 1 commit into
apify:v2from
Mantisus:adaptive-shared-collaborators
Open

Mantisus wants to merge 1 commit into
apify:v2from
Mantisus:adaptive-shared-collaborators

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • Both sub crawlers of AdaptivePlaywrightCrawler use the HTTP client and the session pool of the adaptive crawler, instead of building defaults of their own that nothing entered or closed.
  • Static requests and robots.txt lookups go through the client that the crawler enters and closes on every run.

Issues

Testing

  • Integration test, for static and browser requests, that the sub crawlers send through the HTTP client the crawler closes.

✍️ Drafted by Claude Code

@Mantisus
Mantisus marked this pull request as draft November 15, 2026 17:15
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 15, 2026
@Mantisus Mantisus self-assigned this Nov 15, 2026
@Mantisus
Mantisus force-pushed the adaptive-shared-collaborators branch from ed84b3c to 176b3f8 Compare November 15, 2026 22:40
@Mantisus
Mantisus force-pushed the adaptive-shared-collaborators branch from 176b3f8 to 5880cff Compare November 18, 2026 14:53
@Mantisus
Mantisus marked this pull request as ready for review November 18, 2026 14:53
@Mantisus
Mantisus requested a review from vdusek November 18, 2026 14:53

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Outside this PR, but it fits #2301: a standalone PlaywrightCrawler(respect_robots_txt_file=True) still creates a new ImpitHttpClient() for each robots.txt fetch and never closes it (_playwright_crawler.py:639). Could be a follow-up.

✍️ Drafted by Claude Code


sending_clients = {send_call.args[0] for send_call in send_request.await_args_list}
closed_clients = {cleanup_call.args[0] for cleanup_call in cleanup.await_args_list}
assert sending_clients == {crawler._http_client}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: In the browser case, only the sub crawler's robots.txt fetch proves anything, and the test doesn't check that the request was handled. If the browser run fails before enqueue_links(), this assertion still holds. Could you assert requests_finished == 1 on the result of crawler.run()? Also, once #2452 reaches v2, the sub crawlers share the robots.txt cache, so this case passes without the fix and the robots.txt comment above gets stale.

✍️ Drafted by Claude Code

pytest.param('client only', id='browser'),
],
)
async def test_closes_shared_default_client(test_urls: list[str], rendering_type: RenderingType) -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: closed_clients == {crawler._http_client} also passes on v2, because the sub crawlers' clients were never closed. Only sending_clients catches the bug. A name like test_sub_crawlers_send_through_crawler_http_client would match what the test proves.

✍️ Drafted by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let crawlers manage only the collaborators they create and extract per-run state

2 participants