Repository navigation
fix(crawlers): Use the HTTP client and session pool of AdaptivePlaywrightCrawler in its sub crawlers - #2451
fix(crawlers): Use the HTTP client and session pool of AdaptivePlaywrightCrawler in its sub crawlers#2451Mantisus wants to merge 1 commit into
AdaptivePlaywrightCrawler in its sub crawlers#2451Conversation
ed84b3c to
176b3f8
Compare
…n its sub crawlers
176b3f8 to
5880cff
Compare
vdusek
left a comment
There was a problem hiding this comment.
Note: Outside this PR, but it fits #2301: a standalone PlaywrightCrawler(respect_robots_txt_file=True) still creates a new ImpitHttpClient() for each robots.txt fetch and never closes it (_playwright_crawler.py:639). Could be a follow-up.
✍️ Drafted by Claude Code
|
|
||
| sending_clients = {send_call.args[0] for send_call in send_request.await_args_list} | ||
| closed_clients = {cleanup_call.args[0] for cleanup_call in cleanup.await_args_list} | ||
| assert sending_clients == {crawler._http_client} |
There was a problem hiding this comment.
Suggestion: In the browser case, only the sub crawler's robots.txt fetch proves anything, and the test doesn't check that the request was handled. If the browser run fails before enqueue_links(), this assertion still holds. Could you assert requests_finished == 1 on the result of crawler.run()? Also, once #2452 reaches v2, the sub crawlers share the robots.txt cache, so this case passes without the fix and the robots.txt comment above gets stale.
✍️ Drafted by Claude Code
| pytest.param('client only', id='browser'), | ||
| ], | ||
| ) | ||
| async def test_closes_shared_default_client(test_urls: list[str], rendering_type: RenderingType) -> None: |
There was a problem hiding this comment.
Nit: closed_clients == {crawler._http_client} also passes on v2, because the sub crawlers' clients were never closed. Only sending_clients catches the bug. A name like test_sub_crawlers_send_through_crawler_http_client would match what the test proves.
✍️ Drafted by Claude Code
Description
AdaptivePlaywrightCrawleruse the HTTP client and the session pool of the adaptive crawler, instead of building defaults of their own that nothing entered or closed.Issues
Testing
✍️ Drafted by Claude Code