You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PlaywrightCrawler: context.send_request in the handler uses stale cookies when a custom http_client is set #2483
While building a scraper, I load one page with PlaywrightCrawler so the browser can pass a bot check. Then I fetch other pages with context.send_request from the handler, using a custom http_client (curl_cffi).
Expected:send_request sends the cookies the browser received during navigation. Actual: It sends the cookies from before navigation. Nothing is logged or raised.
Cause (found by reading the code):
The crawler copies browser cookies into context.session.cookies only after the handler returns (_playwright_crawler.py L568-570, crawlee 1.11.3). A custom client reads the Session cookies. The default PlaywrightHttpClient reads the live browser cookies, so it isn't affected.
Workaround:
Inside the handler, read the cookies from the browser context and write them into context.session.cookies by hand before calling send_request.
Questions:
PR #1163 moved the copy to after the handler, which may have caused this. Could the crawler also copy the cookies after navigation, before the handler, and keep the copy after it? Or is a docs note enough?
I can open a PR for either.
Activity