Skip to content

PlaywrightCrawler: context.send_request in the handler uses stale cookies when a custom http_client is set #2483

Open

Description

@mido-99

While building a scraper, I load one page with PlaywrightCrawler so the browser can pass a bot check. Then I fetch other pages with context.send_request from the handler, using a custom http_client (curl_cffi).

Expected: send_request sends the cookies the browser received during navigation.
Actual: It sends the cookies from before navigation. Nothing is logged or raised.

Cause (found by reading the code):
The crawler copies browser cookies into context.session.cookies only after the handler returns (_playwright_crawler.py L568-570, crawlee 1.11.3). A custom client reads the Session cookies. The default PlaywrightHttpClient reads the live browser cookies, so it isn't affected.

Workaround:
Inside the handler, read the cookies from the browser context and write them into context.session.cookies by hand before calling send_request.

Questions:
PR #1163 moved the copy to after the handler, which may have caused this. Could the crawler also copy the cookies after navigation, before the handler, and keep the copy after it? Or is a docs note enough?
I can open a PR for either.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions