Handling cookie CAPTCHAs in Scrapy with DeathByCaptcha

Handling cookie CAPTCHAs in Scrapy with DeathByCaptcha

Posted on 2026-11-20 | 3 min read | Category: scrapy | By DeathByCaptcha Engineering Team

Scrapy


Scrapy is built for volume: many requests, many domains, retries and pipelines. A challenge in the middle of a crawl breaks that rhythm. The official Python client ships an example downloader middleware that detects a cookie-based challenge, solves it with DeathByCaptcha and retries the request with the token as a cookie.

The example lives in the Python client:

  • examples/scrapy/middleware.py
  • examples/scrapy/spider.py
  • examples/scrapy/README.md

It is an example to copy, not an installed package.

What it handles

  • Works without a browser: Amazon WAF (type 16) and DataDome (type 21) when the service returns a cookie value. The example implements WAF and documents the DataDome shape.
  • Does not work without a browser: reCAPTCHA and GeeTest. Scrapy does not execute JavaScript, so the middleware raises EscalationRequired and points to the Playwright samples. Pretending otherwise would only move the failure somewhere less obvious.

Setup

From the repository root:

pip install -e .
pip install -r examples/scrapy/requirements.txt

Then enable the middleware in your Scrapy project:

DOWNLOADER_MIDDLEWARES = {
    "middleware.DeathByCaptchaMiddleware": 543,
}

DBC_USERNAME = "your_username"
DBC_PASSWORD = "your_password"
# or: DBC_AUTHTOKEN = "your_authtoken"

# Required for challenges tied to the exit IP.
DBC_PROXY = "http://user:[email protected]:1234"
DBC_PROXYTYPE = "HTTP"

DBC_MAX_RETRIES = 2

Copy middleware.py where Python can import it, or add the example folder to PYTHONPATH.

The core of the middleware

process_response sees the challenge, extracts the WAF parameters from the page, asks DeathByCaptcha for the token and retries:

def process_response(self, request, response, spider):
    retries = request.meta.get("dbc_retries", 0)
    if retries >= self.max_retries:
        return response

    if self._is_recaptcha_or_geetest(response):
        raise EscalationRequired(
            "This challenge needs JavaScript: use the Playwright samples."
        )

    if not self._is_waf_challenge(response):
        return response

    params = self._waf_params(response)
    payload = {
        "sitekey": params["sitekey"],
        "pageurl": response.url,
        "iv": params["iv"],
        "context": params["context"],
    }

    captcha = self.client.decode(type=16, waf_params=json.dumps(payload))
    if not captcha or not captcha.get("text"):
        return response

    retry = request.replace(
        dont_filter=True,
        meta={**request.meta, "dbc_retries": retries + 1},
    )
    retry.cookies["aws-waf-token"] = captcha["text"]
    return retry

The detection is deliberately conservative: a 403, 429 or 503 with Amazon WAF markers, or a body that mentions aws-waf-token. The parameters are read from the inline scripts (key, iv, context), the same values the browser uses when it loads the challenge.

Why a proxy is not optional

Amazon WAF, DataDome and Turnstile tokens are tied to the exit IP. The service solves the challenge from its own IP unless you send a proxy, and a token solved from one IP is rejected by a request that leaves from another. The middleware passes DBC_PROXY through to the service, and the crawler must use the same proxy. The example documents this because a missing proxy looks exactly like a solver bug.

Retries and failure

  • DBC_MAX_RETRIES caps the attempts. A challenge that keeps coming back is dropped with a log line; the middleware never loops forever.
  • The token is never written to the crawl log; verbose mode stays off.
  • If the page exposes reCAPTCHA or GeeTest, the middleware raises instead of retrying with a token that Scrapy cannot inject.

Things to know

  • Scrapy is not a browser. Any challenge that needs JavaScript needs the Playwright path. Use the middleware for cookie challenges and escalate for the rest.
  • Point the spider at a page you own or are allowed to crawl. The sample spider uses a placeholder URL for that reason.
  • The middleware is an example, not a package. Copy it, adapt the selectors and the parameter extraction to your target, and keep the retry cap.

Next steps

For reCAPTCHA and Turnstile on pages that run JavaScript, the Playwright samples in the same client show the injection step: set the response field, fire the events and call the widget callback. For the API parameters of each type, see the per-type examples and the API reference.

  • Python client: https://github.com/deathbycaptcha/deathbycaptcha-api-client-python
  • API documentation: https://deathbycaptcha.com/api

Common pitfalls

  • Using a CAPTCHA solving service for illegitimate purposes instead of legitimate automation and testing.
  • Hard-coding credentials or API keys in client-side code that users can inspect.
  • Sending the wrong CAPTCHA type parameter, which returns incorrect or empty responses.
  • Failing to poll for the solution status and not handling timeouts gracefully.
  • Scaling automation without monitoring error rates, response times, and CAPTCHA type coverage.
DBC
Written by DeathByCaptcha Engineering Team
DeathByCaptcha engineers build and operate the CAPTCHA solving technology behind this site. Articles are written by our technical team and checked for accuracy before publishing.
Reviewed by DeathByCaptcha Editorial Team

Get your API key — start solving

Free to try · No credit card · Python, Node.js, Java, C# SDKs

Get your API key


Status: OK

Servers are fully operational with faster than average response time.
  • Average solving time
  • 5 seconds - Normal CAPTCHAs (1 min. ago)
  • 7 seconds - reCAPTCHA V2, V3 (1 min. ago)
  • 4 seconds - others (1 min. ago)
Chrome and Firefox logos
Browser extensions available

Updates

  1. Nov 19: We are currently updating our cryptocurrency payment processor, Hekelet, so crypto payments may be temporarily unavailable. In the meantime, you can purchase credits through 2Checkout or contact our support team for assistance. We expect normal crypto payment processing to be restored within this week. Thank you for your patience and understanding!
  2. May 13: Crypto payments got better! You can now purchase your CAPTCHAs using cryptocurrency through the Hekelet payment processor at https://deathbycaptcha.com/user-pay and receive an extra 20% FREE CAPTCHA credit with every package purchased this way.
  3. Apr 15: GitHub Updates: We’ve upgraded our libraries, expanded sample code, enhanced documentation, and added support for C++ and Go, making integration smoother than ever. Explore what’s new at github.com/deathbycaptcha!

  4. Previous updates…

Support

Our system is designed to be completely user-friendly and easy-to-use. Should you have any trouble with it, simply email us at DBC technical support emailcom, and a support agent will get back to you as soon as possible.

Live Support

Available Monday to Friday (10am to 4pm EST) Live support image. Link to live support page