Handling cookie CAPTCHAs in Scrapy with DeathByCaptcha
Scrapy is built for volume: many requests, many domains, retries and pipelines. A challenge in the middle of a crawl breaks that rhythm. The official Python client ships an example downloader middleware that detects a cookie-based challenge, solves it with DeathByCaptcha and retries the request with the token as a cookie.
The example lives in the Python client:
examples/scrapy/middleware.pyexamples/scrapy/spider.pyexamples/scrapy/README.md
It is an example to copy, not an installed package.
What it handles
- Works without a browser: Amazon WAF (type 16) and DataDome (type 21) when the service returns a cookie value. The example implements WAF and documents the DataDome shape.
- Does not work without a browser: reCAPTCHA and GeeTest. Scrapy does not
execute JavaScript, so the middleware raises
EscalationRequiredand points to the Playwright samples. Pretending otherwise would only move the failure somewhere less obvious.
Setup
From the repository root:
pip install -e .
pip install -r examples/scrapy/requirements.txt
Then enable the middleware in your Scrapy project:
DOWNLOADER_MIDDLEWARES = {
"middleware.DeathByCaptchaMiddleware": 543,
}
DBC_USERNAME = "your_username"
DBC_PASSWORD = "your_password"
# or: DBC_AUTHTOKEN = "your_authtoken"
# Required for challenges tied to the exit IP.
DBC_PROXY = "http://user:[email protected]:1234"
DBC_PROXYTYPE = "HTTP"
DBC_MAX_RETRIES = 2
Copy middleware.py where Python can import it, or add the example folder
to PYTHONPATH.
The core of the middleware
process_response sees the challenge, extracts the WAF parameters from the
page, asks DeathByCaptcha for the token and retries:
def process_response(self, request, response, spider):
retries = request.meta.get("dbc_retries", 0)
if retries >= self.max_retries:
return response
if self._is_recaptcha_or_geetest(response):
raise EscalationRequired(
"This challenge needs JavaScript: use the Playwright samples."
)
if not self._is_waf_challenge(response):
return response
params = self._waf_params(response)
payload = {
"sitekey": params["sitekey"],
"pageurl": response.url,
"iv": params["iv"],
"context": params["context"],
}
captcha = self.client.decode(type=16, waf_params=json.dumps(payload))
if not captcha or not captcha.get("text"):
return response
retry = request.replace(
dont_filter=True,
meta={**request.meta, "dbc_retries": retries + 1},
)
retry.cookies["aws-waf-token"] = captcha["text"]
return retry
The detection is deliberately conservative: a 403, 429 or 503 with Amazon WAF
markers, or a body that mentions aws-waf-token. The parameters are read
from the inline scripts (key, iv, context), the same values the browser
uses when it loads the challenge.
Why a proxy is not optional
Amazon WAF, DataDome and Turnstile tokens are tied to the exit IP. The
service solves the challenge from its own IP unless you send a proxy, and a
token solved from one IP is rejected by a request that leaves from another.
The middleware passes DBC_PROXY through to the service, and the crawler
must use the same proxy. The example documents this because a missing proxy
looks exactly like a solver bug.
Retries and failure
DBC_MAX_RETRIEScaps the attempts. A challenge that keeps coming back is dropped with a log line; the middleware never loops forever.- The token is never written to the crawl log; verbose mode stays off.
- If the page exposes reCAPTCHA or GeeTest, the middleware raises instead of retrying with a token that Scrapy cannot inject.
Things to know
- Scrapy is not a browser. Any challenge that needs JavaScript needs the Playwright path. Use the middleware for cookie challenges and escalate for the rest.
- Point the spider at a page you own or are allowed to crawl. The sample spider uses a placeholder URL for that reason.
- The middleware is an example, not a package. Copy it, adapt the selectors and the parameter extraction to your target, and keep the retry cap.
Next steps
For reCAPTCHA and Turnstile on pages that run JavaScript, the Playwright samples in the same client show the injection step: set the response field, fire the events and call the widget callback. For the API parameters of each type, see the per-type examples and the API reference.
- Python client: https://github.com/deathbycaptcha/deathbycaptcha-api-client-python
- API documentation: https://deathbycaptcha.com/api

English
Spanish
Russian
Chinese
French
Hindi
Arabic
Bengali
Indonesian
Portuguese
com, 