Direct Answer: The "Just a moment..." error in Scrapy happens because Cloudflare detects Scrapy's default headers and TLS fingerprint, triggering a JavaScript challenge. To fix it without using heavy headless browsers like Playwright, route your Scrapy requests through a bypass API like Volkonskii8.
Instead of yielding a standard scrapy.Request, you can send a POST request to the bypass API from your start_requests or parse method, receive the raw HTML, and parse it.
import scrapy
import requests
import json
class MySpider(scrapy.Spider):
name = "bypass_spider"
def start_requests(self):
target_url = "https://target-site.com/protected-page"
# 1. Call the Bypass API
api_response = requests.post(
"https://cloudflare-bypass.shop/v1/solve",
headers={
"X-API-Key": "YOUR_API_KEY_HERE",
"Content-Type": "application/json"
},
json={"url": target_url},
timeout=200
)
# 2. Check for success and yield a Scrapy response
if api_response.status_code == 200:
# We got the raw HTML. Feed it directly into Scrapy's parser.
yield scrapy.http.HtmlResponse(
url=target_url,
body=api_response.text.encode('utf-8'),
encoding='utf-8'
)
else:
self.logger.error(f"Bypass failed: {api_response.text}")
def parse(self, response):
# Your parsing logic goes here
title = response.css("title::text").get()
self.logger.info(f"Successfully parsed page: {title}")
Cloudflare injects its analytical scripts (like cdn-cgi/challenge-platform) at the bottom of every successful page. Do not write Scrapy middlewares that drop responses if they contain the word "cloudflare" or "challenge" — this will break your scraper on successful requests.
No. The bypass API handles JS rendering on its own servers, so you only need standard HTTP requests in your Scrapy spider.
Cloudflare detects Scrapy's default headers and TLS fingerprint, which do not match a real browser. This triggers the JavaScript challenge instead of returning the page.