Fixing "Just a moment..." Cloudflare Error in Scrapy

Direct Answer: The "Just a moment..." error in Scrapy happens because Cloudflare detects Scrapy's default headers and TLS fingerprint, triggering a JavaScript challenge. To fix it without using heavy headless browsers like Playwright, route your Scrapy requests through a bypass API like Volkonskii8.

How to integrate the bypass API into Scrapy

Instead of yielding a standard scrapy.Request, you can send a POST request to the bypass API from your start_requests or parse method, receive the raw HTML, and parse it.

import scrapy
import requests
import json

class MySpider(scrapy.Spider):
    name = "bypass_spider"

    def start_requests(self):
        target_url = "https://target-site.com/protected-page"
        
        # 1. Call the Bypass API
        api_response = requests.post(
            "https://cloudflare-bypass.shop/v1/solve",
            headers={
                "X-API-Key": "YOUR_API_KEY_HERE",
                "Content-Type": "application/json"
            },
            json={"url": target_url},
            timeout=200
        )
        
        # 2. Check for success and yield a Scrapy response
        if api_response.status_code == 200:
            # We got the raw HTML. Feed it directly into Scrapy's parser.
            yield scrapy.http.HtmlResponse(
                url=target_url, 
                body=api_response.text.encode('utf-8'), 
                encoding='utf-8'
            )
        else:
            self.logger.error(f"Bypass failed: {api_response.text}")

    def parse(self, response):
        # Your parsing logic goes here
        title = response.css("title::text").get()
        self.logger.info(f"Successfully parsed page: {title}")

Important Note for Scrapy Users

Cloudflare injects its analytical scripts (like cdn-cgi/challenge-platform) at the bottom of every successful page. Do not write Scrapy middlewares that drop responses if they contain the word "cloudflare" or "challenge" — this will break your scraper on successful requests.

Frequently Asked Questions

Do I need Playwright or Selenium?

No. The bypass API handles JS rendering on its own servers, so you only need standard HTTP requests in your Scrapy spider.

Why does Scrapy trigger the "Just a moment..." page?

Cloudflare detects Scrapy's default headers and TLS fingerprint, which do not match a real browser. This triggers the JavaScript challenge instead of returning the page.