← Back to CF Bypass

Web Scraping Without Getting Blocked: 2026 Playbook

Updated November 2026 · 8 min read

Nobody wants to get blocked. But most guides on this topic are from 2019 and give outdated advice. This is the practical 2026 playbook — what works, what doesn't, and how to decide whether to build or buy your bypass layer.

1. Start with rate limits (always)

Before you touch proxies or browsers, respect the rate limit. A single IP making 1 request per second to a mid-size site is usually fine. 10 per second gets you banned. 100 per second gets you banned plus a firewall rule against your IP range.

import time
import requests

for url in urls:
    r = requests.get(url, headers=HEADERS)
    process(r)
    time.sleep(1.0)  # 1 req/sec — safe on most targets

2. Send real headers

Default python-requests/2.x in the User-Agent is a giant "block me" sign. Copy a real Chrome header set. But don't stop there — order matters, and headers must match the browser you claim to be.

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1",
}

3. TLS fingerprinting is the new wall

Cloudflare, Akamai, DataDome and PerimeterX all fingerprint your TLS handshake (JA3/JA4). Python's requests and httpx produce a fingerprint that no real browser has. You can fake it with curl_cffi:

from curl_cffi import requests as cffi_requests

r = cffi_requests.get(url, impersonate="chrome120")

This alone gets you past ~40% of Cloudflare-protected sites. Not all — managed challenges still need JS.

4. Proxies: residential or bust

Datacenter proxies (Hetzner, DigitalOcean IPs) get challenged immediately. Residential proxies cost $2–5/GB but actually work. In 2026, the ranking of proxy types by "gets past Cloudflare" is:

  1. Mobile proxies — nearly unstoppable, very expensive ($15–30/GB)
  2. Residential rotating — works well, $2–5/GB
  3. ISP proxies — depends on the provider, $1–3/proxy/month
  4. Datacenter — gets blocked on major sites, fine for small ones

5. Cloudflare is its own problem

If a site uses Cloudflare's managed challenge (Turnstile), none of the above is enough — you must actually run JavaScript and solve a proof-of-work. You have two options:

Option A: DIY with Playwright + proxy

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": "http://proxy:port"})
    page = browser.new_page()
    page.goto("https://target.com", wait_until="networkidle")
    html = page.content()

Costs: ~2 GB RAM per browser, ~3–5 seconds per request, and you still need to solve Turnstile somehow (services like 2Captcha charge $1–3 per 1000 solves).

Option B: bypass API

import requests

r = requests.post(
    "https://cloudflare-bypass.shop/v1/solve",
    headers={"X-API-Key": "YOUR_KEY"},
    json={"url": "https://target.com"},
    timeout=200
)
html = r.text

Costs: $0.03 per success. No RAM, no proxies, no browser. The API handles all of the above on its side.

6. The "should I build or buy" decision

Do the math on a 3-month horizon:

ScenarioDIY costBypass API cost
10K requests, no blocks$0$300
10K requests, 50% blocked$30 + 20 hrs work$150
100K requests, Cloudflare$500 proxies + $300 server + 60 hrs work$3000

Rule: if your time is worth more than $20/hour, buy the bypass for anything Cloudflare-related. Build for simple targets.

Bottom line: send real headers, respect rate limits, use residential proxies, and when Cloudflare is involved, use an API. Don't burn weeks on a homemade solution that Cloudflare will break next Tuesday.
Start scraping in 3 lines: get a key at cloudflare-bypass.shop. Pay only for successes, USDT or USDC.

7. Verifying the response

Once you're past the block, don't second-guess. A 200 OK with > 10,000 characters is a real page. If you're seeing 200 OK with 500 characters, you got a challenge page — retry or escalate to a bypass API.

Frequently Asked Questions

What is the most common reason scrapers get blocked?

Rate limits and bot-like TLS fingerprints. Sending too many requests from one IP gets you banned within minutes. Even at low rates, Python's default requests fingerprint is flagged by Cloudflare, Akamai and DataDome as a known bot signature.

Do residential proxies actually help?

Yes, on sites that score IP reputation. Datacenter IPs (Hetzner, DigitalOcean) are challenged immediately; residential proxies pass most of them. But residential proxies do not solve the JavaScript challenge — for that you still need a browser or a bypass API.

Should I build my own bypass or use an API?

Build only if you have time to maintain it. Cloudflare changes its challenge every few weeks, and self-built bypasses break. If your time is worth more than $20 per hour, a bypass API at $0.03 per success is cheaper than DIY.

How do I know a bypass API actually returned the page?

Check three things: 200 OK status, body length over 10,000 characters for real pages, and a Content-Type that matches the target. Do not scan the HTML for "cloudflare" — every protected page contains that string, so you will get false positives.

What is the single cheapest way to scrape a Cloudflare-protected site?

POST the URL to a bypass API that returns raw HTML and charges per success. CF Bypass charges $0.03 per bypass, refunds failures, and requires no subscription. For a few thousand requests a month, it is cheaper than any proxy + captcha solver combination.

Related: Python guide · ZenRows alternatives · Scrapy + Cloudflare · API comparison 2026 · requests without Selenium