Cloudflare Bypass Techniques for Web Scraping: 2026 Guide

Most articles about Cloudflare bypass techniques for web scraping are written by people who have never watched a scraper die at 3am. They give you a list of libraries, tell you to rotate residential proxies, and stop. Then you deploy, and you get a 403 with a Ray ID on every third request, and none of the advice tells you which of the six things you changed was the one that mattered.

This guide is organised differently. It walks the request path from the TCP handshake up to the behavioural telemetry, because Cloudflare is not one gate — it is five overlapping checks that happen in a fixed order, and each one fails in a way you can recognise. If you can read the failure, you can fix the right layer instead of throwing money at proxies for a problem your TLS stack caused.

One framing note before we start. Cloudflare protects a huge share of the public web, including sites whose data is genuinely public and whose owners have no objection to being read politely. It also protects login pages, checkout flows and private APIs. The techniques below are the same either way; the judgement about where to point them is yours, and there is a section near the end about where the lines actually sit.

What Cloudflare Is Actually Doing When It Blocks You

The first mental model to throw away is the idea that there is a single Cloudflare check you either pass or fail. What exists instead is a bot score — a number, roughly 1 to 99, where 1 means almost certainly automated and 99 means almost certainly a person — computed at the edge from dozens of signals. Cloudflare's own documentation on how the bot score is calculated describes it as a blend of machine learning, heuristics, JavaScript detections and behavioural analysis.

The site owner then writes rules against that score. One zone blocks below 30. Another challenges below 50 only on /search and /login. A third runs Bot Fight Mode, which is the free-tier blunt instrument: it JS-challenges nearly everything and hard-blocks traffic from ASNs it considers hosting providers.

This matters because bypass is the wrong word. You are not defeating a lock. You are raising a score above whatever threshold this particular site chose, using signals you control. That reframing changes your priorities immediately: on a lenient zone, fixing your TLS stack alone might take you from blocked to fine, and a full browser farm would be an expensive waste.

The five layers, in the order they are evaluated

1. Network and IP reputation. Your exit IP, its ASN, its history, whether it belongs to a known hosting provider or a residential ISP.
2. TLS handshake. The exact shape of your ClientHello, summarised as a JA3 or JA4 hash.
3. HTTP/2 framing and headers. SETTINGS values, window sizes, pseudo-header order, header order and casing, Client Hints coherence.
4. The JavaScript challenge. A script from /cdn-cgi/challenge-platform/ that probes your browser environment and posts the result back.
5. Behaviour over time. Request rate, concurrency, navigation patterns, telemetry from Cloudflare's beacon.

A request that fails at layer 2 never reaches layer 4. That is genuinely good news: it means you can bisect.

Read the error before you write the fix

Cloudflare tells you far more than people notice. The status code, the numbered error on the block page and the response headers together narrow the problem to one or two layers.

The distinction between 1015 and a bot-score block is the single most expensive confusion in this field. Teams spend a fortnight perfecting TLS impersonation for a site that was only ever rate limiting them, and a two-line change to their concurrency would have fixed it on day one.

Layer 1: The TLS Handshake

Before your scraper sends a single byte of HTTP, it has already announced what it is. The ClientHello message contains an ordered list of cipher suites, a set of extensions in a specific order, supported elliptic curves, signature algorithms and ALPN protocols. Hash that tuple and you get a JA3 fingerprint — the technique originally published by Salesforce and now largely superseded by JA4, which is more resistant to trivial shuffling.

Here is the problem in one sentence: Python's requests, httpx and aiohttp all hand the handshake to OpenSSL with their own defaults, and no shipping version of Chrome produces that handshake. Neither does Node's undici. Your fingerprint is not merely unusual — it is a known-scraper fingerprint that appears in every bot-detection vendor's corpus.

By 2026 the gap has widened. Chrome sends GREASE values in randomised slots, negotiates a post-quantum key share by default (X25519MLKEM768 since Chrome 131), and offers Encrypted Client Hello where the server supports it. A stack that offers none of that, while claiming in its User-Agent to be a recent Chrome, is not just anomalous. It is self-contradicting, and contradiction is the strongest bot signal there is.

What actually works

The practical options are libraries that borrow a real browser's TLS stack rather than reimplementing it:

• curl_cffi (Python) wraps curl-impersonate, which is patched to reproduce Chrome, Safari and Firefox handshakes byte for byte.
• tls-client and similar Go tools built on uTLS, usable from any language over a local HTTP proxy.
• rnet / primp and other Rust-backed clients with impersonation profiles.
• A real browser, which has the correct handshake by construction.

The trap is version drift. Impersonation libraries lag Chrome releases by weeks or months. If you impersonate chrome124 while advertising Chrome 141 in your User-Agent, you have replaced one mismatch with a subtler and more damning one. Pin your advertised UA to the impersonation target you actually ship, and update both together or neither.

One more subtlety: because these libraries are popular, the exact combination of a curl-impersonate JA4 plus a datacenter ASN plus a Python-shaped request cadence is itself a recognisable pattern. TLS impersonation raises your floor. It does not make you invisible.

Layer 2: HTTP/2 Framing and Header Discipline

Once the handshake passes, HTTP/2 offers a second fingerprint that most scrapers never think about. It is sometimes called the Akamai fingerprint, and it is built from things you did not know you were choosing:

• The SETTINGS frame: which parameters you send, their values, and the order you send them in. Chrome, Firefox and Safari each have a distinct signature here.
• The initial WINDOW_UPDATE increment.
• Whether you send PRIORITY frames, and with what dependencies.
• The order of pseudo-headers. Chrome sends :method, :authority, :scheme, :path. Firefox sends :method, :path, :authority, :scheme. Getting this wrong while claiming to be Chrome is a free giveaway.
• The order of regular headers after that.

Good impersonation libraries handle all of this for you. Hand-rolled HTTP/2 clients almost never do. If you are writing raw h2 frames, you are choosing a very hard game.

Client Hints and Sec-Fetch coherence

Above the framing sits the header set itself. Real Chrome sends sec-ch-ua, sec-ch-ua-mobile and sec-ch-ua-platform on every navigation, plus the Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-User and Sec-Fetch-Dest quartet. MDN's reference on User-Agent Client Hints is worth reading properly, because the details are where scrapers slip.

Things that get flagged:

• A top-level document request with no sec-fetch-dest: document.
• sec-ch-ua listing a Chrome major version that disagrees with the UA string.
• Missing the GREASE brand entry that Chrome inserts ("Not_A Brand";v="8" and its rotating variants).
• accept-language set to en-US,en;q=0.9 while the proxy exits in Warsaw and the browser reports Europe/Warsaw.
• Header names title-cased over HTTP/1.1 in the Python style when Chrome would not.
• Sending Accept-Encoding: gzip only, while Chrome offers gzip, deflate, br, zstd.

None of these individually gets you blocked on a lenient zone. Three of them together on a strict one will.

Layer 3: The JavaScript Challenge and the cf_clearance Cookie

When a request looks borderline rather than bad, Cloudflare issues a challenge instead of a block. Cloudflare documents the types of challenge it serves: Managed Challenge, JS Challenge and Interactive Challenge. In practice you get a small HTML page that loads obfuscated JavaScript from /cdn-cgi/challenge-platform/, which probes the browser environment, sometimes renders a Turnstile widget, and posts a result back.

Pass it and you receive a cf_clearance cookie. This cookie is the most useful object in the entire scraping stack, and the most commonly misused.

The cookie is bound to more than your session

cf_clearance is valid only for the combination of the zone, the exit IP and the User-Agent that were in play when it was issued. Change any of the three and the next request gets re-challenged. This explains an enormous amount of confused debugging:

• You solve the challenge in a browser on your laptop, paste the cookie into your scraper running on a server, and it fails. Different IP.
• You solve it through a rotating residential proxy that gives you a new IP every request. It fails on request two.
• You solve it in Chrome and replay through Python with a slightly different UA string. It fails.

The lifetime is set by the site owner. Thirty minutes is common; some zones extend it to days.

The highest-leverage technique in this article

Once you internalise the binding rule, an efficient architecture falls out of it:

Use a real browser as a key issuer, not as a fetcher.

Open one real browser session on a sticky proxy IP. Let it pass the challenge naturally. Extract cf_clearance, the exact User-Agent, and the rest of the cookie jar. Then hand all of that to a cheap HTTP client — one with correct TLS impersonation and matching headers — pinned to the same exit IP, and let it fetch hundreds of pages at a fraction of the CPU cost. When the cookie expires or a 403 returns, mint a new one.

The browser session costs you perhaps 300MB of RAM for ten seconds. Amortised over 500 page fetches, it is nearly free. Compare that to running a headless browser for every single page, which is what most tutorials implicitly recommend and what most scraping bills are actually made of.

The catch: your follow-up requests must still present a plausible TLS and HTTP/2 fingerprint. A valid cf_clearance presented over a raw Python handshake will often be rejected anyway, because the edge re-evaluates the whole request, not just the cookie.

Layer 4: Browser Fingerprint and Automation Tells

If you do run a browser — to mint cookies, to render an SPA, or because the target is aggressive — the challenge script gets to interrogate it directly. This is where the antidetect world and the scraping world converge, and where most stealth plugins quietly fail.

The automation tells

The short list of things a challenge script looks for:

• navigator.webdriver returning true — the standardised, honest flag that Playwright and Puppeteer set by default.
• Chrome launched with --enable-automation, which changes several observable behaviours and shows an infobar.
• --disable-blink-features=AutomationControlled, which is itself a detectable configuration in some builds. Fixing a tell by adding another tell is a lateral move.
• CDP attachment. This is the deep one. Calling Runtime.enable over the Chrome DevTools Protocol changes observable behaviour inside the page — the classic probe involves the cost or side effects of accessing the stack property on an Error object while a debugger is listening. Any framework that keeps a DevTools client attached with Runtime enabled is announcing itself continuously.
• Timing regularity in synthetic input events. Real mouse movement has jitter, acceleration and overshoot; a scripted click at exact element centre with zero preceding movement does not.

The practical conclusion is that automation frameworks are easier to detect than automation itself. Driving a browser over a narrow slice of CDP — DOM, Input, Page, Network — without ever enabling Runtime keeps navigator.webdriver false and leaves far less residue than a stock Playwright session. That is the design choice behind Dual Login's automation layer, and it is worth copying regardless of which tool you use.

The fingerprint tells

The other half is the device identity itself: canvas and WebGL rendering, audio context output, installed fonts, hardwareConcurrency, deviceMemory, screen dimensions, timezone, language list. If you want the ground-up version of how these combine, Browser Fingerprinting Explained for Beginners is the primer, and the EFF's Cover Your Tracks will show you your own entropy in about ten seconds.

For scraping specifically, one principle dominates everything else:

Consistency beats rarity. A fingerprint that is uncommon but internally coherent survives. A fingerprint that is common but self-contradictory does not.

Contradictions that get scrapers caught:

• User-Agent says macOS; WebGL renderer says ANGLE (NVIDIA GeForce RTX 3060 Direct3D11 vs_5_0 ps_5_0).
• Timezone resolves to America/New_York; proxy exits in Jakarta.
• screen.width of 800 and screen.height of 600, which effectively no real user has.
• WebGL renderer reporting SwiftShader or Mesa llvmpipe — the software renderer signature of a GPU-less server.
• navigator.plugins empty on a UA claiming desktop Chrome, which ships a PDF viewer entry.
• Notification.permission returning denied while navigator.permissions.query reports prompt — a headless mismatch that has been used as a detection for years.

Modern headless Chrome closes most of the old gaps, so headless is no longer automatically fatal. But it still runs without a GPU on most servers, and the WebGL renderer string is the tell that follows from that. Either give the container GPU access, or spoof the renderer coherently — the practical mechanics of doing that without creating new contradictions are covered in How to Change Browser Fingerprint.

Layer 5: IP Reputation, Rate and Behaviour

Everything above concerns a single request. Layer 5 is about the shape of a thousand of them.

Proxy types, honestly ranked

The non-obvious point is that stickiness matters more than pool size for Cloudflare work, precisely because of the cf_clearance IP binding. A hundred sticky ISP IPs you can hold for an hour each will outperform a million rotating residential IPs you get for one request each, for anything that involves passing a challenge.

Also worth stating plainly: a proxy changes where you appear to be. It does nothing about your TLS handshake, your headers or your browser fingerprint. That difference is the whole subject of Antidetect Browser vs VPN, and it is why proxy-only strategies plateau so quickly.

Pacing and concurrency

Cloudflare's rate limiting is typically per IP, often per path. Concurrency hurts you more than total volume does. Some rules of thumb that have held up:

• One to three concurrent connections per exit IP. A human browsing does not open twenty simultaneous document requests.
• Jitter your gaps. Requests at exactly 1000ms intervals are a machine signature that survives every other disguise. Draw from a distribution.
• Honour Retry-After. Ignoring it converts a temporary throttle into a zone-level ban.
• Back off exponentially on 429 and 1015, but only on the affected IP. Global backoff wastes your whole pool because one exit got hot.
• Cache aggressively and use conditional requests. If-Modified-Since and ETag cost you a 304 instead of a full page, and 304s barely register.
• Do not crawl breadth-first through a sitemap at 3am local time for a site whose traffic is 90% daytime and regional. Distribute across the day.

Behavioural telemetry

On zones running the full challenge platform, Cloudflare's JavaScript beacon posts environment and interaction telemetry back to /cdn-cgi/challenge-platform/ endpoints. Loading the page HTML and never fetching or executing that script is itself a pattern. If you are operating at Tier 2 (browser-issued cookies, HTTP-client fetches), be aware that pure HTTP fetches never produce beacon traffic, which is one reason clearance cookies sometimes expire early on strict zones.

A Tiered Architecture That Does Not Waste Money

Put the layers together and a cost-shaped strategy emerges. The rule is simple: never start at the most expensive tier, and always instrument so you know which tier a given domain actually needs.

The factor between Tier 1 and Tier 3 is not a rounding error. A browser instance costs roughly 200 to 400MB of RAM and a meaningful slice of a CPU core; an HTTP request costs kilobytes. Teams that default to Playwright for everything are frequently paying a hundred times over for pages that curl_cffi would have fetched.

Tier 0 deserves more attention than it gets

Before any of this, spend an hour looking for the unprotected route. Genuinely often there is one:

• sitemap.xml and its children, usually served from cache with no challenge.
• RSS or Atom feeds, still widely published and rarely protected.
• The JSON endpoint the site's own front end calls, which frequently sits behind lighter rules than the HTML page — check the Network tab.
• The mobile app's API, which sometimes bypasses the web zone's rules entirely.
• An official, documented API. Paying for it is often cheaper than the engineering time you are about to spend.

A half-day of reconnaissance regularly eliminates the need for the other four layers.

Debugging Playbook: Isolating the Layer in Four Tests

When something breaks, resist the urge to change three things at once. Bisect instead. Each test below isolates exactly one variable:

1. Open the URL in a normal browser on your own connection. Works? Then the site is not down and the content is reachable. Fails? The zone tightened, or you are already IP-banned.
2. Same URL, curl_cffi with impersonation, from your own connection. Works? Your TLS and header layer is fine, and the problem is your server's IP. Fails? The problem is in layers 2 to 4.
3. Same request, same client, through your proxy. Works? Your server itself is the issue. Fails? The proxy IP or ASN is the issue.
4. Same request from the production server. Now any remaining difference is environmental — a stale UA constant, a missing header your framework strips, a different TLS build.

Always log the Ray ID, the numbered error, and the presence of cf-mitigated. Save the full response body of the first failure of each kind; the block page text distinguishes a WAF rule from a bot-score block from a rate limit, and you cannot recover that from a status code.

Finally, keep a canary: one known-protected URL you fetch hourly with your standard stack, with the result graphed. When a zone flips on Bot Fight Mode on a Tuesday afternoon, you want to know it was them, not a regression in your code. In my experience roughly half of all sudden scraper failures are configuration changes on the target's side, and teams without canaries burn days looking for a bug that does not exist.

Where an Antidetect Browser Fits — and Where It Does Not

Let us be precise, because the marketing in this space is not.

An antidetect browser is not a Cloudflare bypass tool. It will not defeat a WAF rule, it will not clean a burned IP, and installing one will not make a badly paced crawler polite. If someone tells you otherwise, discount everything else they say.

What it is is infrastructure for running many coherent, isolated, persistent browser identities at once. For scraping at scale, that buys you three specific things:

Clearance minting at scale. Tier 2 needs a supply of distinct browser identities, each pinned to a sticky proxy, each producing a clean cf_clearance. Doing that with hand-rolled Playwright contexts means writing and maintaining your own fingerprint consistency layer. That is a real project.

Persistent authenticated sessions. Plenty of valuable data sits behind a login. A per-profile data directory means cookies, localStorage and IndexedDB survive restarts, so you are not re-authenticating (and re-triggering risk checks) on every run. If your work involves logged-in accounts, the account-safety side of this is covered in Best Antidetect Browser for Multiple Accounts.

A test bench. Before committing to a proxy plan, you want to see what a given fingerprint-and-proxy pair actually looks like from the outside. Running one profile against a fingerprint test page tells you more in five minutes than a week of theorising.

The implementation details that matter for scraping, in Dual Login's case: the fingerprint is applied natively inside the engine rather than injected as JavaScript, so a challenge script cannot find a patched getter or an unusual toString on a spoofed function; each profile gets its own data directory and its own proxy; and the automation layer drives tabs over a deliberately narrow slice of CDP that never enables Runtime, so navigator.webdriver stays false and there is no debugger residue in the page. If the whole category is new to you, What Is an Antidetect Browser and How Does It Work is the ground floor.

Mistakes That Kill Otherwise-Competent Scrapers

A list assembled from real post-mortems rather than theory:

• Rotating the exit IP mid-session. Instantly invalidates cf_clearance. If you rotate per request, you are re-solving the challenge per request and paying for the privilege.
• Copy-pasting a User-Agent from a Stack Overflow answer. Chrome 108 does not exist in the wild any more. An impossibly old UA is a louder signal than a slightly wrong one.
• Running two hundred headless instances on one datacenter /24. The ASN reputation collapses, and it takes the whole range with it — including whatever else you host there.
• Solving a challenge and discarding the cookie. Extraordinarily common, and the single biggest source of wasted compute in scraping stacks.
• Treating a 1015 as a fingerprint problem. Rate limits do not care what your JA4 hash is.
• Updating the impersonation profile without updating the advertised UA. Or vice versa. Pin them together in one config value.
• Ignoring robots.txt completely. Beyond the ethics, aggressive crawling of disallowed paths is exactly what prompts an operator to tighten their rules — and then everyone, including polite crawlers, suffers.
• No canary, no per-tier success metrics, no saved failure bodies. You cannot debug what you did not record.

Legal and Ethical Guardrails

This section is short but not optional.

Scraping publicly accessible data is broadly lawful in many jurisdictions, and US courts have generally been reluctant to treat access to public web pages as unauthorised computer access — the hiQ v. LinkedIn line of cases and the Supreme Court's narrowing of the CFAA in Van Buren both point that way. That is not legal advice, and it is not a global rule.

What changes the analysis:

• Authentication. Data behind a login you agreed to terms to obtain is a materially different situation from data served to anonymous visitors.
• Personal data. GDPR and similar regimes apply to personal information regardless of whether it was easy to collect. Public does not mean unregulated.
• Copyright. Facts are generally not protected; substantial verbatim reproduction of expressive content usually is.
• Load. There is a real line between reading a site and degrading it. Stay far on the right side of it. Cache, use conditional requests, and cap your rate well below anything that could affect other users.

Practical decency also happens to be good engineering: identify your crawler where you reasonably can, honour robots.txt as a statement of intent even where you are not legally bound by it, respond to takedown requests, and prefer an official API when one exists. Sites that feel scraped into the ground tighten their rules, and the people who pay for that are everyone who comes after.

FAQ

Is it legal to bypass Cloudflare for web scraping?

There is no single answer, because Cloudflare is a service the site owner configured, not a law. Accessing publicly available pages has generally been treated leniently by US courts, while circumventing authentication, ignoring an explicit ban, collecting personal data without a lawful basis, or degrading a site's availability all carry real risk. Jurisdiction matters, terms of service matter, and the nature of the data matters most of all. Get advice for anything commercial or large scale.

Does changing my User-Agent bypass Cloudflare?

No, and it can make things worse. The User-Agent is one of the weakest signals in the stack — Cloudflare has already fingerprinted your TLS handshake and HTTP/2 framing before it reads a single header. A Chrome User-Agent attached to a Python TLS fingerprint is a contradiction, and contradictions score worse than a plain, honest client would.

Why does my cf_clearance cookie stop working immediately?

Almost always because the exit IP or the User-Agent changed. The cookie is bound to the zone, the IP and the UA that were used when it was issued. Rotating proxies per request, solving the challenge locally and replaying from a server, or normalising the UA string somewhere in your pipeline will each break it. Pin all three together and reuse the cookie until it genuinely expires.

Can I scrape Cloudflare-protected sites without a headless browser?

Often yes, and it is usually far cheaper. A HTTP client with proper TLS impersonation, correct HTTP/2 framing and coherent headers clears a large share of Cloudflare-protected sites on its own. For zones that challenge on first contact, use a browser once to mint a cf_clearance cookie and then serve the volume through the cheap client on the same sticky IP. Reserve full browsers for JavaScript-rendered content and the most aggressive zones.

Are residential proxies enough on their own?

No. A residential IP fixes your ASN reputation and nothing else. Your TLS fingerprint, HTTP/2 signature, headers and browser environment are all unchanged, and any one of them can still fail you. Residential and mobile proxies are best thought of as removing a negative signal, not adding a positive one.

Is an antidetect browser better than a stealth plugin for this?

For one-off scripts, a stealth plugin is fine and free. It stops being enough when you need many persistent, mutually isolated, internally consistent identities running at once, each with its own proxy and its own cookie jar — because at that point you are maintaining a fingerprint-consistency layer yourself, and patched JavaScript getters are exactly what challenge scripts probe for. Native, engine-level spoofing with per-profile data directories solves a different-sized problem. If you are weighing tools, Cheaper Multilogin Alternatives That Actually Work compares the practical trade-offs.

Wrapping Up

The useful summary of Cloudflare bypass techniques for web scraping is not a library list. It is a diagnostic habit: identify which of the five layers rejected you, fix that one, and measure whether it helped. Most scrapers fail at layer 2 or layer 5 — a wrong TLS fingerprint or a careless request rate — and both are cheap to fix once you know that is where the problem is. The expensive layers, real browsers and premium proxies, should be the exception you escalate to, not the default you start from.

And keep the cost model in view. A browser session that mints a clearance cookie for five hundred subsequent HTTP fetches is a hundred times cheaper than a browser per page, and it fails less often, because fewer moving parts touch the target.

If your work needs many consistent, isolated browser identities — for minting clearance cookies at scale, for keeping authenticated sessions alive across runs, or simply for testing what a given fingerprint and proxy pair looks like from the other side — that is the part Dual Login is built for: native fingerprinting with no injected JavaScript, a real data directory per profile, a proxy per profile, and automation that never announces itself. Spin up a couple of profiles, point them at a fingerprint test page, and see what your setup actually looks like before you commit a budget to it.