Scaling Web Scraping to Millions of Pages: A Field Guide

Anyone can scrape ten thousand pages. A weekend, a Python script, a handful of free proxies — done. The interesting problems start two orders of magnitude later, and they are almost never the problems people prepare for. Teams budget for servers and bandwidth, then discover that the thing actually killing their crawl is a slowly rising block rate they didn't notice for nine days, or a queue that quietly stopped deduplicating, or the fact that all four hundred of their workers present the exact same browser fingerprint to a detection system that only needed to see it twice.

Scaling web scraping to millions of pages is not the same activity as scraping, done harder. It is a different discipline with different failure modes, and most of the hard-won knowledge lives in the heads of people who have burned proxy budgets learning it. This guide is an attempt to write that knowledge down: the math you should do before you spin up a single worker, the architecture that survives contact with reality, how proxies and browser fingerprints interact (and why fixing one without the other fixes nothing), and the monitoring that tells you you're in trouble while there's still time to react.

Where Large Crawls Actually Die

Before building anything, it helps to know what you're defending against. Large-scale scrapers rarely die of the causes beginners expect — CPU exhaustion, bandwidth limits, code crashes. Those are loud failures, and loud failures get fixed. The killers are quiet:

Block rate creep. Your crawl starts at a 0.5% failure rate. Two weeks in it's 4%. A month in it's 22%, and because your retry logic keeps hammering the same burned IPs and identities, every retry makes the reputation problem worse. Nobody noticed because nobody was graphing it.

Soft blocks. The nastier cousin. The server returns HTTP 200 with a CAPTCHA page, a stripped-down page, or subtly wrong data. Your pipeline reports success. Your dataset rots. You find out when a downstream consumer asks why 30% of product prices are null for the last three weeks.

Queue collapse. URL frontiers at scale are adversarial. Sites generate infinite URL spaces (calendars, faceted search, session IDs in paths), and a frontier without aggressive normalization and dedup will happily schedule the same logical page four million times while your actual targets starve.

Identity uniformity. You rotated IPs religiously, but every request came from an identical TLS stack, identical headers in identical order, identical canvas hash, identical screen size. Modern anti-bot systems cluster on those signals. One fingerprint making requests from 500 IPs doesn't look like 500 users — it looks like one bot with a proxy budget.

Everything in the rest of this guide is aimed at one of those four.

Do the Boring Math First

Most scaling mistakes are made before the first line of code, in the form of arithmetic nobody did. Run these numbers for your specific project:

Sustained throughput. One million pages per week is only ~1.65 pages per second, sustained. Ten million per month is ~3.9/sec. These are small numbers. With retries, redirects and asset overhead, budget 2–3x that in actual requests, but the point stands: for most projects, the bottleneck is never raw compute. A single modest VM can push 50 requests per second at the HTTP layer. The bottleneck is how many requests per second the target will tolerate from identities it trusts — which is a proxy and fingerprint problem, not a hardware problem.

Concurrency. Little's Law does the work: concurrency = target rate × average request latency. At 4 requests/second with a 3-second average latency through residential proxies, you need only ~12 concurrent requests in flight. Through slow mobile proxies at 8 seconds, ~32. People routinely provision 500-worker fleets for workloads that need 30 in-flight requests, then wonder why they're blocked — the fleet wasn't the constraint, the per-domain politeness ceiling was.

Cost per page. This is the number that decides your architecture. Residential proxy bandwidth commonly runs $2–8/GB. An average HTML document is 50–150KB, but a full browser page load with scripts, styles and images is 2–5MB. At $4/GB, a raw HTML fetch costs a fraction of a cent; a full unfiltered browser load can cost 1–2 cents. Multiply by ten million pages and the difference between those two numbers is the difference between a $1,500 project and a $150,000 one. This is why request filtering (blocking images, fonts, media, analytics) inside browser sessions isn't an optimization — it's survival.

Write these three numbers down for your project before you build. They will tell you, concretely, whether you can afford browsers everywhere, browsers for a subset, or must stay at the HTTP layer for the bulk of the crawl.

Stay at the HTTP Layer as Long as You Can

The single highest-leverage decision in scaling web scraping to millions of pages is deciding which pages actually need a browser. The answer is almost always: far fewer than you think.

What you can get without rendering anything

Before rendering a single page, spend a day on reconnaissance:

• Hidden JSON APIs. Most modern sites are a JavaScript shell over a JSON backend. Open devtools, watch the network tab, and you'll frequently find a clean, paginated, structured API feeding the page. Scraping the API instead of the rendered HTML is 50–100x cheaper, faster, and gives you typed data instead of brittle selectors.
• Server-rendered HTML. E-commerce category pages, listings, articles — a huge fraction of the web still arrives fully formed in the initial response. If curl shows you the data, you don't need Chromium to fetch it.
• Sitemaps and feeds. sitemap.xml files enumerate exactly the URLs a site wants indexed, often with last-modified timestamps that let you skip unchanged pages entirely. For recurring crawls, conditional requests with ETag/If-Modified-Since headers — well documented in MDN's HTTP caching guide — can turn a re-crawl of a million pages into a re-download of the forty thousand that actually changed.

A sensible large-scale system is a pyramid: HTTP-layer fetching for the 90% of pages that allow it, real browsers reserved for the 10% that don't.

When you genuinely need a real browser

Some targets leave you no choice:

• Content assembled client-side with no discoverable API.
• Anti-bot systems (Cloudflare, DataDome, PerimeterX/HUMAN, Akamai) that fingerprint the TLS handshake, HTTP/2 frame ordering and JavaScript environment before serving content. A plain HTTP client fails these checks by existing.
• Logged-in surfaces where a session must look like a continuous, human, stateful browsing history.

And here's the trap: reaching for stock headless Chrome often isn't enough. Headless Chromium — even the new headless mode Chromium shipped that unified it with the real browser — still differs from a real user's browser in ways detection vendors have spent a decade cataloguing: missing GPU-backed rendering quirks, telltale automation properties, an environment that has clearly never had a user in it. If you're going to pay the cost of running browsers at all, they need to be browsers that hold up under inspection. We'll come back to this in the fingerprinting section, because it's where most large crawls quietly fail.

An Architecture That Survives the First Million

At small scale, architecture doesn't matter; a loop over a URL list works. Past a few hundred thousand pages, three components decide whether your system is durable or a nightly firefight.

The frontier: queue, normalize, dedup

The URL frontier is the heart of a crawler and the first thing to collapse at scale.

• Normalize aggressively before enqueueing. Lowercase hosts, strip fragments, sort or strip tracking query parameters (utm_*, ref, session IDs), resolve relative URLs. Two spellings of the same logical page must hash identically, or your dedup is decorative.
• Dedup at enqueue time, not fetch time. A Bloom filter or Redis set of seen-URL hashes costs almost nothing and prevents the classic death spiral where a faceted-search page generates combinatorial URL variants until the queue is 95% garbage.
• Partition politeness by domain. A single global rate limit is wrong in both directions — it hammers small sites and underutilizes large ones. Give each domain (or each domain-per-identity pair) its own token bucket. This is also where per-domain concurrency caps live: no matter how big your fleet is, only N requests in flight per site.
• Priority tiers. Fresh product pages might matter more than pagination page 4,000. A frontier with priority lanes lets you keep the valuable crawl current even while the long tail churns.

Treat workers as disposable

At millions of pages, individual worker failure is not an event; it's weather. Design for it:

• Workers must be stateless — they lease a batch of URLs from the frontier, fetch, write results, and ack. A worker that dies mid-batch loses nothing; the lease expires and another worker picks it up.
• Writes must be idempotent — keyed by URL hash + fetch timestamp, so a retried batch overwrites rather than duplicates.
• Every URL carries a retry budget (three attempts, exponential backoff, different proxy/identity each attempt), and exhausted URLs go to a dead-letter queue for inspection rather than silent oblivion. The dead-letter queue is one of your best diagnostic instruments: when one domain suddenly dominates it, that domain changed something.

Store raw first, parse later

The most expensive mistake in large-scale scraping is coupling fetching to parsing. Fetch and parse in one step, and every parser bug or site redesign means re-fetching millions of pages — repaying your entire proxy bill to fix a regex.

Instead: store the raw response (compressed HTML plus response headers and fetch metadata — proxy used, identity used, timing, status) in cheap object storage, and run parsing as a separate, re-runnable stage. Compressed HTML averages 15–30KB per page; a million pages is ~20–30GB, which costs almost nothing to keep. When the site redesigns — and at this scale, some site in your target set redesigns every week — you fix the parser and re-run it over stored bytes in minutes.

Track parse yield (fraction of stored pages producing valid records) per domain per day. A yield cliff on one domain with a healthy fetch rate means a redesign or a soft block. It's one of the earliest honest signals you'll get.

Proxies: The Line Item That Decides Your Unit Economics

Proxies are usually the largest cost in a serious crawl and the most common thing teams get wrong — typically by buying one tier of proxy and using it for everything.

The economics push you toward a tiered strategy: cheap datacenter IPs for the large fraction of your URL set that doesn't fight back, metered residential for protected targets, and a small pool of ISP or mobile IPs pinned to your logged-in identities. Route by target, measured empirically: start every new domain on the cheap tier, and let your block-rate monitoring (below) promote it to a more expensive tier only when the data says so.

Rotation versus sticky sessions

Rotation policy is where proxy strategy meets identity strategy, and mismatching them is a classic self-inflicted block:

• Stateless fetches (public pages, no login) can rotate per request or per small batch. The goal is spreading load so no single IP exceeds a human-plausible request rate against the target.
• Stateful sessions need stickiness. A browsing session whose IP hops from Frankfurt to São Paulo to Denver across three pageviews is a stronger bot signal than either IP alone. Session-level stickiness — one IP for the life of one session — is the minimum; for logged-in work, pin one IP (or at least one city/ASN) to one identity for the identity's whole life.

The deeper point, which trips up almost everyone migrating from small-scale scraping: an IP is not an identity. Detection systems stopped treating it as one years ago. Your IP is one signal among dozens, and the others — TLS characteristics, header order, JavaScript environment, canvas and WebGL output, timezone, fonts — are all still screaming in unison if all you rotated was the address. That's the difference between routing traffic and changing identity, the same distinction covered in our breakdown of the antidetect browser vs VPN difference. Which brings us to the part of scaling that proxy vendors won't solve for you.

Fingerprints: Why Blocks Persist After You Fix Your Proxies

Here is the pattern, and if you run large crawls long enough you will live it: block rates climb, you upgrade to expensive residential proxies, block rates dip for a week, then climb right back. The proxies weren't the problem. The identity behind them was.

What modern detection actually looks at

Every browser exposes a bundle of measurable characteristics: canvas rendering output, WebGL renderer strings, installed fonts, audio processing quirks, screen geometry, navigator properties, language and timezone, TLS handshake shape, HTTP/2 behaviour. Individually mundane; combined, they identify a browser instance with unsettling precision — the EFF's Cover Your Tracks project has demonstrated for years that a typical browser is unique or near-unique among millions. If the mechanics are new to you, our primer on browser fingerprinting for beginners walks through each signal; the short version is that anti-bot vendors collect these signals on every request and cluster them.

At scale, that clustering is lethal in a way small-scale scrapers never experience. Run 400 workers from one Docker image and every single one presents the identical fingerprint: same headless tells, same rendering hashes, same screen size, same everything. The detection system doesn't need to decide whether your fingerprint looks botty. It just notices that one exact fingerprint made two million requests from 500 different IPs this week. No human population looks like that. Cluster blocked.

The uniformity trap — and the randomness trap

The naive fix is randomizing everything, and it fails in the opposite direction. Fingerprints are internally consistent in real populations: a macOS user agent comes with macOS fonts, Apple GPU strings and macOS-shaped TLS behaviour; a 1366×768 screen doesn't report a 4K viewport; a German residential IP rarely pairs with America/Chicago and en-US. Random per-request noise produces impossible combinations that are easier to flag than uniformity, and a fingerprint that changes between consecutive requests in one session is itself a signature. What you need is a population of fingerprints that are each internally coherent, each stable over time, and collectively diverse — which is precisely what's hard to do by hand, and why we wrote a whole practical guide to changing your browser fingerprint about doing it properly rather than crudely.

Isolated browser profiles at scale

This is the problem antidetect browsers were built to solve, and it's why they've migrated from multi-account marketing into serious scraping stacks. Dual Login's approach maps directly onto the requirements above:

• Each profile is a complete, isolated identity: a generated fingerprint that's internally consistent (canvas, WebGL, fonts, navigator, screen, UA, languages, timezone all agreeing with each other and with the profile's declared OS), its own persistent data directory for cookies and storage, and its own proxy assignment. One profile = one coherent device that stays the same device tomorrow.
• The fingerprint is applied natively in the browser engine, not by injecting JavaScript overrides into pages. That distinction matters at scale: JS-layer spoofing is detectable (patched functions, timing anomalies, coverage gaps in workers and iframes) precisely by the vendors you're trying to satisfy, and it's per-page overhead on every load. Engine-level spoofing is invisible to page scripts and reaches everywhere, including web workers.
• Automation is driven over raw CDP without the classic tells — no navigator.webdriver = true, no automation banners, input events that register as trusted. Your scraper drives the profile through an HTTP API (navigate, click, type, extract, screenshot, network capture), which slots cleanly into the worker architecture described earlier: the frontier leases a URL to a worker, the worker leases a profile from the pool, drives it, writes results, releases both.

Operationally, you treat profiles the way you treat proxies: as a pool of reusable identities. A crawl needing 40 concurrent browser sessions might maintain a pool of 150 profiles, rotating them through work with rest periods, each permanently paired with a sticky IP from a matching geography. Profiles that start attracting challenges get benched or retired; healthy profiles accumulate believable history (cookies, cache, prior visits) that makes them more trustworthy over time — the exact opposite of the fresh-cold-start problem every stock headless container has on every launch.

Logged-In Scraping: Sessions Are Assets, Not Overhead

Some of the most valuable data lives behind logins — marketplace dashboards, member pricing, gated listings. Scraping it at scale inverts your priorities: identities stop being disposable and become capital.

Rules that hold up in practice:

1. One account, one profile, one IP, forever. The account was created in a specific fingerprint from a specific region; it should live there. Every property that silently changes between sessions is a risk signal to the platform.
2. Warm identities before working them. A fresh account that immediately pa'ges through 5,000 listings at machine speed is the easiest possible pattern to catch. Slow ramp, human-plausible session shapes, actual variety in behaviour.
3. Persist the session, don't relogin. Cookie and storage persistence in a per-profile data directory means fewer login events — and repeated logins from anywhere unexpected are one of the loudest triggers on any platform. Dual Login persists cookies, localStorage and IndexedDB per profile, and can carry them between machines, so a session survives infrastructure churn.
4. Rate-limit per identity, hard. The ceiling isn't your fleet's capacity; it's what one plausible human does in a day. Scaling logged-in work means adding identities, not adding requests per identity.
5. Budget for attrition. Some accounts will be lost. Track health per identity and replace steadily rather than in panicked batches.

The operational discipline here overlaps heavily with multi-account management generally — the account-safety practices in our Amazon seller account bans playbook and the multiple Facebook accounts guide are the same discipline applied to different platforms. Whether the goal is running ad accounts or harvesting data, the underlying rule is identical: an identity is a device plus a network plus a behavioural history, and all three must stay consistent.

And the legal/ethical dimension is genuinely different behind a login. Public-page scraping and authenticated scraping sit in different places contractually. Read the terms you agreed to, respect the Robots Exclusion Protocol where it applies, avoid personal data you have no basis to process, and don't degrade a site's service — a crawl that measurably hurts a target is both wrong and self-defeating.

Monitoring: You Cannot Fix What You Don't Graph

Most large crawls that fail did so visibly for days before anyone noticed. Instrument these from day one — not after the first incident.

The metrics that matter

• Block rate per domain per hour. The most important number in the system. Any sustained upward trend is an emergency, even at a low absolute level, because it compounds.
• Soft-block rate. Detect challenge pages, CAPTCHA markup, noindex interstitials and suspiciously short responses on a 200. A per-domain rule that classifies a response as "success but wrong" catches the silent killer. Track content-length distributions too: a bimodal distribution where a chunk of pages are 8KB and the rest 90KB usually means half your responses are challenge pages.
• Parse yield per domain per day. Fetches succeeding while yield collapses = redesign or soft block.
• Per-identity health. Requests served, challenges hit, last success. Retire declining profiles before they take a whole cohort's reputation with them.
• Cost per successful page. The unit-economics number. It rises the moment retries increase, and it rises long before anyone gets around to complaining about the invoice.
• Queue depth and lease age. Depth growing steadily means throughput is below intake; old leases mean workers are dying mid-batch.

Respond gradually, not catastrophically

When block rate rises on a domain, the wrong reaction is to keep hammering (poisoning identities and IPs) and the second-wrong reaction is to stop everything. Build a graduated response: slow the per-domain rate, rotate to a fresher identity cohort, escalate the proxy tier for that domain, and only then pause and alert. Automate the first three; a system that can back off on its own survives weekends.

One more habit: keep a canary set of ~100 known-good URLs per major domain, fetched on a slow schedule through a separate identity pool, with content assertions. When the canary breaks but the main crawl looks fine, you've caught a soft block. When both break, the site changed. That distinction saves hours of misdiagnosis every time.

Choosing Tooling Without Overbuying

The tooling market splits into three groups, and the right answer depends almost entirely on whether you need browsers and how many identities you're maintaining.

HTTP-layer frameworks (Scrapy and similar) are excellent at what the pyramid's base needs: fast, cheap, well-instrumented fetching with mature scheduling and middleware. If most of your crawl can live here, keep it here.

Browser automation libraries (Playwright, Puppeteer) handle rendering, but out of the box they announce themselves. Stealth plugins help against naive checks and lose against serious vendors, because they patch at the JavaScript layer — which is exactly the layer detection scripts inspect. They're fine for JS-rendering on unprotected sites; they're not an identity system.

Antidetect browsers with automation APIs are the layer that provides identity: many isolated profiles, engine-level fingerprint spoofing, per-profile storage and proxy, driven programmatically. If you're maintaining more than a handful of persistent identities, or you're hitting targets with real anti-bot vendors in front of them, this stops being optional.

Within that third category, the practical selection criteria for scraping differ from the marketing-team criteria most reviews use. Prioritize: a stable local automation API (so your workers aren't screen-scraping a GUI), per-profile proxy assignment with sticky mapping, native rather than JS-injected fingerprinting, persistent per-profile storage, honest concurrency on hardware you can afford, and pricing that doesn't punish you for holding hundreds of profiles you only run occasionally. That last one eliminates a surprising number of vendors — per-profile pricing designed for a ten-person marketing team becomes absurd at 500 identities. We compared the field on cost and capability in cheaper Multilogin alternatives that actually work and head-to-head in GoLogin vs AdsPower; for scraping specifically, weight the API and the concurrency-per-dollar far above the profile-management UI.

Hardware and concurrency reality

A real browser profile costs roughly 200–400MB of RAM under load and a meaningful slice of CPU. With low-memory tuning you can run five to eight profiles per 4GB of RAM; without it, three or four. A 64GB machine therefore lands somewhere around 60–100 concurrent profiles depending on page weight — which, at 4 seconds per page, is 15–25 pages/second, or roughly 1.3–2 million pages/day from one box. That is usually far more browser capacity than a well-designed pyramid actually needs, which is the point: get the HTTP/browser split right and your hardware bill stops being interesting.

Request filtering inside browser sessions is the other multiplier. Blocking images, media, fonts and third-party analytics typically cuts page weight 60–80%. Through metered residential proxies, that single setting is often the difference between a viable project and an unaffordable one.

A Sane Rollout Sequence

If you're starting a large crawl now, this order minimizes wasted money:

1. Reconnaissance (days, not hours). For each target: is there a JSON API? Is the HTML server-rendered? What anti-bot vendor sits in front? What does sitemap.xml contain? This is the highest-ROI time you'll spend.
2. Pilot at 1,000 pages per target, single identity, single proxy tier. Measure block rate, latency, page weight, parse yield. These numbers extrapolate; guesses don't.
3. Build the frontier and storage layer before scaling workers. Normalization, dedup, per-domain buckets, raw storage, idempotent writes. Scaling workers on a weak frontier just produces garbage faster.
4. Introduce identity pooling. Provision profiles for the browser tier, pin proxies, set rotation and rest policies. Start with more identities than you think you need and work them lightly.
5. Instrument everything, then scale in steps. 10k → 100k → 1M, checking block rate and cost per successful page at each step. If a metric degrades, stop and fix; degradation never resolves itself at higher volume.
6. Automate the graduated backoff before you go unattended. A system that can't slow itself down will eventually run all weekend into a wall.

Most teams that fail at scale skipped steps 1 and 3 — they scaled workers against unexamined targets on a frontier that couldn't hold, and spent the next three months paying residential-proxy prices to fetch duplicate URLs from sites that would have handed them a JSON API for free.

FAQ

How many proxies do I need to scrape a million pages a month?

It depends almost entirely on the target's tolerance, not on your page count. A million pages a month is ~23 pages/minute sustained. Against a permissive site allowing ~1 request/second per IP, a few dozen datacenter IPs suffice. Against a protected site where each IP can safely do a handful of requests per hour, you need residential rotation measured in bandwidth rather than IP count — budget by GB (page weight × pages × overhead) rather than by IP. Run a 1,000-page pilot per target and extrapolate from measured block rates; every estimate made without a pilot is fiction.

Do I really need an antidetect browser, or is a headless browser with a stealth plugin enough?

Stealth plugins defeat naive checks (the obvious navigator.webdriver flag and friends) and lose against commercial anti-bot vendors, because they patch the JavaScript environment — the exact layer detection scripts examine, where patched natives, timing anomalies and gaps in workers and iframes are all discoverable. They also don't solve the bigger scaling problem: every container in your fleet still shares one fingerprint. If your targets have no serious protection, stealth plugins are fine. If they use Cloudflare, DataDome, Akamai or similar, or if you need many persistent identities, you need engine-level fingerprinting with per-profile isolation.

What's a realistic block rate to aim for?

Under 2% on unprotected targets and under 5–8% on protected ones is healthy at scale. The absolute number matters less than the trend: a stable 6% is a functioning system, while a 2% that has been climbing for a week is a system in decline. Watch the derivative, and always measure soft blocks separately — a 200 response containing a challenge page counts as a block no matter what your HTTP status histogram says.

Should I store raw HTML or parsed data?

Both, in that order. Store the compressed raw response plus fetch metadata in object storage, then parse as a separate stage. Compressed HTML runs 15–30KB per page, so a million pages is roughly 20–30GB — trivially cheap next to what re-fetching costs. When a site redesigns or you find a parser bug, you re-run parsing over stored bytes in minutes instead of paying your entire proxy bill again.

How do I scrape pages that require logging in without getting accounts banned?

Treat each account as a long-lived identity: one account, one browser profile with a stable internally-consistent fingerprint, one sticky IP from a matching region, persistent cookies so you rarely re-authenticate. Rate-limit per identity to what a plausible human does, warm new accounts gradually before working them, and scale by adding identities rather than by pushing more requests through existing ones. Track per-account health and retire declining ones early — a burned account is cheaper to replace than a burned cohort.

Is large-scale web scraping legal?

It depends on jurisdiction, what data you collect, and how you obtain it — this isn't legal advice. In broad terms, collecting publicly accessible non-personal data sits on far safer ground than scraping behind a login you agreed to terms for, or collecting personal data without a lawful basis under regimes like GDPR. Respect robots.txt where it applies, don't degrade the service you're crawling, keep authenticated scraping within the terms you accepted, and get real legal advice before building a business on a dataset whose provenance you can't defend.

Wrapping Up

Scaling web scraping to millions of pages is mostly an exercise in refusing to solve the wrong problem. The instinct is to add machines; the reality is that the constraints are almost always the target's tolerance, the coherence of the identities you present, and the cost per successful page. Get the HTTP/browser split right and hardware stops mattering. Get the frontier right and you stop paying to fetch the same page repeatedly. Get identity right — coherent fingerprints, sticky proxies, isolated persistent profiles — and the block rate that was quietly strangling your crawl flattens out.

That last piece is the one teams reach for last and should reach for earlier, because it's the one that makes everything upstream worth doing. If your crawl needs many browser identities that stay coherent over weeks rather than minutes, Dual Login gives you isolated profiles with native engine-level fingerprinting, per-profile proxies and storage, and an automation API your workers can drive directly. Spin up a handful of profiles, point them at the target that's been blocking you, and compare the block rate against whatever you're running now — a small pilot answers the question faster than any comparison table can. If you're weighing options, our notes on what to test during an antidetect browser free trial are a reasonable checklist to steal.