Scraping JavaScript-Heavy Websites at Scale in 2026

Somewhere around 2019, the easy web quietly disappeared. You used to be able to point requests and BeautifulSoup at almost anything, parse the HTML, and go home early. Then the frontend world moved to React, Vue, Next.js and their cousins, and the HTML that arrives over the wire stopped containing the data. Open the page source of a modern e-commerce list, a real-estate portal, a flight aggregator, a job board — you'll find a <div id="root"></div>, a few hundred kilobytes of bundled JavaScript, and nothing you can grep.

That's the first wall. The second wall is worse. Once you start driving real browsers to render those pages, you stop being an anonymous HTTP client and become a device — one with a fingerprint, a canvas hash, a WebGL vendor string, a font list, a timezone, and a very specific way of moving the mouse. Anti-bot vendors got very good at reading those. So the problem of scraping javascript heavy websites at scale is really two problems wearing one coat: rendering the page, and not looking like one machine pretending to be ten thousand people.

This guide is about both. It's written for people who have already hit the wall — whose Playwright cluster works beautifully for forty minutes and then starts returning 403s, empty arrays, or CAPTCHA walls forever.

Why JavaScript-Heavy Sites Break Normal Scrapers

Before you pick a tool, it helps to be precise about what is breaking. "The page needs JavaScript" covers at least four different failure modes, and they have different cheapest fixes.

1. Client-side rendering (CSR)

The server sends a shell. The bundle boots, calls an API, and paints the data. Your HTTP client gets the shell and sees nothing. This is the classic case and, ironically, often the easiest to solve without a browser at all — because there's an API underneath (more on that in a moment).

2. Hydration and streamed server rendering

Next.js, Remix, Nuxt and friends often do send HTML. But the useful parts arrive in a <script id="__NEXT_DATA__" type="application/json"> blob, or as streamed RSC payloads appended after the initial flush. Fetch too early, or with a client that doesn't handle chunked streaming properly, and you get half a page. The data is there; you're just reading it wrong.

3. Interaction-gated content

Infinite scroll. "Load 24 more." Tabs that fetch on click. Filters that live entirely in client state. Price that only resolves after a variant is selected. No amount of clever HTTP will get this — something has to actually scroll and click.

4. Anti-bot execution challenges

The page ships a JavaScript challenge that must execute correctly, in a real engine, with a real DOM, and produce a token that gates every subsequent request. Cloudflare's managed challenges, Akamai's sensor data, DataDome's payloads, PerimeterX — these aren't rendering problems, they're proof-of-being-a-browser problems. They are the reason "just use a headless browser" stopped being sufficient advice around 2021.

Most real targets are a mix. A retailer might be Next.js hydrated (case 2) with infinite-scroll categories (case 3) behind a bot-management edge (case 4). Diagnose all four before you build.

The Cheapest Thing That Works: Find the API First

I want to say this loudly because it saves more money than any other advice in this article: a huge fraction of JavaScript-heavy sites are easier to scrape than static ones, because the data arrives as clean JSON.

A browser is expensive. A rendered page costs roughly 200–600 MB of RAM at peak, 0.5–3 seconds of CPU-bound work, and 1–5 MB of bandwidth. An HTTP call to the same underlying endpoint costs a few kilobytes and 80 milliseconds. At a thousand pages that's a rounding error. At ten million pages it's the difference between a laptop and a small data centre.

So spend the first two hours in DevTools, not in code.

How to actually find the endpoint

Open the Network tab, filter to Fetch/XHR, and reload. Then:

• Sort by size. The response holding your data is usually the largest JSON on the page.
• Look for __NEXT_DATA__ or self.__next_f in the document response. Next.js sites frequently embed the entire props payload — including fields the UI never displays. One json.loads and you're done.
• Check for GraphQL. A single POST /graphql with a persisted-query hash is very common. Grab the hash and variables, and you can often request far more per call than the UI does (bigger first: values, extra fields) — though be careful: unusual query shapes are exactly what server-side anomaly detection watches for.
• Try removing headers one at a time. Copy the request as cURL, then strip. Very often only User-Agent, one API key header and a cookie are load-bearing. If nothing is required, congratulations, you have a public API.
• Look for pagination that the UI hides. ?limit=24 frequently accepts ?limit=200. Sometimes ?limit=1000. Test politely.

The cases where this fails: the endpoint is signed with a token minted by obfuscated client-side JS; the token is bound to a bot-management cookie that only a real browser can obtain; or the payload requires a device fingerprint hash computed in a Web Worker. That's when you need a browser — not for rendering, but for credentialing.

The hybrid pattern (this is the one to build)

The architecture that actually scales for JS-heavy targets is a two-tier pipeline:

1. A small pool of real browser profiles whose only job is to solve the challenge, obtain cookies/tokens, and keep them warm.
2. A large pool of cheap HTTP workers that reuse those credentials to hit the JSON endpoints directly, at high volume.

One browser can credential hundreds or thousands of HTTP requests before its session ages out. The ratio is your entire cost model. When someone tells me they need 500 concurrent browsers, my first question is always: do you need 500 browsers, or do you need 12 browsers and 500 HTTP workers sharing their cookie jars?

The catch — and it's a real one — is that the credential must stay consistent with the requests that use it. If a token was issued to a browser fingerprint claiming to be Chrome 141 on Windows 11 from a São Paulo residential IP, then your HTTP workers reusing that token had better send Chrome 141 headers, in Chrome's exact header order, over that same IP. Mismatch is a louder signal than either half alone. This is where per-profile isolation stops being a nice-to-have.

When You Genuinely Need a Real Browser

Sometimes there's no way around rendering. Interaction-gated content, canvas-drawn charts, PDF-in-viewer, sites where the meaningful state only exists after four clicks, or challenge flows that demand real execution. Fine. Then the question becomes which browser, and how honestly it presents itself.

Headless is a fingerprint, not a mode

Old headless Chrome was trivially detectable — navigator.webdriver, a missing chrome object, no plugins, a distinctive HeadlessChrome token in the UA. The new headless mode shipped in Chrome 112 closed most of the obvious gaps by running the same code path as headful (Chrome's own writeup on the new headless mode is worth reading). But "most" is doing a lot of work in that sentence.

What still leaks, in my experience:

• GPU and WebGL. Headless on a GPU-less server falls back to SwiftShader. WEBGL_debug_renderer_info then returns something like Google SwiftShader or ANGLE (Google, Vulkan 1.3.0 (SwiftShader Device...)). Roughly zero real consumer machines report that. It's one of the highest-signal tells in existence, and it's why software rendering on a VPS is such a common silent killer.
• Font enumeration. A minimal Linux container has a dozen fonts. Real Windows machines have 200+, including a very predictable core set. Font-metric probing is cheap and stable.
• Missing platform APIs. Battery status, media devices, navigator.plugins shape, permission query results, WebGL2RenderingContext extension lists, audio context DSP fingerprints.
• The automation control channel itself. CDP attachment is observable. Some detectors watch for the timing artifacts of an attached debugger, or for Runtime.enable side effects on the page — enabling the Runtime domain has measurable consequences that scripts can notice.

That last point is the one most people get wrong. It is not enough to patch navigator.webdriver with a JS override; a detector can check whether the property descriptor looks native, whether toString has been tampered with, and whether the page's own error stack traces contain injected frames.

Where automation frameworks help and where they hurt

Playwright and Puppeteer are excellent engineering tools. They were built for testing your own site, not for looking like a stranger's user. Their defaults are honest about being automation, because honesty is a feature in CI. Stealth plugins patch the visible symptoms and lag behind detection vendors by weeks to months.

The more durable approach is to change the browser, not to lie from inside it. If the engine itself reports a plausible GPU string, a plausible font list, a plausible canvas hash and a plausible audio fingerprint — at the C++ level, before any JavaScript runs — then there's nothing for a page script to catch out. No injected function to inspect, no toString mismatch, no property descriptor that smells wrong. That's the architectural bet Dual Login makes: fingerprints are applied natively by the engine, and the driving layer avoids the observable CDP domains entirely, using only DOM/Input/Page/Network so clicks and keystrokes are trusted events rather than synthesized ones.

If you want the underlying mechanics of what's being measured, Browser Fingerprinting Explained for Beginners walks through the surfaces one by one, and How to Change Browser Fingerprint covers what actually changes versus what merely appears to.

The Consistency Problem Nobody Warns You About

Here's the failure I've watched teams hit most often, and it has nothing to do with rendering.

You spin up 200 workers. Each gets a random User-Agent from a list, a random screen size, a random proxy. It feels like diversity. It is, in fact, a giant flashing sign — because you've created 200 devices that cannot exist.

A real device is internally consistent in ways you don't think about:

Spoof one field and you break three relationships. A UA saying Windows with a WebGL renderer saying Apple GPU is a stronger bot signal than the default headless fingerprint you were trying to hide. A device in Kyiv by IP, America/New_York by timezone, and de-DE by language is not a person; it's a misconfiguration, and misconfiguration is a classifier's favourite food.

This is why coherent profiles beat randomised fields. Each profile in Dual Login carries one internally consistent identity — canvas, WebGL, audio, fonts, navigator, screen, UA, timezone, geolocation, languages — plus its own persistent data directory (cookies, localStorage, IndexedDB, cache) and its own proxy. It launches as a separate OS process, so there's no shared state to leak between identities. When you want fifty workers, you want fifty devices, not one device with fifty hats.

The same logic that keeps multiple accounts from being linked applies exactly to scraping fleets — How Websites Detect Multiple Accounts on the Same Device covers the linkage signals in detail, and they're the same ones that cluster your scrapers.

Proxies: Where Most Scaling Budgets Die

You can have a perfect fingerprint and still get blocked in ten minutes if your IPs are wrong. Conversely, plenty of mediocre scrapers survive on excellent residential IPs. Proxy quality is usually the dominant term in the equation.

Pick the pool for the target, not the price

Datacenter IPs are fast, cheap (often $0.50–$2/GB or flat per-IP), and stable. They're also trivially identifiable by ASN. For unprotected APIs, public data, documentation sites, or targets that simply don't care — datacenter is correct and anything else is wasting money. Do not pay residential rates to scrape a government open-data portal.

Residential IPs come from real consumer ISPs. They cost $2–$12/GB, they're slower (100–800 ms added latency is normal), less stable, and sometimes the exit is somebody's phone on a train. But they carry consumer ASN reputation, which is exactly what heavily-protected commercial targets grade on.

Mobile IPs are carrier-NAT'd, meaning thousands of real users share one address. That makes them extremely hard to block wholesale — a site that bans a mobile IP bans a chunk of a city. They cost the most ($8–$30/GB) and you use them for the hardest targets or where a genuine mobile context matters.

ISP/static residential sits in between: residential ASN, datacenter hosting, so you get reputation plus stability. Excellent when you need a persistent identity — a logged-in session that must look like it lives in one house.

Rotation strategy matters more than pool size

The common mistake is rotating on every request. It feels safe. It isn't, for two reasons: it destroys session continuity (your cookies now come from thirty countries), and it produces a traffic pattern no human generates.

Better patterns:

• Sticky-per-profile. One profile keeps one IP for its whole working session — minutes to hours. This is the default you should reach for. A browser identity that teleports mid-session is broken by construction.
• Sticky-per-task. Rotate between logical units of work (one category tree, one search-result set), not between requests.
• Rotate on signal, not on schedule. Watch for 403/429, challenge pages, sudden empty results, or response-size collapse. Burn the IP then. Rotating a healthy IP is throwing away reputation you paid for.
• Geo-lock to the content. Scraping amazon.de from a Vietnamese exit gets you a different site, different prices, and a suspicious profile. Match the exit country to the market you're reading.

Budget honestly. Rendered pages are heavy: a modern e-commerce page is easily 3–6 MB with images. At $5/GB residential, that's roughly $0.02 per page — $200 for 10,000 pages. Block images, fonts and media at the request-interception layer and you'll typically cut 70–85% of that. This one change has saved clients more money than any other optimisation I've made. Then push as much volume as possible down to the JSON tier, where a page costs kilobytes.

For the deeper mechanics of pairing identities to exits — and the traps around DNS leaks and WebRTC — see Antidetect Browser with Residential Proxies. And if you're wondering whether a VPN could substitute here: it can't, and Antidetect Browser vs VPN Difference explains why one IP shared by all your workers is worse than no proxy at all.

Architecting the Fleet: Real Concurrency Math

"At scale" needs a number. Let's do the arithmetic that determines your infrastructure.

Sizing from the target, backwards

Start with what you need, not what you can build:

• Pages needed per day: say 500,000.
• Fraction that genuinely needs a browser: after API discovery, maybe 5% (25,000). The rest goes to HTTP workers.
• Seconds per rendered page: 4 s realistically, including navigation, waiting for the network to settle, scrolling, and extraction.
• Effective browser-seconds needed: 25,000 × 4 = 100,000 s/day.
• Seconds available per browser slot: 86,400 × 0.6 utilisation ≈ 52,000 s.
• Concurrent browsers required: ~2. Not 200.

That number will shock people who've been running 100-instance clusters. It's the point: most "we need massive concurrency" problems are actually "we're rendering things we didn't need to render" problems.

RAM is the binding constraint

When you do need concurrent browsers, memory decides density, not CPU. Practical figures:

Site isolation off is a genuine security trade-off — you're removing a process boundary between origins. For a dedicated scraping box visiting known targets, that's usually acceptable. For a machine that also holds your logged-in accounts, think harder. Dual Login ships a low-RAM mode on by default that does exactly this (capped V8 heap, reduced caches, site isolation relaxed) while deliberately keeping the GPU on — because turning the GPU off is what forces SwiftShader and hands the detector its best signal.

Queue discipline, not thread counts

Don't spawn a browser per URL. Build:

• A durable URL frontier (Redis, Postgres, SQS) with per-domain politeness and priority.
• Long-lived workers that lease URLs, so browser startup (1–3 s) amortises across hundreds of pages.
• Per-domain rate limiting at the queue level. Global concurrency of 50 across 50 domains is polite; the same 50 against one domain is an attack.
• Idempotent writes. Every page will be scraped more than once. Key on a content hash or canonical URL.
• Dead-letter with reason codes. "Failed" is useless. CHALLENGE_PRESENTED, SELECTOR_MISSING, EMPTY_PAYLOAD, PROXY_TIMEOUT, RATE_LIMITED each demand a different response.

That last one deserves emphasis. When output quality drops, you need to know which wall you hit within minutes. Teams without reason codes spend days guessing whether their extractor broke or the site changed its defences.

Recycle sessions on purpose

A profile that has been running for six hours has accumulated cookies, cache, a long history, and possibly a raised risk score. Define a lifecycle: warm up (a few normal pages), work (N requests or M minutes), then either rest or retire. Reset the data directory when you retire a profile so the next identity starts genuinely fresh — not with the previous identity's IndexedDB still on disk. Per-profile persistent data directories make this a file operation rather than an archaeology project.

Extraction That Doesn't Break Every Tuesday

Getting the page is half the job. Getting data out of it, repeatedly, for months, is where maintenance cost lives.

Selector strategy, ranked

1. Embedded JSON (__NEXT_DATA__, application/ld+json, inline state). Most stable by a mile — it's the site's own data contract, and it changes when the backend changes, not when a designer moves a button.
2. Structured data / microdata. Sites maintain schema.org markup for Google's benefit, which means they maintain it for yours. Broadly documented on schema.org and worth checking before writing a single CSS selector.
3. Stable attributes — data-testid, data-product-id, itemprop, ARIA roles. Test IDs survive redesigns because the site's own tests depend on them.
4. Semantic + text anchors. "The <dd> following the <dt> containing 'Model'." Verbose, surprisingly durable.
5. CSS class chains. div.sc-1x2y3z > span:nth-child(3). These are generated by CSS-in-JS and will change on the next deploy. Use as a last resort and expect breakage.

Wait for data, never for time

sleep(5) is both slow and unreliable. Wait for the condition that matters: the specific element, the specific network response, or a predicate on the page's own state. networkidle is a decent fallback but it lies on pages with polling, analytics beacons or open WebSockets — which is most pages now. Prefer "the response to /api/products arrived and had ≥1 item."

Validate at write time

Build a schema check into the pipeline and alarm on drift:

• Field fill rates. If price was 99.4% populated yesterday and 61% today, something changed — even though nothing threw an exception.
• Type and range sanity. Prices as strings. Prices of 0. Dates in 1970.
• Volume deltas. A category that returned 4,200 items and now returns 24 means you're being served a stub, or you got soft-blocked.

Silent degradation is the real enemy. A crash wakes you up. A scraper that quietly returns plausible-looking but wrong data for three weeks poisons whatever decisions you built on it. Some anti-bot systems deliberately serve subtly wrong data to suspected bots rather than blocking them — because a blocked scraper gets fixed and a poisoned one doesn't. Fill-rate monitoring is your only defence.

Respect the boring stuff

Read robots.txt and the site's terms. Honour crawl-delay. Send a real, identifiable UA when the target permits it. Cache aggressively so you don't re-fetch unchanged pages — conditional requests with If-Modified-Since cost almost nothing and a 304 is free. MDN's HTTP caching documentation is the reference. Legality varies enormously by jurisdiction, by whether data is public, and by whether you're bypassing authentication; the hiQ v. LinkedIn line of cases is a useful starting point but not legal advice, and it is not a global rule. Get counsel for anything commercial.

Beyond law, there's craft: a scraper that hammers a small business's checkout endpoint at 3 a.m. is a bad neighbour regardless of what's legal. Concurrency ceilings per domain, honest backoff, and off-peak scheduling cost you almost nothing and keep the whole ecosystem tolerable.

Handling Anti-Bot Challenges Without Fooling Yourself

When you hit a managed challenge, there are only four honest responses.

First, avoid it. Is the data available via a public API, a sitemap, an RSS feed, a bulk export, a partner feed, or a licensed data provider? Astonishingly often, yes. An afternoon of looking beats a permanent arms race.

Second, present a genuinely clean browser. Most challenge failures aren't sophisticated defeats — they're the obvious stuff. SwiftShader instead of a real GPU. Twelve fonts. A UA/platform mismatch. A datacenter IP on a consumer-facing retail site. Timezone contradicting the exit country. Fix those five and a large share of "unsolvable" targets simply open. This is unglamorous and it's where the wins are.

Third, behave plausibly. Real users move the mouse in curves with variable velocity, overshoot targets slightly, scroll in irregular bursts, pause to read, and occasionally hit the back button. Perfectly linear 200 ms-interval clicks at exact element centres are a signature. Trusted input events generated at the browser layer (rather than JS-dispatched MouseEvents, which carry isTrusted: false) matter here — a synthesized click is visible to any listener that bothers to check.

Fourth, know when to stop. Some targets have made a deliberate, well-funded decision to be unscrapable. Recognising that in week one instead of month six is a professional skill. Buy the data, partner, or change the question you're asking.

What I'd steer away from: stacking six stealth plugins and hoping. Each patch is a fingerprint of its own. Detection vendors test against the popular plugins — that's literally their job — so a heavily-patched browser can end up more identifiable than a plain one. Depth beats layering.

Putting It Together: A Reference Pipeline

Here's the shape I'd build today for a JS-heavy target at meaningful volume.

Discovery (once, by hand). Map the site. Find every JSON endpoint. Determine what genuinely requires rendering. Document the auth/token flow. Note challenge type and trigger conditions. This phase decides your cost structure for the next year — do not rush it.

Credential tier (small, browser-based). A handful of isolated profiles, each with a coherent fingerprint and a sticky residential IP matched to the target's market. Their job: navigate normally, pass the challenge, harvest cookies and tokens, keep them warm with occasional real page loads. Persistent data directories mean these sessions survive restarts, which matters more than it sounds — a warm session with two weeks of history is worth far more than a fresh one.

Bulk tier (large, HTTP-based). Hundreds of lightweight workers consuming credentials from the credential tier, hitting JSON endpoints, respecting per-domain rate limits. Header sets must exactly match the browser that minted the credential — same UA, same Client Hints, same header order, same TLS profile if you can manage it, and crucially the same egress IP.

Render tier (medium, browser-based). For the pages that truly need interaction. Images/fonts/media blocked. Low-RAM mode. Workers leased from a queue with per-domain politeness. Screenshot on failure — you will need to see what the page actually looked like when the selector missed.

Storage and validation. Raw responses to object storage (cheap, and re-parseable when you find a bug six months later). Parsed records to your database with schema validation. Fill-rate and volume metrics on a dashboard with alerts.

Observability. Per-domain success rate, challenge rate, median latency, bytes per record, cost per thousand records. When something degrades, you want a graph that tells you which tier, which domain, and which reason code within about five minutes.

That last metric — cost per thousand records — is the one to put on the wall. It ties fingerprint quality, proxy choice, render ratio and extraction efficiency into a single number that a non-engineer can reason about. Every optimisation in this article is ultimately an attempt to move it down.

Where Dual Login Fits

Dual Login is a browser-profile manager, not a scraping framework, and it's worth being precise about the division of labour. It handles the identity layer: many isolated profiles, each with its own natively-applied fingerprint, its own persistent data directory, its own proxy. Each profile launches as a real OS process, so isolation is enforced by the operating system rather than by convention.

For scraping work specifically, the pieces that matter are:

• Native fingerprint application. Canvas, WebGL, audio, fonts, navigator, screen, UA, timezone and geolocation are set by the engine before any page script runs, so there's no injected JavaScript for a detector to notice — and the spoofing reaches Web Workers, where JS-injection approaches typically don't.
• A stealth launch path with no held CDP client. Profiles run without an attached debugger by default, because a persistent CDP attachment is itself observable.
• Runtime-free automation. The driving layer uses only the DOM, Input, Page and Network CDP domains — never Runtime.enable — so clicks and keystrokes are trusted events and navigator.webdriver stays false.
• A local HTTP API per profile. Navigate, click, type, wait, screenshot, capture network traffic, export cookies. Which means your existing orchestrator — Python, Node, C#, whatever — drives profiles over plain HTTP without adopting a new framework.
• Portable sessions. Cookies and localStorage are captured continuously and travel with the profile, so a warm credentialing session isn't lost to a restart or a machine migration.

If you're weighing it against the incumbents, GoLogin vs AdsPower covers how those two compare, and Cheaper Multilogin Alternatives That Actually Work is the honest per-profile cost breakdown — which matters a lot when a scraping fleet means dozens or hundreds of profiles rather than five.

FAQ

Can I scrape JavaScript-heavy sites without a headless browser at all?

Often, yes — and you should try before reaching for one. Most client-rendered sites fetch their data from a JSON API you can call directly, and framework sites frequently embed the whole payload in a __NEXT_DATA__ script tag. Spend an hour in DevTools first. You'll need a browser only when the endpoint is signed by obfuscated client code, gated behind a bot-management token, or when the content genuinely requires interaction like infinite scroll or multi-step filtering. Even then, use browsers to obtain credentials and HTTP workers to do the volume.

How many concurrent browsers do I actually need for large-scale scraping?

Far fewer than most people assume. Work backwards: pages needed per day, times seconds per rendered page, divided by realistic per-slot uptime. A workload of 500,000 pages/day where only 5% need rendering comes out at roughly two concurrent browsers plus a fleet of HTTP workers. RAM is the binding constraint when you do need concurrency — budget 180–300 MB per headless instance with images and fonts blocked, and 400–700 MB for full headful.

Why does my scraper work for an hour and then get blocked?

Three usual suspects, in order of likelihood. First, IP reputation: you're burning through a datacenter range and the ASN got flagged. Second, rate pattern — perfectly regular request intervals with no variance and no human pauses. Third, fingerprint clustering: all your workers share one identity, so blocking one blocks all. The tell is which changes when you fix each: a new IP that works immediately points at proxies; a new IP that fails instantly points at your fingerprint, because the site recognised the device, not the address.

Do residential proxies alone solve scraping at scale?

No, though they solve a large share of it. Residential IPs give you consumer ASN reputation, which is what most commercial anti-bot systems grade on first. But an excellent IP paired with a headless browser reporting SwiftShader as its GPU and twelve installed fonts still gets flagged — the IP passes and the device fails. Conversely, a perfect fingerprint on a known datacenter range fails at the edge before any JavaScript runs. You need both layers, and they must agree with each other: timezone matching the exit country, language matching the region, geolocation matching the IP.

Is it legal to scrape JavaScript-heavy websites?

It depends heavily on jurisdiction, what data you're collecting, whether it's publicly accessible, whether you're bypassing authentication or technical protections, and what you do with the results afterwards. Scraping public data has been treated more favourably in some US cases than scraping behind a login, but that's not a global rule and it's not legal advice. Personal data brings GDPR and similar regimes into play regardless of accessibility. Read the terms of service, honour robots.txt, don't overload infrastructure, and get real legal counsel before anything commercial.

What's the single biggest cost saver when scraping at scale?

Blocking images, fonts, media and third-party trackers at the request-interception layer. A modern e-commerce page is 3–6 MB, and 70–85% of that is assets you never parse. On residential proxies at $5/GB, that turns roughly $0.02 per page into well under half a cent. The second biggest is shifting volume from the render tier to the HTTP tier — a JSON call costs kilobytes where a rendered page costs megabytes, and the ratio between those two tiers is essentially your entire infrastructure bill.

Wrapping Up

The hard part of scraping javascript heavy websites at scale was never the JavaScript. Rendering a page is a solved problem — several excellent open-source projects do it well. The hard part is doing it ten thousand times without your fleet collapsing into one recognisable machine, and doing it cheaply enough that the data is worth what it cost.

That comes down to a few unglamorous disciplines. Find the API before you launch a browser. Render only what genuinely needs rendering. Give every worker a coherent identity instead of a bag of random fields. Match your proxies to your fingerprints, and both to the market you're reading. Block the bytes you'll never parse. Monitor fill rates, because silence is not success. And build your extractors against the site's own data contracts rather than its CSS.

Get those right and the concurrency numbers you need turn out to be surprisingly small — which is the whole point. Scale isn't a bigger cluster; it's a pipeline that doesn't waste anything.

If the identity layer is what's holding you back — coherent per-profile fingerprints, isolated data directories, per-profile proxies, and a local HTTP API your existing orchestrator can drive without adopting a new framework — that's the part Dual Login is built for. Spin up a couple of profiles, point them at your hardest target, and see how far a genuinely clean browser gets you before you build anything bigger.