Web Scraping Rate Limiting Best Practices That Actually Scale

Most scraping projects don't die because the parser broke. They die because the operator treated the target site like a static file server: hundreds of requests per second from a single IP, identical headers on every hit, zero reaction to the first 429. Two hours later the whole subnet is blocked, the session cookies are burned, and the 'quick data pull' has turned into a week of firefighting.

Rate limiting is where scraping projects are won or lost. Not parsing. Not even proxies. Pacing. Get the pacing right and everything downstream gets easier — fewer CAPTCHAs, fewer poisoned responses, longer-lived sessions, smaller proxy bills. Get it wrong and no amount of infrastructure spend will compensate, because every block event makes the next one more likely.

This guide walks through web scraping rate limiting best practices from the perspective of someone who has to keep crawlers alive for months, not minutes: how detection actually works in 2026, how to build request budgets that hold up, what each block signal is really telling you, and why the browser identity layer — the part most rate-limiting guides skip entirely — quietly decides how much throughput you can sustain.

Why rate limits exist (and why brute-forcing them fails)

Sites throttle traffic for three distinct reasons, and it pays to know which one you're up against, because the right response differs for each.

The first is infrastructure protection. A mid-sized e-commerce site might comfortably serve 200 requests per second across all visitors. One careless scraper can double that load on its own. Sites in this category tend to be honest about their limits: you'll get a clean HTTP 429 Too Many Requests, often with a Retry-After header telling you exactly how long to wait. Respect it and you can usually coexist indefinitely.

The second is abuse prevention. Credential stuffing, coupon farming, inventory sniping — the anti-bot vendor doesn't know you only want product prices. You inherit defenses built for far worse actors, which means behavioral scoring, JavaScript challenges and fingerprint checks on top of raw request counting.

The third is that the data itself is the product. Airlines, ticketing platforms, real-estate portals and marketplaces treat their listings as a competitive asset. These sites are the least honest about blocking: instead of a 429 you get silently degraded results, shadow bans, or — nastiest of all — subtly wrong data served only to suspected bots. If you're scraping a data-is-the-product site and your numbers look plausible but slightly off, suspect poisoning before you suspect your parser.

Brute force fails against all three, but for different reasons. Against the first, you simply take the site down or get firewalled at the network edge. Against the second and third, aggression compounds: each block event raises your traffic profile's risk score, which lowers the threshold for the next block, which produces more failed requests, which — if your scraper retries naively — produces even more traffic. Plenty of scraping operations have effectively DDoS'd themselves into a permanent ban this way. The way out is never more requests. It's smarter ones.

How sites decide you're scraping too fast

'Too fast' is not an absolute number. Modern anti-bot stacks — Cloudflare, Akamai, DataDome, HUMAN — score traffic across several layers at once, and your sustainable request rate is a function of how trustworthy the rest of your traffic looks. The same 1,000 pages per hour can be invisible from one setup and instantly fatal from another.

Layer 1: the network

The oldest signal is still the loudest: requests per IP per time window, extended to per-subnet and per-ASN counting. Datacenter IP ranges are catalogued and pre-penalized — traffic from an AWS or Hetzner address starts with a worse score than a residential connection before it sends a single request. This is why serious operations lean on residential or mobile proxies, a topic we covered in depth in the antidetect browser with residential proxies playbook.

Layer 2: protocol and headers

Before your request body is even considered, the connection itself is fingerprinted. TLS handshake characteristics (the JA3/JA4 family of hashes), HTTP/2 settings frames, header ordering, the presence or absence of Accept-Language and Sec-CH-UA headers — all of these differ between a real Chrome and a Python script pretending to be one. A requests-library scraper with a copied User-Agent string fails this layer instantly, no matter how slowly it crawls. Rate limits for traffic that fails protocol checks are effectively zero.

Layer 3: the browser fingerprint

If the site runs JavaScript checks — and every protected site does now — it reads canvas rendering output, WebGL renderer strings, installed fonts, screen geometry, audio stack quirks and dozens of other properties, then hashes them into a device identity. If you're new to how that works, our browser fingerprinting explainer covers the mechanics. The part that matters for rate limiting: when fifty of your 'different users' on fifty different IPs share one canvas hash and one WebGL renderer, the anti-bot system has a stable key to aggregate them. Your fifty polite sessions become one very impolite user, and the limit is enforced against the fingerprint, not the IP.

Layer 4: behavior over time

Finally, the temporal shape of your traffic. Requests arriving at metronomic intervals. Sessions that fetch HTML but never load images, CSS or fonts. Crawls that walk product IDs in perfect numeric order. Sessions that run for nine hours without a single idle gap. Accounts that are only ever active between 03:00 and 05:00. None of these is damning alone; together they form a signature that no delay setting can hide.

The takeaway that shapes everything below: your request rate is evaluated per identity, and identity is a composite of network, protocol, fingerprint and behavior. Treating pacing, proxies and browser identity as separate problems is the root mistake most scraping teams make.

The core best practices

1. Budget concurrency per domain and per identity, not globally

A global 'max 100 concurrent requests' setting is meaningless — 100 spread over 100 domains is gentle, while 100 against one domain is an assault. The unit that matters is the (domain, identity) pair.

The cleanest implementation is a token bucket per pair: tokens refill at your target rate, each request consumes one, and an empty bucket means the request waits. It naturally allows small human-like bursts while capping the sustained average, which is exactly the shape real browsing has.

For starting numbers, err low and let data raise them. Against a protected commercial site, one concurrent request and a 2–5 second average gap per identity is a sane opening position. Against a small independent site, be gentler still — you can crash a hobby server at rates a marketplace wouldn't notice, and crashing the source is both rude and self-defeating. Against large properties you can scale total throughput almost arbitrarily, but by adding identities, not by making each identity faster. Ten sessions at six requests per minute survive; one session at sixty per minute does not.

2. Honor Retry-After, and back off exponentially with jitter

When a server sends a 429 with a Retry-After header, it is handing you the exact number that keeps you unblocked. Parse it, wait it out, and treat it as a signal to lower your steady-state rate — a server that had to say 'slow down' once will say it again sooner next time.

When there's no Retry-After, use exponential backoff with full jitter: on attempt n, sleep a random duration between zero and base × 2ⁿ, capped at a sensible ceiling like five minutes. The jitter part is not optional. If twenty of your workers hit a limit at the same moment and all retry after exactly 60 seconds, they arrive together as a synchronized burst — the classic thundering-herd pattern — and get blocked together again. Randomization desynchronizes them.

Pair backoff with a circuit breaker: after four or five consecutive failures against a domain, stop sending entirely for fifteen to thirty minutes. Requests fired into an active block don't just fail — they document your persistence for the anti-bot system and deepen the hole.

3. Read robots.txt — even though you're not a search engine

Whether robots.txt binds you is a policy question; why you should read it anyway is purely practical. Google's robts.txt documentation explains the syntax, and any Crawl-delay directive tells you what pacing the site's own operators consider acceptable. Disallowed paths are frequently honeypots or expensive endpoints — the exact URLs whose traffic gets flagged fastest. Reading robots.txt costs one request and tells you where the mines are.

4. Cache aggressively and fetch conditionally

The cheapest way to respect a rate limit is to need fewer requests. Store ETag and Last-Modified values, send If-None-Match and If-Modified-Since on refetch, and treat a 304 Not Modified as a win: no body transfer, negligible server load, and on many stacks it doesn't count against the limit at all.

Think hard about refresh frequency per field, too. Product titles and descriptions change monthly. Prices change daily. Stock status changes hourly. Scraping the entire catalog every hour to catch stock changes is throwing away 95% of your budget on static fields. Split your crawl into tiers by volatility and you'll often cut request volume by an order of magnitude — which, in turn, means you can afford to be slower and safer on the requests that remain.

5. Randomize timing without faking randomness badly

sleep(random.uniform(1, 3)) is better than sleep(2), but a uniform distribution is itself a fingerprint — human inter-request gaps aren't uniform, they're long-tailed. Most actions come quickly; occasionally someone reads for four minutes or wanders off for a coffee.

A log-normal distribution models this far better. Add structural realism on top: pauses that scale with page length, occasional back-navigation, an idle gap every twenty or thirty pages, and sessions that end after a plausible number of pages rather than running for eleven straight hours. Whole-day rhythm matters too — an identity that only ever wakes at 4am and hammers for two hours looks nothing like the human it claims to be.

6. Distribute across identities the right way

Here's where most operations go wrong. They buy a large proxy pool, rotate the IP on every request, and assume they've solved rate limiting. What they've actually built is a session that changes country three times mid-checkout — behavior no real user exhibits, and a strong signal in itself.

The correct model is the sticky identity: one persistent browser profile, bound to one proxy, with one consistent fingerprint, one cookie jar and one behavioral pattern, for the entire lifetime of that identity. You scale throughput by running more identities in parallel, not by shuffling attributes within one.

That's precisely the shape Dual Login is built around: each profile gets its own persistent data directory, its own natively-applied fingerprint, and its own proxy binding, and each runs as a genuinely separate browser process. Twenty profiles crawling at a human pace is twenty ordinary visitors. One profile making twenty times the requests is one obvious bot — even if the total request count is identical.

7. Match your tool to the target

Not every target needs a full browser. HTTP clients are dramatically cheaper — roughly 5–20 MB of RAM per worker versus 150–400 MB for a browser instance — and for a static site with no JavaScript challenge, a well-configured HTTP client with correct header ordering is the right answer.

The moment the target runs fingerprinting JavaScript, though, the calculus flips completely. No amount of header spoofing survives a canvas check. Trying to reverse-engineer a challenge script that ships obfuscated and changes weekly is a treadmill; running a real browser that passes the check because it genuinely is a browser is not.

The bottom two rows are where most people underestimate the problem. Once you're logged in, rate limiting stops being about request counts and becomes about account safety — the same discipline as avoiding Amazon seller account bans. One bad hour doesn't cost you a crawl, it costs you the account and everything attached to it.

Reading block signals correctly

Different failures mean different things, and the wrong reaction to a given signal makes it worse. This is diagnostic work, not guesswork.

HTTP 429. The honest signal. You exceeded a counted limit. Back off, honor Retry-After, permanently lower your steady-state rate for that domain. Recovery is usually fast and complete.

HTTP 403 (immediate). Not a rate problem. You failed a fingerprint, TLS or header check before pacing was even evaluated. Slowing down will not help — fix the identity layer. A 403 that arrives in 50 milliseconds is an edge rule; one that arrives after a full page load is a challenge failure.

HTTP 503 or a challenge interstitial. You've been promoted from 'unknown' to 'suspicious'. The site wants proof you're a browser. If your browser is real, you'll usually pass; if you're using an HTTP client or a poorly-masked headless browser, you won't.

CAPTCHA. Widely misread. A CAPTCHA is overwhelmingly a proxy IP reputation signal, not a fingerprint signal. Residential IPs get recycled, and the previous tenant may have been running something ugly. Before you spend a day tuning your fingerprint, swap the IP and see whether the CAPTCHA follows.

Silent degradation. The hardest to catch and the most expensive to miss: HTTP 200, well-formed page, but with results truncated, prices stale, or listings missing. Guard against it with canary checks — a handful of URLs whose correct content you know, fetched periodically, with an alert if what comes back doesn't match. Without canaries you can poison a dataset for weeks and never know.

Session invalidation. You were logged in; now you're not. Something about the session looked wrong — an IP that jumped continents, a fingerprint that shifted mid-session, or concurrent use of the same account from two apparent devices. This one is almost always an identity-consistency bug rather than a rate problem.

A worked example

Say you need 50,000 product pages daily from a Cloudflare-protected marketplace. The naive plan — 50,000 requests spread over 24 hours from a small pool — is roughly 35 requests per minute, and it will be dead within a day.

Work it from the other end instead. Assume a safe pace of 20 pages per hour per identity, and an 8-hour active window per identity (nobody browses for 24). That's 160 pages per identity per day, so you need about 315 identities — call it 350 with headroom for failures.

That number sounds intimidating until you compare the alternatives. 350 sessions at 20 pages an hour is a rounding error to a large marketplace. Fifty sessions at 140 pages an hour is a screaming anomaly. Same total volume, wildly different survival rate. Then apply the cheaper optimizations: cache with ETags so unchanged pages cost a 304, tier your refresh so only prices and stock refetch daily while descriptions refetch monthly. Suddenly 50,000 daily page fetches becomes maybe 12,000 real ones, and your identity requirement drops with it.

Building an architecture that respects limits by default

Good pacing has to be structural. Anything that depends on every developer remembering to add a sleep will fail the first time someone writes a quick backfill script.

Centralize scheduling

Workers should never decide their own timing. Put a scheduler in front that owns the token buckets, hands out permits, and knows the current health of every domain. Workers ask for permission to fetch and block until granted. This makes your global rate a single configurable number rather than an emergent property of however many workers happen to be running — which is the difference between turning throughput down in one place and hunting through six repositories during an incident.

Separate the queue from the fetcher

URLs go into a durable queue with priority and scheduled-earliest-fetch metadata. Fetchers pull work when they have both a permit and a healthy identity. Decoupling means a rate limit event doesn't lose work — the URL simply returns to the queue with a later timestamp. It also lets you drain gracefully instead of dropping in-flight jobs when you need to stop.

Track health per identity

Every profile needs a live scorecard: success rate, CAPTCHA frequency, average latency, last block. Identities degrade — a residential IP that was clean last week may have been recycled to someone abusive. Rest a struggling identity for a few hours rather than retiring it immediately; many recover on their own. Retire it permanently only after repeated failures across separate rest cycles.

And log the combination, not just the component. 'Profile 47 got a CAPTCHA' is barely useful. 'Profile 47, residential IP in Frankfurt, Windows 11 fingerprint, on its 340th request of the session, got a CAPTCHA' is a data point you can actually learn from. After a few thousand of those you'll know your real safe rate per target — which beats every rule of thumb in this article, including mine.

Bound your retries

Never retry unbounded. Cap attempts, cap total time in flight, and distinguish retryable failures (429, 503, timeouts, connection resets) from terminal ones (404, 410, malformed URL). A retry loop that treats a permanent 404 as transient will happily burn your entire rate budget on a page that will never exist. Add a dead-letter queue so genuinely failed URLs are visible rather than silently dropped, and review it weekly — patterns in that queue are usually the earliest sign that a target changed its defenses.

Monitor the leading indicators

By the time your success rate craters, you've already been blocked. The signals that move first are subtler: median response time creeping upward (challenge evaluation adds latency), a rising share of responses with unexpected content length, a slow uptick in CAPTCHA rate, an increase in 200-with-empty-results. Alert on those and you get hours of warning instead of a post-mortem.

Where the identity layer meets the rate limit

This is the part conventional rate-limiting advice leaves out, and it's the part that determines your ceiling.

Every best practice above assumes the site can tell your identities apart. If it can't — if all your sessions share a canvas hash, a WebGL renderer string and a font list — then per-identity pacing is a fiction. You have one identity wearing 300 different IP addresses, and the aggregate rate is what gets enforced.

This is why fingerprint quality is a throughput concern, not just a stealth concern. Better isolation between identities means more identities that can operate in parallel, which means more total throughput at the same per-identity pace. Halve your correlation rate and you roughly double your safe ceiling without touching a single delay setting.

Three properties matter most:

Consistency within an identity. A profile that reports a MacBook's WebGL renderer, a Windows font list and a Linux User-Agent is more suspicious than one with an ordinary, unremarkable Windows fingerprint. Internal contradictions are the loudest tell there is. Our guide on how to change your browser fingerprint goes into which attributes must move together.

Genuine variation between identities. Randomizing canvas noise while every profile still reports the same GPU string and the same 1920×1080 screen produces fingerprints that are technically distinct and trivially clusterable. Real device populations vary across screen size, GPU, OS version, font sets, memory and core counts in correlated ways — a real 4K screen usually accompanies a decent GPU.

Native application, not JavaScript injection. Fingerprint spoofing implemented by overriding JavaScript getters can be detected by checking whether those functions have been tampered with — Function.prototype.toString inspection, prototype chain walks, comparing values reported in the main thread against the same values read inside a Web Worker. Spoofing applied natively at the browser engine level has no such seam. Dual Login applies fingerprints natively rather than injecting scripts, which is why the values hold up under worker-context checks that break patch-based tools. It also matters for exactly the reason this section exists: a fingerprint that survives inspection is one that can sustain a higher request rate before it's flagged.

Proxy pairing rules

The binding between profile and proxy has its own discipline. Keep it one-to-one and stable — a profile should not share its exit IP with another profile, because a shared IP re-correlates identities you worked to separate. Match the fingerprint's implied geography to the proxy's actual location: a Germany-exit IP paired with an en-US locale, a New York timezone and US-centric language headers is an obvious mismatch, and the timezone check is one of the cheapest an anti-bot script can run.

Residential and mobile IPs also come with variable, occasionally awful latency. Build timeouts around what real connections do rather than what your office fiber does, and don't treat a slow response as a failure worth retrying immediately.

Ethics and the legal line

A quick word, because it affects how you should build.

Scraping public data is broadly lawful in the US and EU, and the hiQ v. LinkedIn litigation is the usual reference point for public-data access under the CFAA. But 'broadly lawful' hides a lot of specifics. Data behind a login is governed by terms you accepted. Personal data is governed by GDPR and its equivalents regardless of whether it was easy to fetch. Copyrighted content is still copyrighted after you've downloaded it. And degrading a site's service for its real users can create liability that has nothing to do with scraping law.

The practical rules that keep you on the right side: take what you need rather than everything you can reach; never degrade the service for actual users; don't collect personal data you have no lawful basis to hold; honor deletion and opt-out mechanisms; and identify your crawler honestly when you're operating openly rather than researching competitively.

Worth noting: the technically-correct approach and the ethically-defensible one converge almost perfectly. Polite pacing, aggressive caching, respect for stated limits — these are simultaneously the best way to stay unblocked and the best way to stay defensible. That's a rare alignment. Take advantage of it.

A practical rollout plan

If you're starting a new scraping project, this sequence saves the most pain.

Start with one identity and observe. Before writing any concurrency logic, run a single profile against the target at deliberately conservative pace — one request every five seconds. Watch what happens over several hours. Note when the first CAPTCHA appears, whether response times drift, whether content stays consistent. This is your baseline, and it's worth a full day.

Find the wall deliberately. Increase pace step by step until something breaks, and record where. You need to know where the edge is, and finding it on purpose with a disposable identity beats discovering it accidentally with your whole fleet. Note which signal fires first — 429 versus CAPTCHA versus silent degradation tells you which defense layer is dominant.

Set your operating rate at half the wall. Whatever pace triggered problems, run at half. Limits move with site load, time of day and defense updates; the margin absorbs that variance. Half sounds wasteful, and it is far cheaper than a fleet-wide block.

Then scale horizontally. Add identities, keeping per-identity pace fixed. Watch aggregate health as you grow. If success rates fall as you add profiles, you have a correlation problem, not a pacing problem — your identities aren't as distinct as you think, and the fix is in the fingerprint layer.

Instrument before you need it. Add canary URLs, per-identity health tracking and leading-indicator alerts while things are working. Building observability during an incident is miserable, and you'll make bad decisions with bad data.

Teams running multiple accounts alongside their crawls — marketplace sellers, affiliates, agency operators — will recognize this discipline. It's the same one behind managing multiple eBay accounts safely: isolate hard, move at a human pace, and never let one identity's mistake reach another.

FAQ

What is a safe request rate for web scraping?

There's no universal number, but useful anchors exist. For an unprotected static site, one to five requests per second from a single IP is usually fine. For a commercial site behind Cloudflare or DataDome, plan for 10–30 page loads per hour per identity and scale by adding identities. For logged-in scraping, match genuine human pace — a few pages a minute at most. Always test empirically against your specific target rather than trusting a rule of thumb, including these.

Does rotating proxies solve rate limiting?

Only the network layer, which is one of four. If every rotated IP presents the same browser fingerprint, the site correlates them into one identity and applies the limit to the aggregate. Worse, rotating IPs within a session produces behavior no real user exhibits. Rotate identities — profile, fingerprint, cookies and proxy together — not IPs alone.

What's the difference between a 429 and a 403 when scraping?

A 429 is a pacing signal: you exceeded a counted limit, and backing off usually restores access quickly. A 403 is an identity signal: you failed a fingerprint, TLS or header check, often before pacing was evaluated at all. Slowing down fixes a 429 and does nothing for a 403. Check timing to distinguish edge rules from challenge failures — a 403 in under 100ms never reached the application.

Should I respect robots.txt when scraping?

Whether it's legally binding is contested and jurisdiction-dependent. Whether it's operationally smart is not: any Crawl-delay tells you the pacing the operators consider acceptable, and disallowed paths are frequently honeypots or expensive endpoints that trigger flags fastest. Reading it costs one request and gives you a map of the minefield.

How many browser profiles do I need for a large scrape?

Divide your daily page target by (safe per-identity hourly rate × realistic active hours per identity). For 50,000 pages a day at 20 pages/hour over an 8-hour window, that's roughly 315 identities before headroom. Cut the target first, though — conditional requests with ETags and volatility-tiered refresh schedules routinely reduce real fetch volume by 70–90%, and every request you don't make is one you never have to pace.

Can I get away with headless Chrome instead of an antidetect browser?

Against light protection, often yes. Against commercial anti-bot systems, headless Chrome has well-documented tells — missing plugin arrays, distinctive WebGL behavior, navigator.webdriver, and CDP-attachment artifacts — that stealth plugins patch imperfectly and detection vendors track continuously. The deciding factor is usually whether your spoofing is applied natively or injected as JavaScript, since injected overrides can be caught by comparing main-thread values against Web Worker readings.

Why am I getting CAPTCHAs even though I'm crawling slowly?

CAPTCHAs are usually an IP reputation problem, not a pacing problem. Residential IPs get recycled, and the previous user of your exit node may have been doing something abusive. Swap the IP first and see whether the CAPTCHA follows the profile or stays with the address. If it follows the profile across several clean IPs, then it's a fingerprint issue.

Bringing it together

The teams that scrape successfully for years aren't the ones with the biggest proxy pools or the cleverest bypass scripts. They're the ones who understood early that request pacing, browser identity and proxy strategy are one problem with three faces, and who built systems that stay polite by default instead of relying on discipline under pressure.

Start conservative. Measure everything. Find the wall on purpose with a disposable identity, then operate at half of it. Scale by adding well-isolated identities rather than by making each one work harder. Cache like every request costs real money, because in proxy bandwidth and block risk, it does. And treat the identity layer as a throughput lever, not just a stealth feature — better isolation is what raises your ceiling.

If your bottleneck is identity isolation rather than raw request pacing, that's the part Dual Login is built for: real separate browser processes, natively-applied fingerprints that hold up under worker-context inspection, persistent per-profile data directories, and one-to-one proxy binding — so twenty parallel identities actually look like twenty different people rather than one busy machine. Spin up a few profiles, run your baseline test against a target you care about, and see where your real wall sits before you build around a guess.