Web Scraping Infrastructure Setup Guide for 2026

The scraper is never the hard part. Anyone who can read HTML can write something that pulls 500 product pages on a Tuesday afternoon. The hard part starts on the second Tuesday, when the same script returns 403 on every third request, the proxy bill has quietly tripled, and nobody in the room can say whether yesterday's price drop was real or a parsing bug.

This is a web scraping infrastructure setup guide for that second Tuesday. It assumes you can already parse a page, and that what you actually need is the machinery around the parser: the part that decides what gets fetched, what identity fetches it, where it appears to come from, what happens when it fails, and how you find out something is wrong before a stakeholder does.

I will be specific about numbers and, more importantly, specific about the order in which you should make decisions. Getting the order wrong is how teams end up running a hundred headless browsers to scrape a site that exposes a clean JSON endpoint, or buying premium residential bandwidth to download product images they throw away.

Who this guide is for

You are running somewhere between 10,000 and a few million fetches a month. You have at least one target that fights back. You either already pay for proxies or are about to. You are one person or a small team, not a 40-engineer data platform group, so every layer you add has to justify its maintenance cost.

If you are scraping three pages a day from a static site, close this tab and use a cron job and curl. Infrastructure is a response to scale and hostility. Without either, it is overhead.

What "infrastructure" actually means in scraping

Every scraping stack, whether or not anyone designed it, has five layers. Naming them is useful because most outages belong to exactly one, and teams routinely debug the wrong one for days.

1. Fetch tier. How the bytes are retrieved. A raw HTTP client, a headless browser, or a real browser process.
2. Identity layer. What the target believes you are: TLS and HTTP/2 characteristics, headers, cookies, and for browser tiers the full device fingerprint (canvas, WebGL, audio, fonts, screen, timezone).
3. Network layer. Where you appear to be. Proxy type, exit geography, session stickiness, bandwidth cost.
4. Orchestration. What gets fetched, in what order, how often, with what concurrency, and what happens on each class of failure.
5. Persistence and observability. Where the bytes land, how you turn them into rows, and how you know when the pipeline is lying to you.

The design rule that saves the most money: choose from the top down, but budget from the bottom up. The fetch tier you pick dictates your identity requirements, which dictate your proxy requirements, which is where nearly all of the recurring cost lives. A decision that looks like an engineering preference at layer one becomes a five-figure invoice at layer three.

Step 1: Classify every target before you buy anything

Do this before you write a line of code. Open each target, watch the network tab, and put it in a tier. The tier determines everything downstream, and mixing tiers into one pipeline is the single most common cause of overspending.

Tier 0 — open and static. Server-rendered HTML, no bot management, often a real sitemap. Government registries, small ecommerce, documentation, most job boards' listing pages. A plain HTTP client with a polite delay handles these forever.

Tier 1 — a JSON API you can reach. The page is a shell and the data arrives over XHR or fetch. There may be a token, a signed header, or a nonce you have to harvest from the HTML first. Nine times out of ten this is dramatically cheaper than rendering the page, and the payloads are 10 to 50 times smaller. Always look for this before writing a DOM parser.

Tier 2 — rendered plus defended. Content requires JavaScript, and an edge provider inspects TLS characteristics, header ordering, JS challenge results and behavioural signals. Marketplaces, travel, sneakers, ticketing, large retail.

Tier 3 — behind a login. The scarce resource is no longer bandwidth, it is the account. One careless burst gets an identity flagged and you lose weeks of accumulated trust, not just a request.

Treat that cost column as an order of magnitude, not a quote. The point is the ratio: a Tier 2 fetch costs roughly 20 to 100 times a Tier 1 fetch. If 80% of your volume is Tier 0 or 1 and you route all of it through browsers because that was easiest to build, you are paying about fifty times over for most of your data.

Step 2: Pick the cheapest fetch tier that returns real bytes

Start with a plain HTTP client

A well-tuned HTTP client in Python, Go or Node will comfortably do hundreds of requests per second per core, uses tens of megabytes of memory, and has no browser to crash. For Tier 0 and Tier 1 this is not a compromise, it is the correct answer. Add connection pooling, a sensible timeout ladder (connect 5s, read 20s), gzip and brotli, and conditional requests with If-None-Modified where the server honours them.

Understand why plain clients get blocked

When an HTTP client gets 403 on a page a browser loads fine, the header list is usually not the problem. The problem is one layer below. During the TLS handshake your client advertises a specific ordered list of cipher suites, extensions and elliptic curves, and that ordering is a fingerprint. Default Python and Node clients advertise combinations no released Chrome has ever sent. HTTP/2 adds a second layer: the SETTINGS frame values, the order they appear in, window sizes, and pseudo-header ordering all differ between real browsers and libraries.

This is why an impersonating client (curl-impersonate, curl_cffi, a uTLS-based Go client) often fixes a block that no amount of header tweaking would. It is also why you must test with the real target, not with a fingerprint-echo service. Passing a fingerprint test proves you look like Chrome to that test.

Know what headless still leaks

Chrome's headless mode has come a long way; the modern implementation shares the same code path as headed Chrome rather than being a separate shim, which closed a whole family of old tells (see the Chromium headless documentation). What it does not close is the environment: no real GPU, no window manager, a suspiciously clean canvas and audio profile, an automation-controlled flag, and an open DevTools protocol connection that is itself observable. For Tier 2 and especially Tier 3, that residue matters.

Use a real browser only where the target earns it

When a target correlates a device fingerprint with a session over days, you need a real browser process with an identity that is coherent and stable. That is the point where an antidetect browser stops being a growth-hacking curiosity and becomes plumbing: a real Chromium process, per-profile fingerprint applied in the engine rather than injected by JavaScript, its own persistent data directory, and its own proxy. If the vocabulary here is new, browser fingerprinting explained for beginners is the shortest useful primer, and the EFF's Cover Your Tracks tool shows you the raw signal set in about ten seconds.

My standing rule: the cheapest fetch tier that returns the same bytes a real user would see, re-evaluated quarterly. Targets add and remove defences constantly. A Tier 2 target can quietly become Tier 1 when a team ships a public API, and nobody will send you an email about it.

Step 3: Build the identity layer deliberately

Coherence beats uniqueness

The instinct is to make each identity maximally unusual. That is backwards. A rare but internally consistent device survives far longer than a common one that contradicts itself. Detectors are looking for contradictions, because contradictions are cheap to test and almost never occur on real hardware.

The classic self-inflicted wounds: a user agent claiming macOS while WebGL reports a Direct3D ANGLE renderer; a timezone of Asia/Dhaka on an IP that geolocates to Ohio; Accept-Language: en-US on a profile whose declared locale is German; a 4K screen resolution reporting a device pixel ratio of 1 with a 700px inner height. Each of those is a single boolean check on the server side. If you are assembling fingerprints by hand, this practical guide to changing a browser fingerprint walks through which fields have to agree with which.

Decide which identities are disposable and which are permanent

This is an architectural decision, not a runtime one.

• Disposable identities suit anonymous breadth crawling. Generate, use for a bounded window, discard. No cookies worth keeping, no reputation to protect. Cheap.
• Permanent identities are required for anything logged in. The profile is the asset. It gets its own persistent data directory, its own cookie and localStorage state, its own proxy exit, and ideally the same exit for its entire life.

Mixing the two models is how people lose accounts. If an account-bearing profile ever gets a freshly generated fingerprint or a different country's exit IP, you have effectively told the target that the device was replaced overnight. Re-authentication is the single riskiest event in a session's lifecycle, so architect to avoid it: persist storage aggressively, back up the profile directory, and never let a crash-recovery path silently create a clean profile.

A point worth making plainly because it costs people real money: a VPN does not do this job. It changes one signal, the IP, and leaves the other few hundred identical across every account on the machine. The difference between an antidetect browser and a VPN is the difference between changing your return address and changing your handwriting.

Practical identity rules

1. One profile equals one identity equals one data directory equals one proxy exit. No sharing, ever.
2. Derive locale, timezone and language from the exit IP's geography, not from your laptop's settings.
3. Mask WebRTC to the proxy exit, or the real address leaks past a perfectly good proxy.
4. Give every profile its own OS process. Shared processes share caches, and shared caches correlate identities.
5. Snapshot profile directories on a schedule. The session is worth more than the machine it sits on.

If you are running dozens of these rather than a handful, the operational side (bulk launch, per-profile proxy assignment, cookie portability across machines) becomes the actual work. That is the comparison ground covered in the best antidetect browser for multiple accounts.

Step 4: Size the proxy layer with arithmetic, not vibes

Match the proxy type to the tier, not to the marketing

The expensive mistake is buying residential bandwidth for work that datacenter IPs would have handled, because one target once returned a 403. Test per target. A single stubborn site does not justify upgrading your entire pool.

Sticky versus rotating is a session question

Rotate per request only for stateless, anonymous fetching. Anything with a session, a cart, a token or a login needs a sticky exit for the whole session lifetime, and a permanent identity wants the same exit for weeks. A pool that silently rotates mid-session will produce a beautifully mysterious bug class: intermittent logouts, CSRF failures, and challenge pages that never reproduce locally. The trade-offs and the sticky-session configuration details are covered in depth in the antidetect browser with residential proxies playbook.

Do the bandwidth math before you sign anything

This is the calculation that decides your monthly invoice, and almost nobody does it up front.

A modern product page with images, fonts, analytics and video previews is 2 to 4 MB. The same page with images, media, fonts and third-party scripts blocked is typically 200 to 400 KB. The underlying JSON, if you can reach it, is often 10 to 40 KB.

Run that through a million pages:

• Full page load at 3 MB average: 3 TB. At $3.50/GB that is roughly $10,500.
• Asset-blocked rendering at 300 KB: 300 GB, roughly $1,050.
• Direct JSON at 30 KB: 30 GB, roughly $105.

Same data, hundred-fold cost difference. Block images, media, fonts and known analytics domains by default in every browser-tier fetch, and turn them back on only for the specific pages where rendering genuinely depends on them. Then keep a running per-target byte average in your metrics, because a target that starts inlining base64 hero images will move your bill before it moves your error rate.

Step 5: Orchestration — the layer that determines whether you sleep

The frontier

Keep a durable queue of what to fetch, not an in-memory list. A single table with the normalised URL, a hash of it as the unique key, a priority, a state, an attempt count and a next-attempt timestamp will carry you into the millions of rows on ordinary Postgres. Reach for Redis or a real broker when you need throughput above a few thousand jobs per second, not before.

Normalise before hashing: lowercase the host, strip tracking parameters, sort the remaining query keys, drop fragments. Un-normalised URLs are how a crawl fetches the same page eleven times under different UTM tags.

Make every job idempotent

A job should be safe to run twice. Key the stored result by content hash and job identity, so a duplicate run overwrites rather than appends. Once reruns are free, retries, backfills and deploys during a crawl all stop being scary.

Classify failures — never use one retry policy

This is the highest-leverage 40 lines of code in the whole stack. A single generic "retry three times" policy will hammer a target that just asked you politely to slow down, and will give up on a transient TCP reset.

• Connection reset, timeout, 502, 503, 504 — transient. Retry with the same identity, exponential backoff plus jitter.
• 429 — you are being rate limited. Honour the Retry-After header, and apply the backoff to the whole target, not just the worker that got unlucky. One worker backing off while nine others keep pushing is not backing off.
• 403 or a challenge page — an identity or network decision was rejected. Rotate the exit and the identity, and reduce concurrency for that target. Retrying the same identity immediately is how a soft block becomes a permanent one.
• 404 or 410 — mark gone. Do not retry. Track the rate; a spike in 404s usually means a URL pattern changed, not that the content vanished.
• 200 with a challenge body — the most dangerous case, discussed below.

Shape rate per target, not per worker

Maintain a token bucket keyed by target domain. Every worker draws from the shared bucket. Add a minimum inter-request delay and, where politeness matters, a permitted time window. This is also where you honour robots.txt, which since 2022 has an actual specification in RFC 9309 rather than folklore.

Concurrency capacity, measured

HTTP workers are bounded by bandwidth and file descriptors. Hundreds per box is normal.

Browser profiles are bounded by memory and disk I/O. Plan on 350 to 600 MB per live Chromium profile with a handful of tabs, even with low-memory flags enabled. On a 16 GB machine, 12 to 20 concurrent profiles is realistic; 40 is a swap-thrash story. The limit people miss is disk: each profile writes cookies, cache and IndexedDB continuously, so twenty profiles on a spinning disk or a network volume will stall on I/O long before RAM runs out. Put profile data directories on local NVMe, and monitor free space as a first-class metric.

Step 6: Storage, extraction and change detection

Store the raw response, always

Compress and keep the raw body, keyed by content hash, with the fetch timestamp, status, final URL after redirects, the identity used and the exit country. Object storage or a plain directory tree both work. Gzipped HTML is around 40 KB, so a million pages is roughly 40 GB, which is a rounding error against your proxy bill.

The reason is simple: parsers are wrong more often than fetches are. When you discover next month that you have been reading the sale price instead of the list price, reparsing 40 GB of stored HTML takes twenty minutes and costs nothing. Refetching it costs another full crawl and a fresh helping of block risk.

Keep extraction as a separate pass

The crawler writes raw bytes plus metadata. A separate extractor reads raw and writes typed rows. Version the extractor and stamp extractor_version on every row. That one column lets you answer the question that otherwise takes a day of arguing: did the data change, or did our code change?

Detect change instead of refetching blindly

Hash the normalised content (strip timestamps, nonces, session ids, rotating ad slots) and skip downstream work when the hash is unchanged. Then assign every field a volatility class and a cadence: prices hourly, stock every 15 minutes, descriptions weekly, reviews daily. Crawling everything at the fastest field's cadence multiplies cost by a factor of ten for no analytical gain.

The five columns that answer most questions

On every stored row keep source_url, fetched_at, http_status, identity_id and proxy_exit_country. When someone asks why the German prices look wrong, those five columns will tell you in one query that the German crawl ran through a US exit for six hours.

Step 7: Observability, the layer everyone skips

Four numbers, tracked per target

1. Block rate, broken down by target, identity and exit. An aggregate block rate hides the fact that one subnet is responsible for everything.
2. Yield — usable rows per successful fetch. This is your real health metric.
3. Cost per thousand rows, combining bandwidth and compute.
4. Freshness lag — the age of the oldest record that should have been refreshed by now.

Soft blocks are the real enemy

A hard block throws an exception and wakes someone up. A soft block returns HTTP 200 with a challenge page, an empty result set, or a stripped-down version of the content. Exception-based monitoring sees a perfectly healthy pipeline while the warehouse quietly fills with nulls. I have seen this run for three weeks.

Defend with content assertions on every fetch: an expected selector must be present, the body length must fall inside a learned band for that target, and no known challenge markers may appear. A fetch that fails an assertion is a failure regardless of its status code. Then alert on yield collapse rather than error count.

Run a canary crawl

Every hour, fetch twenty known-good URLs whose correct extracted values you have pinned by hand. If the canary passes and production fails, the problem is your scale or your identities. If the canary fails too, the site changed. That single distinction cuts most debugging sessions in half.

Keep the evidence

On any failure, persist the response body, a screenshot for browser-tier fetches, the request headers, and the identity and exit used. Diagnosing a block from a log line reading Error: 403 is guesswork; diagnosing it from the actual challenge page takes about a minute.

Step 8: The failure modes that will actually page you

Proxy pool rot. Exits get burned gradually. Track block rate per exit, retire anything above your threshold automatically, and hold back a fresh reserve. Pools degrade quietly rather than failing loudly.

Session expiry. Cookies lapse, tokens rotate, and refresh flows change. Track session age per identity and refresh on a schedule you control instead of discovering the expiry mid-crawl.

Silent layout change. Caught by yield monitoring and canaries, never by exception handling.

Clock and timezone drift. A container running UTC while its identity claims Europe/Warsaw reintroduces the exact contradiction you paid to avoid. Pin timezone per profile, and verify it from inside the browser rather than trusting the environment variable.

Disk exhaustion. Raw stores and profile directories grow forever. A full disk corrupts profile databases, and a corrupted profile database can cost you a logged-in session.

Cross-target starvation. One target returning 429 for an hour should not stall the queue for the other forty. Per-target buckets and per-target worker caps.

Step 9: A worked cost model

Say you need 250,000 Tier 2 pages a day.

Naive version: full page loads averaging 3 MB through rotating residential proxies. That is 750 GB a day, about 22.5 TB a month. At $3.50/GB, roughly $78,000 a month. This is the invoice that ends scraping projects.

Asset-blocked version: block images, media, fonts and third-party analytics, averaging 300 KB. 75 GB a day, about 2.25 TB a month, roughly $7,900. One configuration change, a tenth of the cost.

JSON-first version: for the 60% of pages whose data is reachable via an internal API at 30 KB, plus rendering the remaining 40%. Roughly 950 GB a month, around $3,300. Compute goes down too, because fewer browsers are running.

On top of that, budget compute. Twenty concurrent browser profiles want roughly 16 GB of RAM and 8 vCPUs, which is $150 to $300 a month on a mid-tier VPS or a spare workstation. Compute is almost never the constraint. Bandwidth and account risk are.

Price everything with your own vendor's rates before committing. The lesson is not the absolute figures, it is that the same dataset can cost $3,300 or $78,000 depending on decisions made in the first week. Teams working to a tight budget should also read the small-team buyer's guide before locking into per-profile pricing that punishes growth.

Step 10: Legal and ethical guardrails

Not legal advice, and jurisdictions differ. But there is a defensible baseline that keeps most projects out of trouble and costs you very little:

• Read robots.txt and honour it for crawling unless you have a specific, documented reason not to. The RFC 9309 semantics are the standard to implement against.
• Respect rate limits and Retry-After. Load you impose is load someone else pays for.
• Do not collect personal data you do not need. If you touch anything that identifies a person, you are in data-protection territory and need a lawful basis, a retention policy and a deletion path.
• Do not access material behind an authentication wall you are not authorised to pass.
• Keep the terms of service of your commercial targets on file, and get a review before you build a product on top of scraped data.
• Cache aggressively. The politest thing a scraper can do is fetch a page once.

If you are scraping in a commercial account-management context rather than a research one, the operational discipline overlaps heavily with account safety work; the Amazon seller ban-avoidance playbook covers the same habits from the account side.

A reference architecture you can copy

Here is a stack that works at a few million fetches a month and that one competent person can maintain:

1. Frontier: a Postgres table with normalised URL, url_hash unique key, target, priority, state, attempts, next_attempt_at.
2. Dispatcher: a small service that leases jobs in batches, enforces a per-target token bucket, and refuses to lease work for a target currently in backoff.
3. Fetchers, two flavours: a pool of HTTP workers with browser-accurate TLS for Tiers 0 and 1, and a pool of real browser profiles for Tiers 2 and 3, each profile bound one-to-one to an identity and a sticky proxy exit.
4. Raw store: gzipped bodies in object storage keyed by content hash, with a metadata row per fetch.
5. Extractors: versioned, idempotent, reading only from the raw store. Never from the network.
6. Warehouse: typed tables plus the five provenance columns.
7. Observability: per-target block rate, yield, bytes per fetch and freshness lag, with content assertions on every fetch and an hourly canary.
8. Identity manager: the inventory of profiles with their exit, locale, session age, last successful fetch and rolling block rate, plus a nightly backup of each profile directory.

Items 3 and 8 are the two most people improvise, and they are the two that cost the most when improvised. Managing a few dozen persistent browser identities with per-profile proxies, portable cookie state and a scriptable launch path is exactly the job Dual Login exists to do, which is why it sits in this architecture as infrastructure rather than as a tool someone opens by hand. If you are still comparing options at this layer, the GoLogin versus AdsPower comparison lays out what the differences actually amount to in daily use.

A pragmatic rollout order

Do not build all of this at once. Build it in the order that surfaces problems early.

Days 1 to 2. Classify your targets into tiers. Hunt for JSON endpoints. Fetch fifty pages per target by hand and record status, byte size and whether the content is complete. You will usually delete a third of your planned browser work at this stage.

Days 3 to 5. Frontier plus one HTTP fetcher plus the raw store. Get Tier 0 and 1 flowing end to end. Add the retry classifier now, not later.

Days 6 to 8. Extractors as a separate pass, versioned. Backfill from raw to prove the separation works.

Days 9 to 11. Observability. Content assertions, yield metrics, canaries, alerting on yield rather than exceptions. Do this before scaling, so that scaling problems are visible.

Days 12 to 16. The browser tier. Two or three identities, sticky exits, asset blocking on by default. Measure bytes per fetch immediately.

Days 17 onward. Scale profile count and concurrency slowly, watching block rate per exit and per identity. Retire burned exits automatically. Only then add targets.

Six mistakes that cost the most

1. Rendering everything. The largest avoidable line item in scraping, by an order of magnitude.
2. One retry policy for all failures. Turns a rate limit into a ban.
3. No raw store. Every parser bug becomes a full refetch.
4. Monitoring exceptions instead of yield. Soft blocks run for weeks in silence.
5. Reusing identities or exits across accounts. One flag becomes a cluster of flags.
6. Never re-classifying targets. You keep paying Tier 2 prices for a target that became Tier 1 six months ago.

FAQ

Do I need an antidetect browser for web scraping, or are proxies enough?

It depends entirely on the tier. For anonymous crawling of undefended pages, proxies plus a browser-accurate HTTP client are enough and far cheaper. You need a real browser with a managed identity once a target correlates a device fingerprint with a session over time, which in practice means defended marketplaces and anything behind a login. Proxies change where you appear from; they do nothing about what device you appear to be.

How many browser profiles can I run on one machine?

Budget 350 to 600 MB of RAM per live profile with a few tabs open, so 12 to 20 concurrent profiles on a 16 GB machine. The ceiling you will hit first, though, is usually disk I/O rather than memory, because every profile writes cookies, cache and IndexedDB continuously. Keep profile data directories on local NVMe and watch free space as a monitored metric.

What is the single biggest cost lever in a scraping stack?

Bytes per fetch. Blocking images, media, fonts and third-party analytics typically cuts a page from 3 MB to around 300 KB, which is a tenfold reduction on a per-gigabyte proxy bill. Finding an underlying JSON endpoint cuts it again by another factor of ten. Both are configuration decisions, not engineering projects.

How do I tell a soft block from a genuine layout change?

Run an hourly canary crawl over twenty URLs with hand-verified expected values. If the canary still passes while production fails, the issue is your scale, identities or exits. If the canary fails too, the site changed. Also assert on content for every fetch, since a soft block usually arrives as HTTP 200 with a challenge or stripped body, which exception-based monitoring will never notice.

Should scraped data be stored raw or parsed?

Both, in that order. Persist the compressed raw response keyed by content hash, then run extraction as a separate versioned pass that reads only from the raw store. Gzipped HTML is around 40 KB per page, so storage is trivial next to your bandwidth cost, and it means a parser bug costs a twenty-minute reprocess instead of an entire refetch.

Is it legal to scrape a website?

It depends on jurisdiction, what you collect, and how you access it, so get advice for anything commercial. A defensible baseline: honour robots.txt and rate limits, stay out of anything requiring authorisation you do not have, avoid personal data unless you have a lawful basis and a retention policy, cache so you fetch each page once, and keep your targets' terms of service on file.

Wrapping up

Scraping infrastructure is mostly a sequence of unglamorous decisions made in the right order. Classify targets before buying anything. Use the cheapest fetch tier that returns real bytes. Make identities coherent and, where accounts are involved, permanent. Do the bandwidth arithmetic before signing a proxy contract. Classify failures instead of retrying blindly. Store raw, extract separately, and monitor yield rather than exceptions.

Get those right and the boring outcome follows: data lands every morning, the bill is predictable, and nobody has to babysit a crawl.

If the identity layer is the part you are still improvising, that is the piece Dual Login was built for: isolated browser profiles with native per-profile fingerprints, their own persistent data directories, per-profile proxies and a scriptable automation API, so your permanent identities behave like real devices instead of like copies of your laptop. Spin up a handful, point them at your hardest target, and watch the block rate before you commit to anything larger.