The worst thing about getting blocked is the silence. A 403 is a single bit of information handed back to you after a decision that weighed several hundred signals, and nobody is going to tell you which one tipped it. I have watched a scraper run happily for eleven weeks and then die overnight because a vendor started reading HTTP/2 SETTINGS frames. I have watched another one die because someone on the team helpfully standardised every profile to the same screen resolution. Both looked identical from the outside: yesterday it worked, today it doesn't.
This article is the map I wish someone had drawn for me early on. Not a list of headers to copy, but a working model of what the detection stack actually looks like, which layer catches which class of mistake, and what you can realistically do about each one. It is written for people collecting data at volume — price monitoring, catalogue enrichment, SERP collection, marketplace research — rather than for people hoping a single flag will fix everything.
One framing note before we start, because it changes how you read everything that follows. There is no such thing as being undetectable. Detection is a probability estimate, continuously updated, and your realistic goal is to stay comfortably inside the distribution of ordinary traffic rather than to become invisible. Every technique below should be judged on that basis: does this move me toward the middle of the crowd, or does it just add one more unusual thing about me?
Detection happens in three layers, and they check each other
It helps enormously to stop thinking about anti-bot systems as one wall and start thinking about them as three sequential filters, each with a different view of you.
The network layer judges you before a single line of your JavaScript runs. It sees where the connection came from, how the TLS handshake was constructed, how the HTTP/2 frames were laid out, and which headers arrived in which order. This layer is cheap to run and it never sleeps, so it handles the overwhelming majority of crude traffic. If you are being blocked instantly, with no challenge page and no delay, you almost certainly failed here.
The client layer runs once a page has loaded and asks what kind of device is on the other end. This is browser fingerprinting: canvas and WebGL rendering, audio processing, installed fonts, screen geometry, timezone, language, hardware concurrency, and a long tail of API quirks. It produces a fairly stable identifier for a machine plus a verdict on whether that machine looks real.
The behavioural layer watches over time. It cares about how fast you move, whether your mouse exists, whether you fetch stylesheets and images like a renderer would, whether your visits have a daily rhythm, and whether this browser has ever been here before.
Why cross-layer consistency is the real gate
Here is the part most people miss, and it is the single most valuable idea in this article: the layers are not scored independently. They are cross-referenced, and contradiction is a far stronger signal than uniqueness.
Consider a request where the TLS handshake matches Chrome 133 on Windows, the User-Agent string claims Chrome 121 on macOS, Intl.DateTimeFormat().resolvedOptions().timeZone returns Asia/Kolkata, the IP geolocates to a Frankfurt datacenter, Accept-Language is en-US, and navigator.hardwareConcurrency reports 128 cores. Every one of those values is individually plausible. Together they describe a device that does not exist. No amount of header tuning saves that request, because the problem is not any single value — it is that the values disagree.
This is why the antidetect approach works when it works and fails badly when it fails. Changing values is easy; changing them coherently is the whole discipline. If you want the foundation for this, browser fingerprinting explained for beginners covers what the signals are before we get into how they are weaponised.
Layer one: your network signature
IP reputation, ASN and the datacenter problem
Every IP address belongs to an autonomous system, and ASNs are trivially classifiable into residential ISP, mobile carrier, hosting provider, VPN, and university. Hosting ASNs — AWS, Hetzner, DigitalOcean, OVH — carry an enormous prior against them, not because they are inherently evil but because almost no human browses a retail site from an EC2 instance. You are not being profiled as an individual at this stage; you are being profiled as a neighbourhood.
Beyond the ASN, commercial reputation feeds track per-IP and per-subnet behaviour: how many distinct accounts have logged in from here, how many failed challenges, how many abuse reports. This is why a cheap proxy pool can be useless on arrival. You are inheriting the sins of everyone who used that /24 last month, and no browser configuration fixes a poisoned IP.
A subtlety worth internalising: rotation is not automatically protective. A hundred requests spread over a hundred IPs that all belong to the same shady subnet, all presenting the same fingerprint, is a more obvious pattern than a hundred requests from one address, because now you have demonstrated that you control a pool. The pairing matters more than the count, which is the core argument in the antidetect browser plus residential proxies playbook.
TLS fingerprinting: JA3, JA4 and the patched-client tell
When your client opens a TLS connection it sends a ClientHello, and the exact composition of that message is a signature. The cipher suites you offer, in the order you offer them; the extensions present and their order; the supported elliptic curves; the point formats; the ALPN list. Different TLS stacks produce different ClientHellos, and hashing those fields gives you a fingerprint. JA3 was the first widely deployed version of this idea; JA4 is the modern successor, designed to survive the extension shuffling and GREASE values that modern Chrome injects deliberately.
What this means in practice is brutal for HTTP client libraries. Python's requests on OpenSSL, Go's net/http, Node's undici, curl — each has a distinctive handshake that looks nothing like a browser, no matter what User-Agent you put on top. You can set every header perfectly and still be identified before the first byte of your request line is parsed. Worse, the tools that patch this — impersonation forks and custom TLS stacks — often produce a handshake that matches no real browser version exactly, which is its own bright flag. A near-miss impersonation is sometimes more identifiable than an honest library, because honest libraries are common and near-misses are rare.
The reason a real browser engine sidesteps this entire category is simple: it is not impersonating a Chrome handshake, it is producing one.
HTTP/2, HTTP/3 and frame-level fingerprints
One layer up, the same logic applies. HTTP/2 gives clients latitude in how they open a connection, and that latitude is a fingerprint: the values in the initial SETTINGS frame (header table size, max concurrent streams, initial window size), whether a WINDOW_UPDATE follows and with what increment, the priority information attached to streams, and the order of the pseudo-headers :method, :authority, :scheme, :path.
Chrome, Firefox and Safari each have a characteristic pattern here, and they change it between major versions. A client that sends browser-like headers over a non-browser-like h2 connection is one of the easiest catches in the entire stack, and it is invisible to anyone debugging at the header level. If you have ever had a scraper that worked over HTTP/1.1 and mysteriously failed when you enabled HTTP/2, this was probably why.
Header order, casing and Client Hints
Browsers emit headers in a stable order, with stable casing, including a set that most libraries never send at all: sec-fetch-site, sec-fetch-mode, sec-fetch-dest, sec-fetch-user, upgrade-insecure-requests, and the User-Agent Client Hints family (sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform).
Those Client Hints are a gift to detection engineers, because they must agree with the legacy User-Agent string, and hand-rolled setups routinely get this wrong. Claiming Chrome 131 in the UA while sec-ch-ua lists version 118 is a contradiction that takes one line of server code to check. Similarly, sec-fetch-mode: navigate on a request for a JSON API endpoint, or sec-fetch-site: none on a request that should have had a referrer, both describe navigation that a real browser would never perform.
The requests you do not make
Finally, and most overlooked: a real browser fetching an HTML page immediately fetches dozens of other things. Stylesheets, fonts, JavaScript bundles, images, the favicon, analytics beacons, sometimes a service worker. It sends conditional requests with If-None-Match and collects a healthy number of 304s. It reuses TLS sessions.
A scraper that fetches exactly one URL and vanishes has left a hole in the shape of a browser. Server-side, that pattern is startlingly clean to detect: one HTML hit, zero subresource hits, same session, repeated ten thousand times. You do not need any client-side JavaScript to catch that, which is why it survives every clever patch people apply in the browser.
Layer two: the browser fingerprint
Canvas, WebGL and audio — the stable trio
The three heavyweight fingerprinting surfaces all work by asking your hardware to compute something and then hashing the result.
Canvas fingerprinting draws text and shapes to an offscreen canvas and reads the pixels back. Font rasterisation, anti-aliasing and subpixel rendering differ between GPU drivers and operating systems, so the resulting bitmap hash is remarkably stable per machine. WebGL goes further: WEBGL_debug_renderer_info exposes vendor and renderer strings that name your GPU and driver almost exactly, and rendering a scene and hashing it produces another hardware-derived value, along with dozens of queryable limits (max texture size, supported extensions, shader precision). AudioContext fingerprinting runs a signal through an oscillator and compressor and hashes the floating-point output, which varies subtly across audio stacks.
The EFF's Cover Your Tracks project is still the clearest public demonstration of how few bits of this you need before a browser is unique in a population of millions.
Here is the trap. Most tooling spoofs these in JavaScript by overriding HTMLCanvasElement.prototype.toDataURL, getImageData, getParameter and friends. That works right up until someone checks the override itself — and checking is easy. Call Function.prototype.toString on the method and see whether it says [native code]. Compare the property descriptor. Run the same test inside a fresh iframe, or inside a Web Worker, or in an OffscreenCanvas, where a patch applied to the main-thread window object never reached. Look for a prototype chain that has one more link than it should.
This is the architectural reason Dual Login applies fingerprints inside the engine rather than by injecting JavaScript into pages: values that come from the C++ layer are consistent everywhere the value can be read — main thread, iframes, workers, offscreen canvases — and there is no patched function to discover, because nothing was patched. If you want the mechanics of doing this properly, how to change your browser fingerprint walks through it field by field.
Screen, timezone, language and the contradiction hunt
This is where the cheap wins and the cheap losses both live, because these fields are trivial to read and trivial to cross-check:
screen.width/heightversuswindow.outerWidth/outerHeightversusdevicePixelRatio. A claimed iPhone reporting 1920x1080 at a pixel ratio of 1 is not an iPhone.Intl.DateTimeFormat().resolvedOptions().timeZoneversus the IP's geolocation. This is the single most common self-inflicted wound in the entire field.navigator.languagesversus theAccept-Languageheader. They come from different code paths and mismatch constantly in hand-built setups.navigator.platformanduserAgentData.platformversus the UA string versus the WebGL renderer. A macOS UA over an ANGLE Direct3D11 renderer is Windows wearing a hat.navigator.hardwareConcurrencyanddeviceMemory. Real consumer machines cluster at 4, 8, 12 and 16 cores. Your 96-core scraping box does not.- Font enumeration by measurement. Claiming macOS while lacking Helvetica Neue, or claiming Windows while lacking Segoe UI, is a two-line check.
None of these requires clever engineering to detect. All of them require discipline to get right, which is precisely why per-profile identities should be generated as coherent bundles rather than assembled by hand from whatever values seem plausible.
Automation flags
navigator.webdriver returns true when the browser is being driven by a standards-compliant automation framework — MDN documents it plainly as exactly that signal, and it is a W3C-specified property, not an accident. Selenium and stock Puppeteer/Playwright launches set it. Detection scripts read it in the first hundred milliseconds.
Around it sits a family of related tells: the --enable-automation switch that produces the infobar and other observable side effects, an empty navigator.plugins and navigator.mimeTypes array, a missing or hollow window.chrome object, Notification.permission reporting denied while the Permissions API reports prompt, and window.outerHeight of zero.
Individually these rarely trigger a hard block. In combination they are decisive, and the combination is exactly what a default automation setup produces.
CDP artefacts: the tell that survives every patched property
This one deserves its own subsection because it is the hardest to diagnose and the most commonly missed.
The Chrome DevTools Protocol is how Puppeteer, Playwright and every CDP-based tool talk to a browser. Attaching a client is not free: enabling certain domains changes observable behaviour inside the page. The best-known case involves the Runtime domain and error serialisation — when a debugger is listening, the act of logging an object can cause the engine to serialise it, which invokes getters that would otherwise never fire. A page can plant a getter on Error.prototype.stack, log an error, and see whether the getter ran. If it did, something is attached. No property patch hides this, because the leak is in the engine's behaviour, not in a value.
The practical consequence is a real architectural choice. Either you drive the browser with a full CDP client and accept a persistent tell, or you keep the driving surface narrow. Dual Login's default launch path spawns the engine as a plain OS process with no attached client, and its automation layer deliberately uses only DOM, Input, Page, Network and Target — never Runtime.enable — so that clicks and keystrokes are genuine trusted input events while navigator.webdriver stays false. It is a constraint rather than a trick: some conveniences (arbitrary JavaScript evaluation) become opt-in, flagged, and used sparingly.
Headless-specific leaks
Even with the obvious flags handled, headless builds differ from headful ones in ways that are measurable. Historically headless_shell shipped without proprietary codecs, so canPlayType('video/mp4; codecs="avc1.42E01E"') came back empty. GPU-less environments fall back to software rendering, and a WebGL renderer string containing SwiftShader is a strong bot indicator on consumer sites. Permission prompts resolve instantly instead of waiting for a user. Fonts installed on a stripped container image bear no resemblance to a real desktop.
Modern headless mode narrowed this gap considerably, but the gap is not zero, and it is not the direction you want to be betting on. Running a real headful browser on a real desktop-like environment removes an entire category of problems in one move.
Layer three: behaviour
Input entropy
Human mouse movement is noisy, curved and jittery, with acceleration and small overshoots before a click. Synthetic movement tends to be perfectly linear, arrive at the exact geometric centre of the target element, and produce a click with no preceding mousemove or mouseover. Human typing has variable inter-keystroke intervals with characteristic bigram timings and occasional corrections; scripted typing frequently uses a fixed delay, or dispatches input events without corresponding keydown/keyup pairs at all.
The strongest version of this check is trusted-event provenance. event.isTrusted distinguishes browser-generated events from dispatchEvent calls, and no amount of realistic-looking synthetic geometry survives it. This is why input synthesis at the protocol level — Input.dispatchMouseEvent through CDP, which the browser treats as real hardware input — beats anything done from inside page JavaScript.
That said: be honest about scope. If you are issuing plain GETs against product pages, there is no mouse to model and this whole layer is close to irrelevant. It matters when you are logging in, paginating through interactive UI, submitting search forms, or doing anything a vendor considers a protected action.
Navigation shape
Real users arrive from somewhere. They land on a homepage or a search result, click into a category, look at three products, go back, look at a fourth. Their Referer chain makes sense. They occasionally hit a 404 and go back.
Scrapers walk sitemaps. They visit URLs in lexical or numeric order, never send a plausible referrer, never revisit, never abandon a page after two seconds, and never look at the same product twice. That path signature is visible in ordinary web logs without any bot vendor involved, and it is one of the least defended-against tells because people focus on making each individual request look right rather than making the sequence look right.
Cadence, volume and rhythm
Machine timing is a giveaway in both directions. Requests spaced exactly 2.000 seconds apart are obviously scheduled. Requests fired as fast as the network allows are obviously not human. And traffic that maintains identical volume at 04:00 local time and 14:00 local time has no diurnal rhythm, which is a population-level signal your per-request tuning cannot touch.
The fix is uncomfortable because it costs throughput: jittered intervals drawn from a heavy-tailed distribution, occasional long pauses, session lengths that vary, and volume that rises and falls with the target's real local business hours.
Session history and the cold-jar problem
Every request from a brand-new cookie jar announces itself as a first-time visitor. No _ga, no session cookie, no prior consent record, no cached assets, no cf_clearance or equivalent from a previously solved challenge. If your system spins up a fresh container per job, you are presenting a stream of visitors who have never been anywhere, forever.
A persistent per-profile data directory — cookies, localStorage, IndexedDB, cache, service workers, all surviving between runs — flips this from a liability into an asset. A profile that has visited a site nine times over three weeks and solved one challenge along the way is treated very differently from one that appeared four seconds ago. This is the same mechanism that makes long-lived accounts survivable on marketplaces, discussed in how to avoid account bans on Amazon Seller.
Honeypots and poisoned data
Some defences do not detect you; they wait for you to detect yourself. Links hidden with display:none, visibility:hidden, zero-size containers or off-screen positioning, which a human can never click and a naive crawler always follows. Form fields hidden the same way, which a human never fills and an autofilling bot does. robots.txt entries that exist purely to see who reads them and then ignores them.
The more advanced version is data poisoning: once a session is flagged, it keeps working but starts receiving subtly wrong prices, stale stock levels or shuffled rankings. Nothing breaks. Your monitoring stays green. Your dataset quietly becomes fiction. This is the failure mode that costs the most money, and the only defence is periodic ground-truth verification from a genuinely clean path.
How the score is actually computed
Almost nobody blocks on a single signal any more. Signals feed a model that emits a risk score, and the score selects a response: serve normally, serve a transparent challenge, serve an interactive challenge, throttle, serve degraded content, or block. The response is often chosen per route, so the same score sails through a product page and hits a wall at checkout.
Here is my honest weighting after years of watching things break:
| Signal | What it reveals | Cost to fake convincingly | How often it is the real reason |
|---|---|---|---|
| IP reputation and ASN class | Whether requests come from where humans browse | Moderate — good residential or mobile IPs cost real money | Very often |
| TLS fingerprint (JA3/JA4) | Which library or patched client opened the socket | High for HTTP clients, free in a real browser | Often, and rising |
| HTTP/2 SETTINGS and header order | Same, one layer up | High for HTTP clients, free in a real browser | Often |
| Canvas / WebGL / audio hashes | Which GPU and driver stack rendered the page | Low with a native engine, high and fragile with JS patches | Sometimes — more often a linkage signal than a block trigger |
| Timezone / language / IP agreement | Whether the identity is internally coherent | Trivial, and constantly got wrong | Very often |
navigator.webdriver and automation flags |
That a framework is driving the tab | Trivial | Rarely alone, fatal in combination |
| CDP artefacts (Runtime domain) | That a debugger is attached right now | Requires architectural discipline, not a flag | Sometimes, and very hard to diagnose |
| Input entropy and trusted events | Whether a human is present | Moderate | Often on interactive flows, rarely on plain GETs |
| Cadence, volume, diurnal rhythm | Whether the traffic has a human shape | Free, but costs throughput | Very often |
| Cookie age and session history | Whether this visitor has ever been here | Cheap with persistent profiles, impossible without | Often |
| Subresource loading pattern | Whether a real renderer fetched the page | Free in a browser, impossible in curl | Often |
Read down the last column and you will notice something uncomfortable. The signals that block you most often are mostly not the exotic ones. They are IP class, internal coherence, request rhythm and session history — the boring operational stuff. People spend weeks on canvas noise and lose to a timezone mismatch.
What the vendors do differently
Cloudflare
Cloudflare sits in front of a very large share of the web and grades traffic into a bot score using network-layer signals, a JavaScript detections bundle and machine learning across its whole traffic sample. Its bot management documentation is unusually candid about the general architecture. Practically: verified good bots are allowlisted by signed identity rather than User-Agent; unverified automation gets a low score; Turnstile replaces classic CAPTCHAs with mostly-invisible proof-of-browser challenges; and a solved challenge deposits a clearance cookie that is bound to your IP and fingerprint, so rotating your IP mid-session throws away the thing you just earned.
Akamai, DataDome and the sensor-data school
Akamai's Bot Manager and DataDome both lean heavily on client-side telemetry collected by an obfuscated script and posted back as an encrypted blob, alongside a state cookie. The blob carries fingerprint data plus behavioural samples — pointer traces, timing, event sequences. Two properties make these harder to work around than pure network checks: the payload is integrity-protected and versioned, so replay and forgery decay quickly, and the decision is made server-side where you cannot see it.
The correct posture against sensor-based systems is not to defeat the sensor. It is to give the sensor genuinely ordinary data to collect, which means a real engine, a real GPU path, real trusted input, and a session with history.
In-house detection
Do not underestimate the target's own logs. A mid-sized retailer with a competent engineer will notice 200,000 hits on /product/* with zero image requests from one ASN inside a week. In-house rules tend to be crude but exquisitely tuned to that specific site's traffic, and they often catch things generic vendors miss — like the fact that nobody browses their catalogue in SKU order.
A playbook that survives contact with reality
Decide per target whether you need a browser
A real browser costs perhaps 200–400 MB of RAM per instance and a fraction of a CPU core. An HTTP client costs almost nothing. If a target serves clean server-rendered HTML with no client-side gating, use the cheap path, accept the TLS-fingerprint risk, and keep volume modest. The moment there is a JavaScript-rendered catalogue, a sensor script, a challenge page or a login, the browser is not a luxury — it is the only way to produce the signals the site expects. Most mature pipelines end up mixed: a fast HTTP tier for the easy 80% and a browser tier for the hard 20%.
One identity means one fingerprint, one IP, one cookie jar
This is the rule that everything else hangs from. A profile should own its fingerprint, its proxy and its data directory, and those three should stay married for the profile's whole life. Swap the IP under a stable fingerprint and you have just told the site that this device teleported. Swap the fingerprint under a stable IP and you have told it that a new machine appeared at the same address with the same cookies. Reuse the same fingerprint across fifty profiles and you have handed over a cluster key that links all fifty in one query.
This is also the actual difference between an antidetect browser and a VPN, which is worth being precise about — a VPN changes layer one and nothing else, as the antidetect browser versus VPN comparison lays out.
Warm sessions rather than burning them
Treat a working session as an asset with a cost basis. Once a profile has cleared a challenge and accumulated cookies, keep it: same data directory, same proxy, same fingerprint, next run. Rotate on failure signals, not on a timer. A pool of 200 warm profiles used gently will out-produce 5,000 cold ones by a wide margin, and it will cost less in proxy bandwidth.
Shape your rate to the site, not to your hardware
Pick concurrency and delay from what the target's normal traffic looks like, not from what your machine can push. Jitter intervals. Vary session length. Follow local business hours. Add an occasional dead end — a page you fetch and do nothing with. Accept that the sustainable throughput of a well-defended target is lower than you want, and size your infrastructure to that number instead of fighting it.
Instrument the block rate before you need to
When something breaks you want the answer in ten minutes, not ten days. Log, per request: proxy IP and subnet, profile ID, fingerprint cohort, HTTP status, response size, challenge type if any, and time to first byte. Then slice block rate by each dimension. A spike isolated to one subnet is a proxy problem. A spike isolated to one fingerprint cohort is a fingerprint problem. A spike across everything at once is the target deploying something new, and no amount of local tweaking will fix it that day.
Response size is the underrated field in that list. A silent shift in average body size on 200 responses is how you catch data poisoning and soft-blocking before it contaminates a quarter of reporting.
Canary profiles and staged rollout
Keep a handful of profiles that only ever fetch a known-stable page and verify a known-correct value. Run them continuously. When they start failing, you know the target changed rather than your code. And when you do change your own configuration, roll it to 5% of profiles first — a bad fingerprint template applied to your whole fleet at once can burn a pool you spent weeks warming.
Where the line is
The technical picture is not the whole picture, and pretending otherwise is how projects end up in legal review.
robots.txt is now a real standard, RFC 9309, and while it is not law, ignoring it is a documented decision that someone will eventually read back to you. Terms of service are contracts, and in several jurisdictions courts have distinguished between accessing public data — where the US Ninth Circuit's reasoning in hiQ Labs v. LinkedIn found the Computer Fraud and Abuse Act did not reach publicly available pages — and breaching an agreement you accepted by logging in. Those are different questions with different answers.
Personal data is its own regime entirely. Under GDPR and similar frameworks, collecting information about identifiable people creates obligations regardless of whether the page was public. Aggregate prices and stock levels sit in very different territory from names, emails and profile photos.
And there is a plain operational argument for restraint: a scraper that hammers a small site into degraded performance is causing real harm and will be blocked with real determination. Crawl politely, cache aggressively, request only what you will actually use, and identify yourself where you reasonably can. Most of the durable pipelines I know of are the polite ones.
FAQ
Can websites detect web scrapers that use a headless browser?
Often, yes. Headless environments differ from real desktops in measurable ways: software rendering shows up in the WebGL renderer string, codec support can be incomplete, installed fonts look nothing like a consumer machine, and permission prompts resolve instantly. Modern headless mode closed much of the gap, but if the target is well defended, running a real headful engine on a desktop-like host removes an entire class of tells rather than narrowing it.
Does changing my user agent help avoid scraper detection?
Barely, and it can hurt. The User-Agent string is one of the weakest signals available, and it must now agree with sec-ch-ua, navigator.platform, the WebGL renderer, the font list and the TLS fingerprint. Changing the UA alone typically creates contradictions where there were none, which raises your risk score instead of lowering it. Change the whole identity coherently, or leave it alone.
Are residential proxies enough on their own?
No. Good residential or mobile IPs solve the layer-one problem, which is genuinely the most common single cause of blocks — but they do nothing about your TLS fingerprint, your automation flags, your canvas hash or your request rhythm. Conversely, a perfect fingerprint from a datacenter IP fails immediately. The two halves are complements, not alternatives.
How can I tell which layer got me blocked?
Use the shape of the failure. An instant 403 or connection reset with no page is almost always network-layer. A challenge page that appears and never resolves points at the client fingerprint or an automation tell. Working normally for a while and then degrading, especially after a login or a burst, points at behaviour, rate or session state. Serving 200s with subtly wrong data means you are being soft-blocked and should verify against a clean path immediately.
Is being detected the same as being blocked?
No, and confusing the two is expensive. Detection produces a score; blocking is only one possible response to it. You can be detected and throttled, detected and challenged, detected and fed poisoned data, or detected and simply logged for later. Assuming that a 200 response means you were undetected is the most common monitoring mistake in scraping.
Can I scrape at scale without a browser at all?
Sometimes, and it is much cheaper when you can. Targets with server-rendered HTML and no bot vendor in front of them are perfectly reachable with a good HTTP client, provided you respect rate limits and accept that your TLS signature is identifiable. As soon as there is client-side rendering, a sensor script, a challenge, or a login, a real browser engine is the practical requirement — which is why most serious operations run both tiers and route per target.
Closing thought
If you take one thing from all of this, take the shift in framing. The question is not how to hide, because you cannot. The question is whether the identity you present is internally coherent, whether it stays coherent across the network, the fingerprint and the behaviour layers, and whether it accumulates history the way a real device does. Most scrapers fail not because they were caught being clever, but because they were caught contradicting themselves.
That is a solvable engineering problem, and it is mostly about architecture rather than tricks: a real browser engine, fingerprints applied natively so they hold everywhere a page can look, one proxy and one persistent data directory per identity, automation that produces trusted input without leaving a debugger attached, and enough measurement to know which layer moved when something breaks.
Dual Login was built around exactly that model — isolated profiles with coherent native fingerprints, per-profile proxies and durable session storage, driven over a deliberately narrow automation surface. If you are running a scraping or research pipeline and spending more time on blocks than on data, it is worth testing the parts that matter before you pay for anything: launch a few profiles, point them at your hardest target, and watch what the detection stack makes of them.