How to Scrape Google Search Results at Scale (2026 Guide)

Most articles on this subject were written by someone who scraped Google for a weekend. This one is written from the other side of the project: the part where you have 40,000 keywords, a proxy invoice that tripled in a month, a parser that quietly returned nulls for six hours before anyone noticed, and a stakeholder asking why Tuesday's rankings look nothing like Monday's.

Scraping ten search result pages is a script. Scraping ten million is a systems problem, and almost none of the hard parts live in the HTTP request. They live in identity management, cost per row, parser durability, and knowing when your data is wrong rather than missing. Those are very different failure modes, and the second one is the expensive one.

This guide covers what actually works when you need to scrape Google search results at scale, what it costs, where the traps are, and — importantly — when you should not build the pipeline at all.

What "at scale" actually means

The phrase is useless without a number. Three orders of magnitude separate projects that share the same vocabulary and share almost no engineering.

The jump from hobby to team is where people get surprised. Nothing gradually degrades. You add the 30th worker, the block rate goes from 2% to 60% inside an hour, and every mitigation you try makes it worse because you are treating a reputation problem as a rate problem.

The jump from team to industrial is different again. By then blocking is a solved, boring cost line. What kills industrial projects is that nobody can answer "is today's data good?" without a human eyeballing a spreadsheet.

Decide the shape of your question first

Before any code, write down which of these you are doing, because they demand different architectures:

• Rank tracking. A fixed keyword list, run on a schedule, where position and context are the payload. Precision matters enormously; volume is predictable. Consistency between runs matters more than absolute truth.
• Corpus building. You want URLs, titles and snippets across a topic to feed something downstream — a content model, a link prospecting list, a competitor map. Volume is bursty, precision per row is forgiving, coverage matters.
• Entity or SERP-feature research. You care about what Google shows: AI Overviews, People Also Ask, local packs, video carousels, shopping units. This is the only workload that genuinely requires a rendering browser for most queries.

A rank tracker built like a corpus scraper produces noisy rankings nobody trusts. A corpus scraper built like a rank tracker costs ten times what it should.

Why Google is the hardest public target on the web

Plenty of sites are annoying to scrape. Google is in its own category, for reasons worth understanding before you fight them.

There is no organic SERP API

This is the root cause of everything else. Google publishes plenty of APIs, but none of them return the organic web results a browser shows:

• The Programmable Search JSON API queries a custom search engine you configure. Even scoped to the whole web, its index, ordering and features differ from google.com. It is not a SERP mirror, and it is quota-limited and priced per thousand queries.
• The Search Console API gives you real, first-party, averaged position data — but only for sites you own, and only aggregated.

If you own the site, the Search Console API is nearly always the better answer, and it is free, accurate and fully sanctioned. Use it. Scraping only becomes necessary when you need SERPs for properties you do not own, or the full page as a user sees it.

The signals stack in four layers

Blocking decisions are not made on one thing. Think of four independent layers, each of which can flag you on its own:

1. Network. The IP's autonomous system, its history, whether it is a known datacenter range, how many distinct sessions have come from it recently.
2. Transport. Your TLS handshake and HTTP/2 characteristics. A Python client claiming to be Chrome 140 in its User-Agent while presenting an OpenSSL cipher order no Chrome build has ever sent is a contradiction that costs nothing to detect.
3. Browser surface. The classic fingerprint: canvas and WebGL output, audio stack, installed fonts, screen metrics, timezone, language list, hardware concurrency, and the client-hint headers derived from all of it. If you are new to this layer, browser fingerprinting explained for beginners is the shortest path to understanding what is being measured.
4. Behaviour. Query cadence, whether pagination happens faster than a human could read, whether the mouse ever moves, whether the same identity searches 400 unrelated commercial terms in an hour.

Most scraping projects harden layer one, ignore layer two entirely, half-fake layer three with a User-Agent string, and never think about layer four. Then they blame the proxies.

The consent, locale and personalization layer

Even a perfectly unblocked request can hand you the wrong data. Google's results vary by country, interface language, approximate location, device class, signed-in state and prior activity in that session. In the EU you will meet consent interstitials that a naive HTTP fetch reads as an empty result page. Two workers with different proxies in the same city can legitimately return different orderings.

This is why the context is part of the data. A position without the country, language, device and timestamp attached is not a measurement, it is a rumour.

The change that reshaped rank tracking

For two decades, &num=100 returned a hundred results in a single request, and essentially every rank tracker on earth was built on it. Around September 2025 Google stopped honouring it reliably; requests for large result counts began returning ten. The arithmetic is brutal: tracking a top-100 position went from one request to ten. Overnight, at constant keyword volume, request counts and proxy bandwidth for deep tracking rose by roughly an order of magnitude.

Two lessons generalise beyond this specific parameter. First, never let a cost model depend on an undocumented convenience — it can be withdrawn without notice. Second, ask honestly how deep you need to go. Most commercial decisions are made on the top 20. If you are paying ten times over to fill positions 30–100 that nobody reads, that is a product decision disguised as an engineering one.

Four ways to collect SERPs, compared honestly

A point that saves real money: these are tiers, not choices. The mature architecture is layered. Fetch the plain HTML for the bulk of queries, because it is ten times cheaper per row. Route to a real browser only the queries that need rendering, that came back suspicious, or that are sampled for verification. Keep a paid API as the overflow valve for deadline days. Purists who insist on one mechanism for everything either overpay or under-deliver.

And yes, sometimes the honest answer is to buy the API. If your business value is the analysis rather than the collection, and your volume is under a few hundred thousand queries a month, an API is frequently cheaper than the engineer-months plus proxy spend of doing it yourself. Build the pipeline when you need control over locale, device, features or provenance that no vendor exposes — or when volume makes per-query pricing absurd.

The proxy layer, where most projects actually die

Types, and when each is right

• Datacenter. Cheap, fast, sold by the IP. For Google specifically, whole ranges carry a permanent reputational discount. Usable for low-value, high-volume sampling where you accept a meaningful block rate. Not usable as the backbone of a rank tracker.
• ISP / static residential. A residential-looking IP hosted in a datacenter, rented per IP per month. The sweet spot for many SERP workloads: predictable cost, stable identity, decent reputation. If a specific IP burns, you notice immediately because it is yours.
• Rotating residential. Real consumer IPs, billed per gigabyte. The best reputation, the worst cost model for browser traffic, and the most operationally awkward: your exit point can change mid-session, latency is high and variable, and some peers are simply broken. Excellent for hard queries, wasteful as a default.
• Mobile. Carrier-grade NAT means thousands of real users share your IP, which makes it very hard to hold against you. Expensive, slow, and worth reserving for the queries that nothing else can complete.

The practical build is a portfolio: ISP proxies as the backbone, residential for retries and for geo-specific work, mobile as the last resort. If you want the full treatment of how proxy choice and browser identity have to be designed together, we wrote it up in the antidetect browser with residential proxies playbook.

Do the concurrency math before you buy anything

This is the calculation almost nobody does, and it determines your entire budget.

Suppose you need 50,000 queries a day. That is about 2,080 an hour, spread evenly. Now pick a per-identity budget — how many queries one browser identity on one exit IP may run per hour before it looks abnormal. A conservative figure for commercial keywords is 15 to 25.

At 20 queries per identity-hour, 2,080 divided by 20 gives roughly 105 identities working continuously. Add headroom for cooldowns, failures and retries and you provision around 130. That number, not your keyword count, is what sizes your proxy pool and your host fleet.

Now the bandwidth. A rendered SERP in a real browser, with images and subresources, runs 400 KB to 1 MB. The raw HTML document alone is typically 100–250 KB. So:

• Browser-rendered, 50,000 × 600 KB ≈ 30 GB per day. At $5/GB residential, about $150 a day.
• HTML-only, 50,000 × 150 KB ≈ 7.5 GB per day. About $37 a day.

Same data, four times the cost, purely from how you fetched it. Block images and fonts in your browser tier, gzip everything, and never render a page whose text you already have. That single discipline has saved more scraping budgets than any anti-detection trick.

Sticky sessions beat per-request rotation

The instinct is to rotate the IP on every request so nothing accumulates. For Google this is precisely wrong. A session that changes country between page one and page two of the same search is describing a person who does not exist. Cookies set on one exit arrive from another. Consent state resets. You have not become anonymous, you have become incoherent.

Bind one identity to one exit IP for the life of a session — a coherent block of ten to forty queries with human-plausible gaps — then retire the whole pairing together. Coherence over churn, always.

Geo targeting: the parameters and the fossil

Country and language belong in the request, not just in the proxy. gl sets the country of search, hl sets the interface language, and both should agree with the browser's Accept-Language and the timezone of the profile. A German exit IP running a US English browser asking for gl=fr is three claims that contradict each other.

There is also uule, an encoded location parameter that has floated around SEO tooling for years. It is undocumented, it has changed behaviour more than once, and it is easy to construct wrongly in a way that silently does nothing. Treat it as an experiment you verify, not a foundation you build on.

The browser layer: one machine cannot look like 200 people

What has to vary, and vary together

Rotating User-Agent strings is not fingerprint management, it is a costume. The measurable surface includes canvas and WebGL rendering output, the WebGL vendor and renderer strings, audio processing characteristics, the font list, screen and available-screen dimensions, device pixel ratio, hardware concurrency, memory hints, timezone, the full language list, and the Sec-CH-UA client hints the browser derives from its own build. MDN's reference on the User-Agent header is a good reminder of how little that one string actually carries in 2026.

Internal consistency is what matters. Two hundred profiles with two hundred different canvas hashes but the same GPU string, the same font list, the same screen size and the same timezone are two hundred obvious siblings. A believable identity is a coherent one: a macOS user agent should come with Mac fonts, a Mac GPU string, plausible Retina metrics and no Windows-only APIs. If you want to see how thin naive spoofing is, run a few profiles through the EFF's Cover Your Tracks and watch the uniqueness score. Our guide on how to change your browser fingerprint walks through which properties are worth touching and which are traps.

Profiles, not tabs

Running 50 tabs in one browser gives you one identity with 50 tabs. Everything shared — cookie jar, storage, cache, GPU, the process itself — is shared. The unit of isolation has to be a full browser profile with its own data directory, its own proxy and its own fingerprint, launched as its own OS process.

That has a hard resource floor. Real Chromium instances cost roughly 300–800 MB each depending on the page, so plan around five to eight concurrent profiles per 4 GB of RAM and one CPU core per two or three active profiles. Our 130-identity example is therefore not one big box; it is a handful of machines, or a smaller fleet running identities in rotating shifts. Plan the hardware from the concurrency number, not from optimism.

Headless is a tell, and automation flags are worse

Headless Chrome has improved a great deal — the Chromium headless documentation is worth reading on how the modern mode differs from the old one — but a headed browser on a real display remains the safer bet for a hostile target.

The bigger issue is the automation surface. Standard driver stacks announce themselves: navigator.webdriver flips to true, an infobar appears, certain protocol domains get enabled that a human session would never trigger. If you drive profiles yourself, keep to the low-level protocol domains that correspond to real input and navigation, and avoid the ones that mark a session as scripted. The Chrome DevTools Protocol reference is the map; the discipline is using as little of it as possible.

A related trap worth naming: a VPN does not solve any of this. It changes one layer and leaves the other three untouched, which is why the difference between an antidetect browser and a VPN matters more here than in almost any other use case.

Request shaping: the parameters that carry weight

The rule that outranks every entry in that table: freeze your parameter set per dataset and record it on every row. Changing filter or gl halfway through a tracking programme creates a step change in your data that looks exactly like a Google algorithm update, and you will spend a week investigating your own commit.

Parsing SERPs that change under you

Anchor on structure, never on class names

Google's markup class names are effectively random and they rotate. A parser keyed on .tF2Cxc will work beautifully for weeks and then return zero rows on a Tuesday. Write predicates that describe shape instead:

• A result is a container holding exactly one heading element, whose nearest ancestor anchor has an external href, and which contains a displayed-URL element.
• Extract the URL from the anchor, the title from the heading text, and the snippet from the largest text block in the container that is not the title or the URL.
• Derive position from document order among the containers that survived those tests, not from any attribute.

This is slower to write and enormously cheaper to own. Redesigns move classes constantly and structure rarely.

Golden files and drift alarms

Save raw HTML for a small rotating sample of every run — say one in five hundred — into cheap object storage. Two things then become possible that are otherwise impossible.

First, regression testing: pin a dozen saved pages as fixtures, run the parser against them in CI, and any refactor that breaks extraction fails before it ships. Second, forensics: when a client disputes last month's numbers, you can show them the page.

Then alarm on the parser's own statistics rather than on errors. Track, per hour: null rate per field, mean results parsed per page, share of pages yielding zero results, and share of responses that were challenges. Errors are loud and get fixed. A parser that silently returns nine results where there were ten is the expensive bug, and only distribution monitoring catches it.

The features that break naive extractors

Modern SERPs interleave AI Overviews, People Also Ask blocks, video and image carousels, local packs, shopping units, sitelinks, FAQ expansions and inline definitions among the organic results. Naive parsers do one of two bad things: count them as organic results, inflating everyone's position, or drop them silently, so a page that visually pushed you below the fold reports you at position three.

Classify every block, store the ones you care about with an explicit type, and record the pixel-ish ordinal alongside the organic ordinal — that is, where the result sat among everything on the page, not just among the ten blue links. Clients care about what a human sees.

Rate, backoff, and the CAPTCHA question

Budget per identity, not per crawler

Global rate limits are the wrong abstraction. The unit that gets blocked is the identity — profile plus exit IP — so the budget belongs there. Give each identity a queries-per-hour cap, a session length, a cooldown, and a lifetime after which it retires regardless of health. Then let the scheduler solve for throughput by adding identities, not by pushing existing ones harder.

Humans are also irregular. A worker that fires at exactly 180-second intervals for nine hours has a machine-perfect signature no matter how good its fingerprint is. Jitter the gaps, cluster a few queries then pause, and let sessions end at odd lengths.

Exponential backoff with jitter, and a hard stop

On a 429, a redirect to a challenge page, or an empty result set that should not be empty: back off exponentially with random jitter, and treat the identity as burned rather than merely paused. Cool that profile and exit pairing for hours, not seconds. Requeue the query to a different identity.

Also set a circuit breaker. If the challenge rate across the fleet exceeds a threshold — say 15% over five minutes — stop the whole crawl and page a human. Something has changed upstream and burning your entire proxy pool discovering that is the most expensive possible way to learn it.

Do not solve-and-hammer

Routing challenges to a solving service and continuing at full speed is the single most common way a working pipeline destroys itself. A challenge is information: that identity has been judged. Solving it does not repair the judgement, it just buys one more page from an already-marked session, and the marking usually propagates to the subnet. Every serious operator I know treats challenge rate as the primary health metric and tunes the fleet to keep it near zero, rather than paying to push through it.

Storage, dedupe, and making the data usable

Store observations, not conclusions. One row per result per query per run, with the full context attached: query text, gl, hl, device class, exit country, timestamp in UTC, the parameter set version, the parser version, organic ordinal, absolute ordinal, block type, URL, title, snippet, and a hash of the raw HTML you archived if you sampled it.

That schema costs a little more disk and buys you the ability to answer questions later that you did not know to ask. Rank tracking without a parser-version column means you can never distinguish an algorithm change from your own bug.

Canonicalise URLs before comparing them — strip tracking parameters, normalise trailing slashes, resolve obvious redirect chains — but keep the original string too. Dedupe on the tuple of query, context and canonical URL within a run, never across runs; a URL that appears twice on different days is data, not a duplicate.

Finally, a discipline that costs nothing: run a handful of canary queries every cycle. Pick queries whose top results are stable and known, in a few different locales. If a canary's top result changes or its fields go null, alert immediately. Canaries catch parser breakage, wrong geo routing and silent challenge pages hours before anyone downstream notices, and they cost a few requests a day.

Legal and ethical guardrails

This is not legal advice and your jurisdiction and purpose matter enormously, but a few facts are worth having straight.

Google's own robots.txt disallows /search, and its terms of service prohibit automated querying. Ignoring robots.txt is not itself a crime in most places, but it is a contractual and reputational matter, and it is exactly the sort of detail that shows up in a dispute. On the US computer-crime side, hiQ Labs v. LinkedIn established that scraping publicly accessible data does not readily constitute unauthorized access under the CFAA — a genuinely important precedent, and one that says nothing about contract claims, copyright in the scraped content, or data-protection law.

So: collect only the fields you need, avoid harvesting personal data incidentally present in snippets, keep your request volume proportionate rather than aggressive, retain raw pages no longer than you need them, and get a lawyer's read before you build a commercial product on the output. Discipline here is also good engineering — the pipelines that are polite are usually the ones that survive.

A reference architecture that holds up

Put together, the pieces look like this:

1. Keyword queue. A durable queue with priority and per-context partitioning, so a stalled locale cannot starve the rest. Idempotent job ids so a retry never double-charges.
2. Scheduler. Owns identity budgets, cooldowns, session assembly and the circuit breaker. This is the component with the actual intelligence; everything else is plumbing.
3. HTML tier. The cheap bulk path: transport-fingerprint-matched HTTP clients through ISP proxies, images and subresources never requested.
4. Browser tier. Isolated real profiles for rendering-dependent queries, retries after a challenge, and a random verification sample of the HTML tier's output. This tier is where fingerprint quality earns its keep.
5. Parser service. Versioned, structure-based, with golden-file tests in CI and per-field null-rate metrics emitted on every run.
6. Storage. Append-only observation rows plus sampled raw HTML in object storage.
7. Observability. Challenge rate, parse null rates, results-per-page distribution, cost per thousand rows, and canaries. Alert on all five.

Notice that anti-detection is one of seven boxes. Teams spend 80% of their attention there and then lose the project to an unmonitored parser.

Where Dual Login fits

Dual Login is the browser tier. It runs many fully isolated browser profiles on your own machine, each with its own persistent data directory, its own proxy, and a unique but internally consistent fingerprint applied natively by the browser engine rather than injected as JavaScript after page load — which matters, because injected spoofing is itself detectable and does not reach Web Workers.

For SERP work at scale, the parts that tend to matter in practice:

• Real processes, real isolation. Each profile is its own OS process with its own cookie jar and cache, so a challenge on one identity does not touch the other 129.
• Per-profile proxy binding, including SOCKS and authenticated upstreams, with WebRTC masked to the exit IP so the layer that leaks a real address for so many setups simply does not.
• A local HTTP API for launching profiles and driving tabs — navigate, click, read, capture — over low-level protocol calls that keep navigator.webdriver false, so your scheduler can orchestrate the fleet programmatically.
• Local-first. Your keyword list and your results never pass through a vendor. For competitive research that is not a nice-to-have.

It is not a scraping product and it will not write your parser. It solves exactly one part of the problem — making a hundred concurrent browser identities look like a hundred different people on a hundred different machines — and leaves the pipeline to you. If you are comparing options at this layer, the best antidetect browser for multiple accounts rundown and our notes on cheaper Multilogin alternatives cover the trade-offs, including the ones where a competitor is the better fit.

FAQ

Is it legal to scrape Google search results?

In the United States, courts have held that accessing publicly available data does not by itself violate the Computer Fraud and Abuse Act — hiQ v. LinkedIn is the reference point. That is separate from Google's terms of service, which prohibit automated querying, from copyright in the content you collect, and from data-protection rules if personal data ends up in your store. Many companies scrape SERPs commercially; they do it with volume discipline, narrow field collection, short retention and legal review. Get advice for your specific use case rather than relying on a blog post.

How many Google searches can one IP handle per day?

There is no published number and anyone quoting a precise one is guessing. It depends on the IP's type and history, the coherence of the browser identity behind it, query commerciality, geography and timing. As a working starting point for planning: 15–25 queries per identity-hour on a residential or ISP exit, with human-plausible gaps and sessions that end and cool down. Measure your own challenge rate and adjust from there — that metric, not a rule of thumb, is the real answer.

Do I need a real browser, or is a plain HTTP request enough?

For titles, URLs, snippets and organic positions, well-shaped HTTP requests with a matched TLS and HTTP/2 fingerprint work and cost roughly a quarter as much in bandwidth. You need a real browser for JavaScript-rendered surfaces such as AI Overviews and some rich result blocks, for retrying anything that got challenged, and for periodically verifying that your cheap tier is still returning the truth. Layer them; do not pick one.

Why did deep rank tracking get roughly ten times more expensive?

Because &num=100 stopped being honoured reliably in late 2025. One request that used to return a hundred results now takes ten paginated requests, which multiplies both request count and proxy bandwidth for top-100 tracking. The practical response for most teams is to shorten tracking depth to the top 20 or 30, where nearly every business decision actually gets made, and reserve deep crawls for specific reports.

Residential or datacenter proxies for SERP scraping?

Neither alone. Static residential (ISP) proxies are the best backbone for SERP work: predictable per-IP pricing, decent reputation, stable identity, and you notice immediately when one burns. Keep rotating residential for retries and hard geo targets, mobile as a last resort, and datacenter only for low-value bulk sampling where you accept losses. Buying one type for everything is how the invoice or the block rate gets out of hand.

Can one antidetect browser profile serve many queries?

Yes, and it should — a profile that runs a single query and dies wastes the accumulated cookie and history state that makes it look human. Give each profile a coherent session of roughly ten to forty queries with realistic gaps, keep it on one exit IP throughout, then cool it down. What you must not do is reuse the same profile for scraping and for a signed-in account you care about; personalization pollutes your data, and a challenge on the scraping side is a risk to the account.

Wrapping up

The hard part of scraping Google at scale is not defeating a defence. It is building something whose cost per row you understand, whose failure modes are visible, and whose output you would defend in a meeting. Get the identity layer coherent, do the concurrency arithmetic before buying proxies, fetch HTML where HTML will do, parse on structure, and instrument the parser as carefully as the crawler. Everything else is tuning.

If the browser tier is your current bottleneck — you need dozens or hundreds of genuinely distinct, proxy-bound, consistently fingerprinted browser identities running on your own hardware — that is the specific problem Dual Login was built for. Spin up a handful of profiles, point them at different exits, check them against a fingerprint test page, and see whether they hold up before you commit to an architecture.