Web Scraping Compliance and GDPR Rules: 2026 Guide

Nobody gets fined for scraping product prices. Companies get fined — sometimes twenty million euros at a time — for what they collect alongside those prices, how long they keep it, and what they quietly do with it afterwards. That distinction is the whole subject. Web scraping compliance and GDPR rules are not really about the act of sending HTTP requests to a public server; they are about the moment personal data enters your pipeline, and everything that happens to it from then on.

I have spent years building and running scraping operations, and the pattern I see over and over is the same: teams obsess over the technical side — proxies, fingerprints, CAPTCHAs, rate limits — and treat the legal side as a checkbox someone else will handle. Then a data subject access request lands, or a regulator opens a file, and it turns out nobody can say what personal data the company holds, where it came from, or why. This guide is the document I wish those teams had read first. It covers the legal layers that apply to scraping, what the GDPR actually demands, the enforcement actions that shaped current practice, and a concrete checklist you can run your own operation against.

Why "is scraping legal?" is the wrong question

Scraping is a technique, and techniques are not legal or illegal in the abstract. A headless browser fetching a page does more or less exactly what Chrome does when a human clicks a link. Courts have repeatedly recognised this: automated access to publicly visible web pages, by itself, is not hacking. What the law cares about is context — whose data, under what terms, with what consequences.

That means the honest answer to "is scraping legal?" is always "it depends on three things." What you collect: personal data triggers data protection law, creative works trigger copyright, whole databases trigger database rights. How you access it: bypassing a login or an explicit technical barrier changes the analysis entirely. And what you do with it: republishing, reselling, enriching, or training models each carry their own obligations.

So instead of asking whether scraping is legal, ask: which laws does my specific project touch, and what does each of them require? For any operation touching European users or European websites, GDPR is nearly always the biggest of those laws — and it applies whether or not your company has an office anywhere near Brussels, because Article 3 extends it to any processing that monitors the behaviour of people in the EU.

The four legal layers every scraping operation sits under

Before going deep on GDPR, it helps to see the whole terrain. A scraping project can be perfectly fine under one legal layer and badly exposed under another. You have to clear all four.

Data protection law

The GDPR — along with the UK GDPR and cousins like Brazil's LGPD and California's CCPA/CPRA — governs personal data: any information relating to an identified or identifiable person. Names, usernames, profile photos, email addresses, phone numbers, job titles tied to people, review author handles, even a distinctive combination of quasi-identifiers. If your scraper touches any of it, this layer is live, and it is by far the most actively enforced against scrapers in 2026.

Contract law: terms of service

Almost every large website prohibits automated collection in its terms of service. Whether those terms bind you depends heavily on whether you affirmatively accepted them. Browsewrap terms — a link in the footer — are weakly enforceable against a visitor who never clicked "I agree." But the moment you create an account and scrape from behind a login, you accepted the terms, and breach-of-contract claims become very real. This is precisely the ground on which LinkedIn ultimately prevailed against hiQ Labs after years of litigation.

Computer misuse statutes

Laws like the US Computer Fraud and Abuse Act (CFAA) criminalise access "without authorization." For a decade, platforms tried to stretch these statutes to cover scraping of public pages. The tide turned with the US Supreme Court's Van Buren decision and the Ninth Circuit's rulings in hiQ Labs v. LinkedIn, which together established that scraping data the public can see, without circumventing an authentication barrier, is generally not a CFAA violation. Password-protected areas are a different world: go through a login you are not entitled to use and the statute has teeth again.

Intellectual property and database rights

Facts are not copyrightable, but expressive content is. Scrape and republish full articles or photographs and you have a copyright problem regardless of how politely you scraped. The EU adds a sui generis database right protecting substantial investment in assembling a database: extracting a substantial part of, say, a classifieds site's entire listings corpus can infringe that right even where no individual listing is protected.

What the GDPR actually requires when you scrape

The full text of the GDPR runs to 99 articles, but for a scraping operation, six ideas do most of the work.

"Publicly available" does not mean "fair game"

This is the misconception that has cost companies the most money. The GDPR contains no exemption for data that people posted publicly. A LinkedIn profile, a public Instagram bio, a review signed with a real name — all of it remains personal data, and scraping it is "processing" that needs a legal justification, transparency, and everything else the regulation demands. Regulators have said this explicitly and repeatedly; the joint statement on data scraping signed by a dozen data protection authorities, including the UK ICO and Canada's OPC, put platforms and scrapers alike on notice that mass collection of publicly accessible personal data is still regulated processing.

You need a lawful basis — and in practice that means legitimate interests

Article 6 offers six lawful bases. Consent is unworkable for scraping — you cannot ask a million profile owners first — so nearly every legitimate scraping operation relies on Article 6(1)(f), legitimate interests. That basis is real and usable, but it is a test, not a label. You must be able to show three things.

First, purpose: a specific, articulable interest — market intelligence, price monitoring, fraud detection, journalism, academic research. "We might find it useful someday" fails. Second, necessity: the scraping must actually be needed for that purpose, at that scale, with those fields. If aggregate counts would serve the purpose, collecting names is not necessary. Third, balancing: your interest weighed against the data subjects' rights and reasonable expectations. Scraping business contact details for B2B research usually balances well. Scraping personal photos to build a biometric database — as Clearview AI learned in four countries — does not.

Write the analysis down. A legitimate interests assessment (LIA) is a two-page document that transforms your position from "we never thought about it" to "we considered the rights of the people involved and here is our reasoning." Regulators treat those two situations very differently.

The Article 14 transparency trap

Here is the requirement most scraping teams have never heard of. When you obtain personal data from somewhere other than the data subject — which is the definition of scraping — Article 14 obliges you to tell those people: who you are, what you collected, why, and what rights they have, generally within a month.

For large-scale scraping this sounds impossible, and the regulation anticipates that. Article 14(5)(b) waives individual notice where it would involve "disproportionate effort," particularly for research and statistical purposes. But the exemption is not automatic. The Polish DPA's landmark fine against Bisnode — a business intelligence firm that scraped public registries and emailed notices only to the people whose addresses it happened to hold, skipping millions of others — established that "it would be expensive" is not disproportionate effort by itself. If you rely on the exemption, you must take compensating measures. The most common one is a prominent public privacy notice describing your scraping activity, backed by records showing why individual notice was genuinely impracticable.

Data minimisation: collect fields, not pages

Article 5 requires that personal data be adequate, relevant, and limited to what is necessary. Translated into engineering terms: parse and keep the fields your purpose needs, and discard everything else at ingestion time. The common anti-pattern is archiving raw HTML "just in case." Those archives are full of usernames, avatars, and comment threads you never needed — and every one of them is personal data you are now accountable for. Filter PII at the parser, not at the report.

Storage limitation: scraped data rots anyway

You may keep personal data only as long as the purpose requires, so set a retention schedule and automate it. The good news is that compliance and data quality point the same direction here. Scraped personal data goes stale fast — people change jobs, delete accounts, move house — so a 90- or 180-day retention window usually improves your dataset while shrinking your legal surface at the same time.

Data subject rights do not disappear because you scraped

People whose data you hold can demand access, correction, or erasure, or object to the processing outright. That means your dataset needs to be searchable by identifier and your team needs a process that answers within a month. If finding one person's records across your scraped corpus would be a three-week engineering project, you have a compliance bug, not just a data-architecture quirk.

For high-volume, systematic scraping of personal data, add one more artifact: a Data Protection Impact Assessment under Article 35. Large-scale systematic monitoring is squarely in DPIA territory, and having one on file is the difference between a routine regulator interaction and an uncomfortable one.

The enforcement record: what actually happened to scrapers

Abstract rules matter less than what regulators actually did. Three lines of cases define the current landscape.

Clearview AI scraped billions of face photos from public websites to build a facial-recognition product. Data protection authorities in France, Italy, Greece, and the UK each fined it roughly twenty million euros; the Dutch DPA followed in 2024 with a €30.5 million fine plus escalating penalties for non-compliance. Clearview's defence — the photos were public — persuaded no one. It is now the canonical answer to anyone in a planning meeting who says "but it's public data."

The Bisnode case in Poland is smaller but arguably more relevant to ordinary scraping businesses, because Bisnode was not doing anything exotic — it aggregated data from public business registers. The fine was for the Article 14 failure alone: not telling people. It made transparency, rather than collection, the pressure point, and it is why a public scraping notice is now standard practice for data vendors.

hiQ Labs v. LinkedIn ran from 2017 to 2022 and delivered a split verdict that captures exactly where the law landed. Scraping public LinkedIn pages did not violate the CFAA — a major win for the scraping industry — but hiQ still lost on breach of LinkedIn's user agreement, and the company shut down. Public-page scraping is not hacking; scraping in breach of terms you accepted can still end your business.

Alongside the case law, regulators have published a wave of guidance aimed at AI-era scraping. The UK Information Commissioner's Office has consulted specifically on whether legitimate interests can support web scraping to train generative AI — its provisional answer being "possibly, but the bar is high" — and the European Data Protection Board addressed scraping in its 2024 opinion on AI models. If you scrape to feed models, read both. The balancing test is applied far more sceptically when the downstream use is unbounded.

A working compliance checklist for scraping at scale

Here is the process I would run for any new scraping project, in order.

Before you write a line of code

• Define the purpose in one sentence. If you cannot, stop — every downstream test depends on it.
• Map the fields. List every field you intend to extract and mark each one personal, non-personal, or special category. Special category data — health, politics, religion, biometrics, sexuality — is close to untouchable for commercial scraping; Article 9 demands conditions you almost certainly cannot meet.
• Write the LIA. Purpose, necessity, balancing, in plain language, dated and stored where you can find it in a hurry.
• Check the access model. Public pages only, or behind a login? A login changes your contract exposure completely, and usually your computer-misuse exposure too.
• Decide retention now. On day one it is a config value; on day four hundred it is a migration project.

While the scrapers run

• Rate-limit like a considerate guest. Hammering a site into degraded service is where civil claims — and, in extreme cases, computer-misuse arguments — gain traction. Spread requests, back off on errors, cache aggressively.
• Treat robots.txt and terms of service as evidence, even where they are not binding. A court or regulator assessing your good faith will ask whether you knew the site objected. Deliberately ignoring clear signals colours everything else you did.
• Filter PII at ingestion. Your parser should be the enforcement point for minimisation: drop the fields you did not map, and never warehouse raw pages containing personal data beyond a short debugging window.
• Log provenance. Record the source URL and collection timestamp for every record. You cannot answer an access request — or defend a lawful basis — for data you cannot trace.

After collection

• Secure it like it is yours, because legally it is. Access controls, encryption at rest, and a named owner. A breach of scraped personal data is still a reportable breach.
• Automate deletion. Retention that depends on someone remembering is retention that does not happen.
• Stand up a rights process. A monitored inbox, a search capability, and a thirty-day clock.
• Audit quarterly. Fields drift. A scraper written for a price element starts swallowing seller profiles after a site redesign. Re-run the field map against what the pipeline actually stores.

Where antidetect browsers fit into a compliant operation

An honest word about the tooling this blog is attached to, because "antidetect" and "compliance" strike some people as opposites. They are not — but the relationship deserves precision.

Websites deploy browser fingerprinting and behavioural analysis to block automated traffic wholesale, including traffic that is perfectly lawful: price comparison on public listings, ad verification, brand-protection monitoring, SERP tracking, academic research, or simply checking how your own pages render across regions. Anti-bot systems do not adjudicate legality; they block patterns. An antidetect browser like Dual Login gives each session a consistent, realistic device identity so that lawful collection is not trivially distinguishable from ordinary browsing — the same reason teams pair it with residential proxies to see the localised, real-user version of a page instead of a datacenter-flavoured one.

The architecture also serves compliance directly, in ways that tend to get overlooked:

• Isolation is segregation. Each Dual Login profile is a fully isolated browser with its own data directory, cookies, and storage. If you scrape for multiple clients or projects, per-profile isolation keeps their data — and their distinct legal contexts — physically separated, which is exactly what a regulator means by "organisational measures." It is the same isolation model that keeps multi-account operations from cross-contaminating, applied to data hygiene.
• Per-profile proxies mean per-jurisdiction discipline. You can pin a project to the geography whose rules you have actually assessed, rather than letting one shared session wander across legal regimes. If you are wondering why a VPN does not achieve the same thing, the difference between an antidetect browser and a VPN is that a VPN moves your IP address and nothing else.
• Reproducibility supports accountability. Persistent profiles with stable fingerprints make collection runs repeatable and auditable — you can say exactly which identity, which exit IP, and which session collected which records, which is precisely the provenance trail the checklist above asks for.

And the boundary, stated plainly: an antidetect browser does not change what the law permits. Using one to avoid indiscriminate blocks on public pages you may lawfully read is using infrastructure. Using one to break into authenticated areas, to impersonate consumers, or to keep collecting personal data your LIA cannot justify is a compliance failure with better camouflage — and camouflage impresses regulators not at all. The tool is neutral; your data map, lawful basis, and retention policy are what make the operation defensible.

Five mistakes that turn a scraping project into a regulatory problem

1. The "it's public" defence. Covered above. Fatal, and still the most common. Public availability affects the balancing test; it does not switch off the GDPR.
2. Scraping behind logins on autopilot. Teams start on public pages, hit a wall, create accounts, and keep going — without noticing they just accepted a contract. Make logged-in scraping an explicit, reviewed decision, never a workaround someone ships on a Friday afternoon.
3. Warehousing raw HTML indefinitely. Every archived page is a liability snapshot. Keep parsed, minimised fields; expire raw captures within days.
4. Enriching and reselling personal data without ever telling anyone. Combining scraped profiles with other sources and selling the result is precisely the business model that produced the biggest fines on record. If that is your model, an Article 14 notice and a rigorous LIA are not optional paperwork — they are the product's licence to exist.
5. Ignoring special category data. A scraper pointed at forums or social platforms will ingest health conditions, political opinions, and religious views whether you wanted them or not. Detect and drop them at the parser.

FAQ

Is web scraping legal under GDPR?

Scraping itself is not prohibited. Scraping personal data is regulated processing: you need a lawful basis (usually legitimate interests, documented in an LIA), Article 14 transparency, data minimisation, a retention limit, security, and a way to honour data subject rights. Scraping genuinely non-personal data — prices, stock levels, product specs — falls outside the GDPR entirely.

Does GDPR apply if my company is not in the EU?

Yes. Article 3(2) applies the regulation to any organisation, anywhere, that monitors the behaviour of people in the EU or offers them goods and services. Systematically scraping EU residents' profiles is monitoring. The location of your servers or your registered office does not change the analysis, and EU authorities have fined non-EU companies — Clearview AI among them — on exactly this basis.

Do I need consent to scrape publicly available data?

No — and consent would be impossible to obtain at scale anyway. Nearly all lawful scraping of personal data relies on legitimate interests under Article 6(1)(f). But that basis must be earned: a documented purpose, a necessity argument, and a balancing test against the data subjects' reasonable expectations, plus a public transparency notice if individual Article 14 notices are impracticable.

Does robots.txt have legal force?

Not directly — it is a convention, not a statute or a signed contract. But it is evidence. A regulator weighing your legitimate interests balancing test, or a court weighing your good faith, will ask whether the site clearly objected to automated access and whether you knowingly ignored that. Respecting robots.txt costs little and strengthens every other argument you will ever need to make.

Can I legally scrape data behind a login?

Treat it as a different project with a different risk profile. Creating an account means accepting the terms of service, so scraping in breach of them is a contract claim waiting to happen — that is how LinkedIn ultimately beat hiQ. Authenticated areas also raise the data subjects' expectation of privacy, which shifts the GDPR balancing test against you, and circumventing revoked access can implicate computer-misuse law. Get legal review before, not after.

What do I do when someone asks me to delete their scraped data?

Verify the request, search your dataset by the identifiers you hold, and erase the records within one month unless a narrow exemption applies. Keep a minimal suppression entry — just enough to recognise the person — so your next crawl does not re-collect what you just deleted. If your pipeline cannot do this today, that gap is itself the compliance finding.

Final thoughts

Web scraping compliance and GDPR rules reward the same habits that make a scraping operation technically good: knowing exactly what you collect, keeping only what you need, and being able to account for every record. The teams that get into trouble are almost never the ones running the biggest crawls — they are the ones who never wrote down a purpose, never set a retention date, and never imagined a regulator would ask. An afternoon spent on a field map and an LIA buys you years of defensibility.

On the infrastructure side, do it with tooling built for isolation and repeatability. Dual Login runs each project in its own fully isolated browser profile — unique fingerprint, dedicated data directory, per-profile proxy — so your collection stays organised, auditable, and separated by client and jurisdiction. If you are scaling up a lawful data operation, give it a try and see how much cleaner the operational side becomes.