Short answer: GDPR does not ban web scraping. But when a scraper collects, stores, organises, or reuses information about identifiable people, it is processing personal data. “It was public” is not a compliance strategy.
That distinction is now especially hard to ignore. In July 2026, the European Data Protection Board (EDPB) said that GDPR applies to scraping when it includes personal-data processing operations, and highlighted purpose limitation and transparency as key concerns. Its new web-scraping guidelines are still open for consultation, but the direction is unmistakable: the technical act of collecting a page is only the start of the compliance question. EDPB update
This guide is a practical operating framework, not legal advice. Use it to make better product and engineering decisions, then involve privacy counsel or your DPO for the cases that carry real risk.
First, separate “can scrape” from “can use”
Three different questions often get bundled together:
| Question | What it asks |
|---|---|
| Access | Are you authorised to access the site and collect the material? |
| Data protection | If people are identifiable, can you process the data under GDPR? |
| Reuse | Can you retain, enrich, sell, train on, publish, or contact people using it? |
Passing one test does not pass the others. A page can be publicly viewable while its data is still personal data. A technically successful crawl can still create GDPR, contract, intellectual-property, database-right, consumer-protection, or marketing-law issues.
So treat compliance as a workflow before you start the job—not a paragraph in a privacy policy added afterwards.
1. Decide whether GDPR is in scope
Start with two questions.
Does the dataset contain personal data?
Personal data is broader than a name or email address. It includes information relating to an identified or identifiable individual, such as a profile photo, username that can be linked to a person, location, IP address, professional history, reviews, or a combination of otherwise ordinary fields. EDPB definition
Company-level data can be harmless in some cases. But a “business contact” record may quickly become personal data when it contains a named sole trader, an employee’s direct email, a mobile number, or a linked profile. Design for the realistic dataset, not the idealised one.
Does GDPR apply to your organisation and purpose?
GDPR can apply where processing is connected to an EU establishment. It can also apply to an organisation outside the EU if it offers goods or services to people in the EU or monitors their behaviour. European Commission overview
If both answers are yes, assume the scrape needs a documented GDPR path. If the answer is unclear, do not use uncertainty as a green light—escalate it.
2. Write a one-page collection brief before the crawler runs
The simplest control is also the most valuable: specify what you need before collection.
Your brief should answer:
- Purpose: What concrete decision, service, or analysis requires this data?
- People and fields: Which categories of people could appear, and which exact fields are necessary?
- Source and access: Is the content freely accessible? Does the source object through terms, robots controls, login walls, or other technical controls?
- Use and recipients: Who will see the results? Will the data be enriched, exported, shared, used for direct marketing, or used for model training?
- Retention: When will the raw data, working files, and derived records be deleted or reviewed?
- Ownership: Who is the controller, who is the processor, and who handles rights requests?
This is not bureaucracy for its own sake. The GDPR principles require a specified purpose and data that is adequate, relevant, and limited to what is necessary. European Commission principles
3. Select and document a lawful basis—don’t assume one
Every personal-data processing activity needs a lawful basis. Consent may fit some products, but it is not the default answer for public web data. For some private organisations, legitimate interests may be a possible basis for narrowly scoped scraping with real safeguards. It is not automatic.
A defensible legitimate-interests assessment asks three questions:
- Is the interest lawful, specific, real, and present?
- Is this collection necessary for that purpose, or is there a less intrusive way to achieve it?
- Do the person’s interests, rights, or reasonable expectations outweigh your interest?
The CNIL says that publicly available data collected by scraping is generally considered through a legitimate-interests lens, but requires additional measures to reduce the impact on people. It also stresses a case-by-case analysis rather than a blanket permission. CNIL guidance
Document the analysis, its assumptions, and the mitigations you chose. “The profile was public” is context for the balancing test—not the test itself.
4. Make minimisation a technical requirement
The best compliant record is often the one your scraper never collected.
Build the following guardrails into the collection job:
- Allowlist fields. Define the fields you need; do not scrape every visible field because it is easy.
- Block sensitive categories. Exclude health, political, religious, union, sexual-life, biometric, and other special-category signals unless counsel has designed a specific lawful path. Ordinary text can unexpectedly reveal these categories.
- Exclude high-risk sources. Maintain a default exclusion list for support groups, health forums, children’s spaces, and other contexts where reuse could be surprising or harmful.
- Delete spillover quickly. If irrelevant personal data is captured, isolate and delete it rather than quietly retaining it “just in case.”
- Record provenance. Store the source URL, collection date, and relevant collection configuration with each dataset. That supports accuracy, deletion, and rights handling.
The CNIL explicitly recommends deciding relevant categories beforehand, filtering unnecessary or sensitive data, deleting irrelevant data, and respecting technical or legal opposition to collection. CNIL safeguards
5. Treat transparency as part of the product
Data collected from a website is usually collected indirectly. That means Article 14 transparency obligations may be relevant: explain who you are, the purpose, the categories and source of data, lawful basis, retention, recipients, transfers, and people’s rights.
The European Commission’s summary says that, where data is obtained from another source, information is generally due within one month, at first communication, or at first disclosure—whichever is applicable. There are exceptions, including cases where notification is impossible or would involve disproportionate effort, but they are conditional and should be assessed and documented rather than assumed. European Commission obligations
For broad collections, a clear public notice, a dataset page, a dedicated contact route, and easy-to-find instructions for objection, access, correction, and deletion can be more meaningful than a hidden legal page. The right format depends on the processing and risk.
6. Build a deletion and rights workflow before launch
Scraping at scale makes later cleanup expensive. Give the data an identifier, keep a controlled source-to-record map, and ensure you can locate and remove or suppress a person’s data across raw captures, databases, exports, indexes, and downstream processors.
At minimum, decide:
- who receives and verifies a rights request;
- how a record is located without asking for unnecessary extra information;
- how deletion or objection is propagated to downstream systems;
- how suppression prevents accidental re-collection;
- how long logs and backups retain the record, and what exception process applies.
If the dataset will feed a model, an enrichment graph, profiling, or direct marketing, make this plan more rigorous. The further the data travels, the harder it is to honour rights meaningfully.
7. Secure the dataset and assess high-risk cases early
GDPR requires measures proportionate to the risk, including protection from unauthorised access, loss, destruction, and unlawful processing. Privacy by design and by default means choosing those controls at the start, not after a breach. European Commission obligations
Useful baseline controls include role-based access, encryption in transit and at rest, secrets management, audit logs, vendor review, data-export controls, and a tested incident process. Pseudonymisation can reduce risk, but it is not the same as anonymisation and usually does not remove GDPR obligations by itself.
A DPIA should be considered before processing that is likely to create high risk, especially where you are combining these factors:
- large-scale collection or monitoring;
- profiling or decisions that affect people;
- special-category or highly personal data;
- children or other vulnerable people;
- combining datasets to draw new inferences;
- persistent identification, location data, or data-broker style reuse;
- AI training or a model that could memorise or expose personal data.
The Commission identifies systematic, extensive automated evaluation, large-scale sensitive-data processing, and large-scale systematic monitoring among the cases requiring a DPIA. DPIA guidance
A launch checklist for web-scraping teams
Before running a production job, confirm all of the following:
- We know whether the collection includes personal data and why GDPR applies or does not apply.
- We have a precise written purpose and an allowlist of required fields.
- We have documented a lawful basis and, if relevant, a legitimate-interests assessment.
- We have excluded sensitive and high-risk sources or categories by default.
- We have assessed source restrictions and do not bypass access controls.
- We have a transparent public explanation and a workable route for rights requests and objections.
- We know the controller/processor roles and have appropriate vendor terms.
- We have retention, deletion, suppression, and downstream-propagation procedures.
- We have proportionate security controls and incident ownership.
- We have completed, or consciously documented why we do not need, a DPIA and cross-border-transfer assessment.
The practical takeaway
GDPR compliance for web scrapers is not about finding a magic line in a robots file or pasting a disclaimer into your product. It is about making collection proportionate to a clearly stated purpose, giving people meaningful visibility and control, and being able to prove your choices later.
Start narrow. Collect less. Keep sources and timestamps. Build deletion into the data model. Escalate sensitive, large-scale, profiling, and AI-training uses before data flows downstream. Those habits make a scraper more trustworthy—and much easier to operate when the first privacy question arrives.
This article provides general information, not legal advice. Seek qualified advice for your organisation’s facts, jurisdictions, data categories, and intended use.
Learn More
