Is scraping publicly available data legal?

Accessing a page that requires no login is materially different from bypassing an authentication or technical barrier, and that distinction has driven the computer-crime analysis in the leading scraping cases. But 'not a computer crime' is not the same as 'no liability' — the same collection can still breach the site's terms, misappropriate a protected database, or violate privacy law depending on what was taken and how.

Does ignoring robots.txt create legal liability?

robots.txt is a convention rather than a statute, so ignoring it is not itself unlawful. It matters as evidence: a plaintiff uses a disallow directive to show the operator communicated an objection, which supports contract, trespass, and unjust-enrichment theories and undercuts any argument that the collection was welcome. Publishers have also begun mirroring those directives into their terms of service, which converts a convention into a contract term.

Are we liable for a dataset we bought rather than collected?

Purchasing does not launder provenance. Privacy statutes reach the controller that processes personal data regardless of where it was acquired, and a right-to-delete request against your model or index is your obligation, not your vendor's. Contractual indemnities help only to the extent the vendor is solvent and the indemnity actually covers the claim, so read the cap before relying on it.

Does retrieval-augmented generation avoid these issues since we don't train on the data?

It changes the copyright argument and does very little for the others. Building a retrieval index still involves systematic collection, storage, and reuse of another site's content, and if that content includes personal information you now hold a database of it. RAG pipelines also tend to surface source text closer to verbatim than a trained model does, which cuts against the transformation arguments teams expect to rely on.

What do enterprise buyers ask about data provenance now?

Security questionnaires increasingly include the sources of training and fine-tuning data, whether any of it was scraped, whether personal data is present, and whether the vendor will indemnify claims arising from it. Companies that cannot answer lose deals for reasons unrelated to any regulator, which is why provenance documentation has become a revenue issue before it becomes a legal one.

Scraping Data to Train or Feed AI: CFAA, Contract, and Vendor Risk in 2026

Contract Is the Claim That Actually Gets Filed

Terms of service on virtually every commercial site now prohibit automated collection and, increasingly, name AI training as a specifically forbidden use. Breach of contract is unglamorous, but it is the easiest of these theories to plead: the plaintiff shows the terms, shows the access, and does not have to establish that anything taken was protectable or that a computer was accessed without authorization. Where an account was created to reach the data, the terms were affirmatively accepted, which removes the browsewrap-notice argument entirely.

This is why the practical question for a data pipeline is not "is this public?" but "how did we get to it, and what did we agree to on the way?" A crawler hitting open pages anonymously, a script logged into a free account, and a scraper rotating credentials to evade rate limits occupy three very different legal positions while producing an identical CSV.

Computer-Crime Statutes Turn on Barriers, Not Volume

Federal and state computer-fraud statutes have narrowed toward a gates-up-or-gates-down framing: the serious exposure attaches to circumventing an authentication or technical barrier rather than to reading pages anyone can load. That is meaningfully protective for straightforward crawling of open content, and it is also where engineering decisions quietly create risk — sharing credentials across scrapers, rotating IPs specifically to defeat blocks, or continuing after receiving a cease-and-desist all move a project toward the wrong side of that line. State statutes are not uniform and several are broader than the federal one, so a single national analysis understates the range.

Personal Data Is the Obligation That Never Expires

Once scraped material includes names, contact details, biometric identifiers, or profile content about identifiable people, privacy law applies to your processing regardless of whether the collection was otherwise lawful. That means notice obligations you cannot satisfy against people who never visited your site, a lawful-basis analysis that legitimate interest does not automatically survive, and deletion and access rights against a dataset that has already been folded into model weights or an embedding index. The engineering answer to a deletion request is much cheaper when source-level lineage was recorded during ingestion, and close to impossible when it wasn't.

Provenance Diligence Has Moved Into the Sales Cycle

The most immediate consequence for most AI companies is commercial rather than legal. Enterprise security reviews now ask where training and fine-tuning data came from, whether any of it was scraped, whether personal or licensed content is present, and whether the vendor indemnifies claims arising from it. A startup that cannot produce a source inventory does not get an enforcement action — it gets a stalled procurement cycle and a competitor who documented theirs. Treating provenance as sales collateral rather than legal overhead tends to get it funded.

Data Pipeline Audit Checklist

Run this against every source feeding a model, a fine-tune, or a retrieval index.

Build a source inventory: every domain or dataset, how it was acquired, and the date range collected— Start here

Read the terms of service for each high-volume source, checking specifically for AI-training and automated-access clauses— Essential

Flag any source reached through an account, a paywall, or a rate-limit workaround — those carry different risk than open crawling— Essential

Identify which sources contain personal data, and confirm you can trace a record back to its source for deletion requests— Essential

Record and honor robots.txt and AI-crawler directives, and keep the logs that show you did— High value

Review purchased-dataset contracts for provenance representations, indemnity scope, and liability caps— Essential

Establish a documented response process for cease-and-desist letters, including immediate suspension of the named source— Ongoing

The Asymmetry Worth Noticing

Companies building AI products are usually on both sides of this. The same business scraping competitor catalogs for a training set is watching its own content get ingested, and the terms it writes to stop that are the terms someone else will use against it. A coherent position — publish clear machine-readable access rules, honor other publishers' rules, keep the logs — is cheaper than maintaining two contradictory legal theories, and considerably easier to defend when a letter arrives.

Know what your own site exposes before you audit anyone else's

The same crawl pass that inventories your pages surfaces the accessibility defects that turn into complaints. RatedWithAI scans your site and reports the issues that actually create legal exposure.

Scan Your Site for Free →

Frequently Asked Questions

Does scraping only facts rather than creative text avoid the problem?

It weakens the copyright theory, since facts themselves are not protected, and does nothing for contract or privacy claims. It also does not necessarily avoid database and misappropriation theories, which protect the investment in compiling a collection rather than the originality of its entries.

We only scrape a few thousand pages a month. Is that too small to notice?

Volume affects the practical likelihood of a complaint, not the legal analysis, and modest volume from a single identifiable company is easier to attribute than a large anonymous crawl. Small operations are also less likely to survive the cost of a dispute, so the risk-adjusted exposure is not proportionally smaller.

Can we rely on our scraping vendor's indemnity?

Read the cap and the carve-outs first. Many data-vendor indemnities exclude privacy claims, limit liability to fees paid, and cover only third-party IP claims — which is precisely the theory least likely to be the one you face. An uncapped indemnity from an undercapitalized vendor is also worth what the vendor can pay.

Does using an open dataset published by a research group protect us?

Research-license terms frequently prohibit commercial use, and inheriting a dataset does not inherit any lawful basis for the personal data inside it. Check the license against your actual use, and check whether the compilers documented their own sources — many did not.

What should we do when a cease-and-desist arrives?

Stop collecting from that source immediately and preserve the logs rather than deleting them. Continued access after clear notice is the fact that converts a contract dispute into a stronger computer-crime or trespass claim, and it is the single most damaging thing a company can do in the week after the letter.