You Blocked GPTBot. That Is Not a Legal Right.
Two lines in robots.txt is the most common AI content policy in existence, and it is the one thing in the stack with no statute behind it. The mechanism that does have a statute takes a different form, and most sites that blocked the crawler never published it.
robots.txt is a voluntary convention, not law, and no US statute makes a Disallow line enforceable on its own. The enforceable opt-out lives in EU law: Article 4 of the 2019 Copyright Directive allows text and data mining of lawfully accessible works unless the rightsholder expressly reserved the use in an appropriate, machine-readable manner, and the EU AI Act requires general-purpose model providers to have a policy for identifying and respecting those reservations. A block tells a crawler what to fetch. A reservation tells the world what may be done with the work. The first is a request; the second is the thing a regulator or a court can act on — and it has to exist before the crawl.
Six Mechanisms, One of Them Statutory
Discussions of AI opt-outs collapse very different instruments into one word. They are not interchangeable: they differ in what they govern, who they bind, and whether anything happens when they are ignored. Sorting them by force rather than by popularity produces an uncomfortable ranking, because the most widely deployed mechanism sits near the bottom.
The US Theories All Need the Same Missing Element
Breach of contract, trespass to chattels and the Computer Fraud and Abuse Act look like three independent routes. In practice they converge on one question: was the access unauthorised? For content that sits on the open web with no login, no rate limit and no acceptance step, that question has repeatedly been answered in the scraper's favour — public means accessible, and a footer link nobody clicked does not form a contract.
This is why robots.txt keeps appearing in complaints despite having no independent legal force. It is the cheapest available proof that the refusal was published, dated and visible to the party that crawled anyway. It does not win the case. It supplies the fact without which the case has no shape.
Blocking Is About Identity. The Problem Is Use.
Every named-agent block encodes a guess about which company will fetch your content and under what string. That guess decays. Agents get renamed, vendors split training from retrieval, new entrants ship crawlers nobody has heard of, and none of it touches a dataset that a third party scraped last year and now licenses. A reservation phrased in terms of use — no text and data mining, no AI training, retrieval with citation permitted — does not need to know who is asking.
Training, live retrieval inside an assistant answer, and traditional search indexing have different business value. Most publishers want to block the first and keep the other two, and a single blanket block gets that wrong.
One paragraph in the terms expressly reserving text and data mining and AI training rights. This is the piece that survives crawler renames and third-party dataset resale.
The EU standard is not satisfied by prose alone for online content. Mirror the reservation in robots.txt, in a policy file and in response headers so a crawler encounters it without parsing English.
Now the Disallow lines, per use rather than per company. Treat the agent list as a quarterly maintenance item, not a one-time task.
Capture the file and the terms with a third-party timestamp on the day they go live. The disputed fact will be when the refusal was published, not whether it was.
Server logs tell you which agents honoured the file and which fetched anyway. A crawler that ignored a Disallow is the fact pattern worth documenting.
The Block That Costs You Money
There is a version of this policy that is strictly worse than doing nothing. A site blanket-blocks every AI-associated user agent, which removes it from live retrieval — the mechanism by which an assistant fetches a page and cites it in an answer — while leaving every historical training set exactly as it was. The site loses the referral channel, gains no removal from any model, and now ranks below competitors who understood that retrieval and training are different products from the same vendor.
The decision worth making is per use, not per company. Training is the one most publishers want to refuse. Retrieval with attribution is the one most publishers should want, because it is the closest thing the current landscape offers to a referral. Search indexing was never the argument. Collapsing all three into one Disallow block is how a policy meant to protect the business ends up shrinking it.
Six Controls Worth Having
Publishing a block feels like a resolution of the question, and it closes internal debate. Legally it is a request that a compliant party will honour and a non-compliant party will not, and it says nothing at all about datasets already scraped, syndicated copies or third-party resellers.
The EU test asks whether the rightsholder expressly reserved the text-and-data-mining use. Language drafted as a prohibition on visitors can read as a conduct rule aimed at users rather than a reservation of rights in the work. Reserve the use explicitly and name text and data mining and AI training.
Retrieval agents that fetch a page to cite it in an assistant answer are a referral channel. Blocking them to make a statement about training removes citations while leaving historical training data untouched — you pay the cost and get none of the benefit.
If the content is genuinely the asset, a login or a paywall does what no policy file can: it stops the fetch, creates provable assent to terms, and makes an unauthorised-access theory plausible. Everything upstream of that is signalling.
Your own file timestamps are yours to edit and will be treated that way. A public archive snapshot of robots.txt and the terms page on the day the reservation went live costs nothing and converts an assertion into a record.
Agent names change, new ones appear, and vendors split training and retrieval agents without announcement. A file written eighteen months ago is blocking a roster that no longer matches the crawlers actually hitting your logs.
Related Reading
- Scraping, the CFAA and contract liability — the same authorisation question from the crawler's side of the table.
- RAG, retrieval and news licensing — why live retrieval is a different legal event from training.
- What the training-data lawsuits have actually decided — the fair use questions that a reservation does not answer.
Start With What Your Own Site Is Signalling
Most sites that believe they have an AI policy have a robots.txt file written by someone who has since left, and terms that never mention text and data mining. The two documents frequently contradict each other.
See what your public pages and policy files are actually saying. Run a free scan and reconcile your crawler directives against your published terms.
This article is general information and not legal advice. The enforceability of crawler directives, terms of service and rights reservations varies by jurisdiction and by the facts of access, and the law in this area is actively developing on both sides of the Atlantic. Consult qualified counsel before relying on any opt-out mechanism.