OpenAI, Google and Microsoft back one robots.txt training opt-out

Website owners may soon be able to tell several AI companies “don’t train on my content” with one entry in one file. OpenAI, Google and Microsoft each said Friday they will honor or plan to support a shared signal in robots.txt, the file sites already use to give instructions to web crawlers.

None of the three has switched it on. OpenAI says it aims to do so “in the coming months.” Google says “in 2027.” Microsoft says it will start Feb. 1, 2027.

The rule in plain English

The signal is a made-up crawler name, __usage_ai-training. A site that wants to opt out of AI training for everything adds two lines:

User-Agent: __usage_ai-training
Disallow: /

A site can also opt out for one folder and allow the rest. The Internet-Draft gives an example that blocks training everywhere except a press-kit folder.

The draft draws on a definition from the IETF’s AI Preferences working group. “AI Training” means using content “to modify the learned parameters of a generative AI model.” Training models used only for tasks like ranking or classification is not covered.

Who wrote it

The proposal is an individual Internet-Draft by Kevin Kelley, who lists Anthropic as his affiliation. The IETF page says the document “is not endorsed by the IETF and has no formal standing in the IETF standards process.” It expires May 3, 2027, unless it is updated. The draft thanks dozens of people, including staff at Google, Microsoft, OpenAI, Meta, News Corp, the BBC and the Financial Times. It says affiliations are listed “for identification only” and does not say any of those companies will implement it.

Who it binds and who it exempts

It binds no one. OpenAI calls its move a “voluntary effort.” The draft covers only companies that choose to read the signal.

Two details matter for site owners. First, a general User-Agent: * rule does not count, under the draft. A blanket rule will not be read as a training opt-out. Second, if the file has no __usage_ai-training line, “no preference has been expressed.”

Older company-specific controls stay. OpenAI says its existing crawler controls “will continue to apply.” Google says an opt-out through the shared token, where Google-Extended is not named, will be read as if the site had blocked Google-Extended.

What changes for site owners and founders

Microsoft goes furthest. It adds a separate page-level “AI Grounding” setting that governs whether content can feed answers in Bing, Copilot and Microsoft 365. Options are Allow, Disallow and a new DisallowExtended, which keeps content out of grounding services Microsoft provides to third parties. The new settings replace Microsoft’s older NoArchive and NoCache tags. Sites using those tags must act by March 1, 2027, or lose their restrictions. Microsoft says the choices do not affect Bing search eligibility. “Neither choice guarantees inclusion, traffic, or payment,” it writes.

Google says sites that specifically allow Google-Extended can appear in Gemini Apps answers through grounding.

Our read

This is a real cut in busywork. Today a publisher that wants out of AI training has to track crawler names company by company. One line is simpler, and three of the largest AI developers agreeing on it is notable. Google ties the effort to the EU’s code of practice for general-purpose AI, which recognizes robots.txt as a main way for publishers to state preferences. That suggests regulatory pressure, not only good will, is pushing the companies toward a common answer.

But the limits are just as plain. The signal is voluntary. It covers training, not the live answers that now compete with publishers for clicks; only Microsoft offers a matching grounding control, and it warns that using it will cut a site’s visibility in AI products. Nothing here creates payment or licensing. The earliest firm date is Feb. 1, 2027. And the underlying document is an individual Internet-Draft, not an IETF consensus. For AI labs, a shared signal costs little and gives them a common answer when regulators ask how sites can say no. For publishers, the hard trade between exposure and control remains.

What’s next

Microsoft opens temporary grounding settings in Bing Webmaster Tools on Dec. 15, 2026, for sites using NoArchive or NoCache. Its new preferences take effect Feb. 1, 2027, with a March 1 deadline for those sites. Temporary settings expire by Sept. 30, 2027, or sooner when a page is recrawled. OpenAI says it will post implementation details later. Google says it will keep working on the proposal through the IETF working group.

Sources: OpenAI; Google Search Central; Microsoft Bing; IETF Internet-Draft