Google, Microsoft and OpenAI Back Shared robots.txt AI Opt-Out

Google, Microsoft and OpenAI announced on Friday that they plan to honor a single shared robots.txt line for AI training preferences — a change that would let a site state one preference to every participating provider instead of writing rules for each company's crawler.

None of the three companies supports the line today. The signal comes from an Internet Draft submitted the same day by Anthropic's Kevin Kelley to the AI Preferences working group at the IETF. The IETF's document tracker lists it as an individual draft, not an official standardization activity. Kelley wrote in a working group email that Anthropic has started development to support it.

How does the shared line work?

Kelley's draft adds no new directive to robots.txt. A site writes an ordinary rule group under a policy token, __usage_ai-training, using the familiar Allow and Disallow. The draft's example opts out of AI training site-wide except for a press-kit folder:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The token's meaning rests on the working group's vocabulary draft, which defines AI Training as using content to modify a generative AI model's learned parameters. Models that only do non-generative tasks like classification or ranking are exempt.

Existing rules don't carry over. A wildcard group (User-Agent: *) can never match the token, and a file without a __usage_ai-training group signals no preference — which Kelley says is true of every robots.txt currently published. The token doesn't control crawling either. One line could replace a stack of per-company rules, if each company reads it.

What does Google's version do?

Google's post, by John Mueller, says a site that opts out via the shared line without specifying Google-Extended will be treated "as if the publisher had disallowed Google-Extended." That control also applies to grounding within Gemini Apps. Google aims to support it in 2027; its crawler documentation says early 2027.

Google-Extended governs whether content trains future Gemini models and whether it grounds answers in Gemini Apps and Vertex AI. Grounding means supplying content from the search index when the model answers. Sites that specifically allow Google-Extended are eligible to appear in Gemini Apps responses. Google-Extended doesn't affect Search inclusion — AI Overviews and AI Mode have a separate opt-out in Search Console that rolled out worldwide in August.

Why is Microsoft's approach different?

Microsoft's post, by Bing principal product manager Krishna Madhavan, adopts the shared line for training and adds a separate AI Grounding preference set only at the page level. A meta tag or HTTP header can override robots.txt per page. Grounding defaults to Allow; Disallow keeps a page out. A new DisallowExtended option allows grounding in Microsoft tools like Copilot and Bing's AI Mode but blocks Microsoft-provided grounding services used by other businesses (Web IQ). Microsoft said either restriction limits brand visibility in AI answers and has no effect on Bing search listings.

Microsoft is retiring NoArchive and NoCache. It begins honoring the new options on February 1, 2027, and stops recognizing the old tags as training or grounding controls after March 1, 2027. Sites using the old tags that want out of grounding must record a temporary preference in Bing Webmaster Tools; the setting opens December 15 and is only available to sites still using those tags on that date. Microsoft's FAQ says pages without a preference by March 1 default to allowing grounding until recrawled. Once Microsoft detects either new preference, it disregards remaining NoArchive or NoCache tags.

Where do OpenAI and Anthropic stand?

OpenAI aims to honor the signal in the next few months, with its current robots.txt policy still in effect. Neither its blog post nor its crawler documentation, checked October 30, says what happens when a shared-line rule conflicts with a GPTBot rule. Anthropic had no newsroom posting as of October 30; Kelley wrote that implementing support across all Anthropic services will take time. Anthropic's crawler-help page lists only three controls: ClaudeBot, Claude-User and Claude-SearchBot.

The opt-out won't be uniform. Microsoft splits training from grounding; Google bundles both under Google-Extended, and its documentation describes no way to block Gemini training while still allowing Gemini Apps grounding. Google, OpenAI and Kelley all say existing controls keep working — so NoArchive and NoCache users have homework first.

OpenAI's post encourages other AI providers to honor the signal, and Kelley wrote that he hopes others join. Google and OpenAI promise more details as support arrives. Kelley's draft expires May 3, 2027, unless replaced, and he plans to propose his design as a change to the Content-Usage draft that the group set aside in October.