AI Policy Token: Four Companies Back Draft Publisher Signal

By Priya Raman

According to Digiday, Anthropic, Google, Microsoft and OpenAI have backed a proposed “policy token” that lets publishers set AI-training preferences in robots.txt with just a few lines of text. The proposal, written by Kevin Kelley, a technical lead for standards at Anthropic, replaces crawler-by-crawler instructions with one common preference.

Publishers have tried blocking AI bots, pursuing legal action and negotiating licensing deals, Digiday said.

Under the proposal, publishers would add User-Agent: __usage_ai-training to robots.txt, the file sites use to direct web crawlers, then write “allow” or “disallow.” Permissions can include exceptions, such as allowing a press kit.

What does this change for publishers?

The immediate change is administrative. Publishers could set a broad training preference instead of listing every crawler and granting or denying access separately. That simplifies declarations, but it does not create enforcement, Digiday said. The token joins existing defenses such as CDN-level blocking and tools like TollBit.

Kelley submitted the proposal to the Internet Engineering Task Force (IETF), an internet standards development organization, on Oct. 30. Digiday reported that all four companies participate in the IETF’s AI Preferences (AIPREF) effort, which is developing standards for website owners to express preferences about AI use. The draft remains unofficial pending IETF deliberation.

Alex Springer, technical lead at SPUR, a publisher coalition, criticized the companies’ unilateral support. “The IETF exists to allow for individuals to work in public to form consensus about the standards and protocols that make the internet work,” he said.

“The way that the platforms have approached this by announcing their unilateral support of a ‘standard’ that is an unreviewed draft is the antithesis of that,” Springer said. He said the approach risks forcing a de facto standard that falls short of content owners’ requirements.

Adoption starts before consensus

• Anthropic: The company says it is rolling out adoption of the signal now.
• OpenAI: It will honor the token “in the coming months,” but said the mechanism will not replace its existing crawler controls.
• Google: The company aims to support the token in 2027. Its plan builds on Google-Extended, the robots.txt crawler control for AI-training access that it introduced in 2023.
• Microsoft: It will comply and let publishers opt out of AI grounding, which uses crawled sites to find real-time information. Starting Feb. 1, 2027, those preferences will replace NoArchive for AI training and NoCache for grounding.
• Meta: It did not commit to honoring the token. Company spokesperson Andy Stone wrote on X that Meta strongly supports publishers’ ability to express preferences and a common signal across AI providers.

ScalePost tracked activity across more than 50 publisher properties and shared its findings with Digiday. Crawlers operated by the four companies backing the token made roughly 1 billion requests over the previous 30 days, but only 0.15% sought robots.txt files. The data suggests crawlers rarely inspect publisher directives before scraping.

Paul Bannister, chief strategy officer at Raptive, called the token “window dressing.” He said it was useless marketing, arrived too late and did not solve the problem. Digiday said AI providers also face copyright-infringement lawsuits, regulatory pressure and proposed legislation targeting crawlers.

Despite that gap, four commitments make this one of the first proposed standards to draw support from such a large group of AI companies, Digiday said.

Bannister compared the token to a front-door lock: “Even though it doesn’t keep all the thieves out, you’d still put a lock on your front door because it keeps some of the thieves out…” He added: “You do lots and lots of things to minimize the chances that your house gets broken into, but you cannot prevent it completely.”

He also said: “Is it one other arrow to put in your quiver of about 1,000 things you need to do? Sure.”

Visibility remains the trade-off

The visibility effect is unclear for most publishers, Digiday said. Many are working to improve their presence in AI tools, and some sell GEO products to brands seeking more mentions and citations in large language models. Blocking training or grounding crawlers can therefore carry a commercial cost.

Microsoft alone among the four backers explicitly addressed visibility. Its blog warned that disallowing crawlers for AI grounding means “you will see reduced brand visibility in AI experiences.”

Digiday also cited publisher concern that Google’s AI opt-out controls, which exclude content from AI-powered search features, can sacrifice visibility and traffic publishers cannot afford to lose.

Training controls leave inference open

The token does not govern inference, also called AI usage, when a model generates responses or performs tasks by retrieving or processing publisher websites in real time. Digiday said that activity has become central to most recent AI licensing deals between publishers and technology companies.

Compliance with robots.txt also does not prevent an AI provider from obtaining the same material elsewhere. Bannister said no major provider has promised to reject data bought from third-party scrapers or open-source crawls such as Common Crawl.

“None of them have said, ‘We won’t buy data from third parties that scrape the content that don’t obey this,’ or ‘We won’t take open source crawls like Common Crawl,’” Bannister said. “They’re going to get all the data anyway, just through a different mechanism, and so it’s completely useless.”

Springer said the draft appears to reinforce default training permission unless a publisher explicitly opts out. He argued that this “opt in unless opted out” approach risks legitimizing crawler access to AI training rather than requiring direct permission from content owners.

The next test is implementation, not endorsement. OpenAI’s rollout in the coming months, IETF deliberations, and Google’s and Microsoft’s 2027 deployments will show whether providers honor explicit preferences and whether third-party crawl data can still bypass them. Until then, publishers gain a simpler declaration mechanism, not reliable control over AI training.