Google's Illyes: Robots.txt Can't Block AI Training, Only Crawling

Robots.txt, as the internet standard RFC 9309 defines it, governs which URLs a crawler may request — not what happens to the content afterward, Google's Gary Illyes said in a Search Central video published Wednesday. That distinction is the one marketing and SEO teams keep missing when they try to block generative AI systems from training on their pages.

The roughly 10-minute clip is the second episode of The Non-Canonical Cabinet, a new format in which Illyes and Cherry Prommawin of Google Search Relations take edge-case questions from the community. Illyes put the bottom line in plain words: "So as defined by RFC 9309, robots.txt is an access preference protocol." It "doesn't dictate what a system does with the data after it has already crawled it."

What a custom AI rule actually does

A novel directive sitting in a robots.txt file — the show illustrated one labeled "No-AI-Chicken-Training" — will be ignored by a parser. Google's robots.txt specification page lists only four supported elements: User-Agent, Allow, Disallow, and Sitemap. Invalid lines get skipped. RFC 9309's Section 2.2.4 backs that up, letting crawlers decide how to process anything outside the protocol's scope.

Where Google-Extended fits in

For Gemini specifically, the documented control is Google-Extended, a robots.txt token with no user agent of its own. Google still crawls under its existing user agents. The token governs whether content Google crawls can be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google's list of common crawlers calls Google-Extended "used in a control capacity" and adds that it doesn't affect a site's inclusion in Google Search and isn't a ranking signal. The Non-Canonical Cabinet hosts didn't discuss Google-Extended on air, though the video's description links to the same list.

The draft that would add a usage rule

Illyes is co-author, with Mozilla's Martin Thomson, of an IETF Internet-Draft that would give robots.txt a standard way to state usage preferences. The current version, dated Aug. 19, would update RFC 9309, if approved, with a Content-Usage rule and a matching HTTP header.

Under the proposal, Allow and Disallow keep governing which URLs a crawler may fetch. Content-Usage would define preferences for how crawled content gets used, and only on paths where fetching is already permitted. The draft's example file adds a Content-Usage field of "train-ai=n" to a rule that allows crawling.

The Working Group has not yet submitted the document for review by the IETF's Steering Group. The planned send date is August 2026. The current Content-Usage draft expires Feb. 20, 2027, unless a new revision replaces it. The "train-ai" label itself appears in a separate Vocabulary Draft dated Sept. 14, which states it reflects no Working Group consensus.

What marketers should do today

If a brand wants to keep a specific AI crawler off a site entirely, Illyes said, you have to "target their specific user agent tokens explicitly in the file." Disallow statements can stop particular AI crawlers from accessing particular webpages, but they cannot stop a permitted crawler from using what it fetched for training.

Prommawin added a separate note for international sites. Paths with non-English characters or emoji "should" use the percent-encoded form, she said, because "other or older crawlers on the web might get confused." Google's parser treats raw UTF-8 and percent-encoded paths the same way, but the on-screen example labeled the raw form "Works for Google, might break others" and the encoded form "The Safe, Standard Way." RFC 9309's grammar allows UTF-8 in paths, so the safer call for any site reaching multiple crawlers is the percent-encoded version.

So what for marketers

There is no standard way today to express a training preference independent of which bot may crawl a page. The closest is the Content-Usage draft, and its path to the IESG runs through August 2026. Until then, marketers who care about generative AI training have three levers: a Disallow rule against a named AI user agent, the Google-Extended token for Gemini, and a careful read of each vendor's documentation. One of those three covers most cases today; the others are still being negotiated.

The next Non-Canonical Cabinet episode, per the video description, will cover the 404 status code. Marketers waiting for a definitive answer on AI training can expect the conversation to keep running for at least another year.