Google's 503 Advice, Robots.txt Limits And AI Agent Signals

By Priya Raman

Google Search Central told site operators on Oct. 21 to return a 503 status code when taking servers offline for PHP patches, because the code "ensures already indexed URLs are preserved in Search while you work." The LinkedIn post appeared the same day PHP released emergency fixes for two critical flaws. Google's crawl-rate documentation advises limiting 503s to no longer than one to two days, because the company may drop a URL from its index if the code persists longer.

Google's John Mueller shared a related reminder on Bluesky: the 503 does not have to be served through PHP. According to Google's guide to pausing a site, if you trigger a 503 in response to robots.txt requests, all crawling is blocked. Site operators should avoid general 404 or 200 error pages during planned downtime.

What does robots.txt actually control?

Gary Illyes of Google stated in an Oct. 21 Search Central video that, according to RFC 9309, standard robots.txt directives only dictate what compliant crawlers are allowed to crawl. They do not control how systems use content after it has been crawled. Illyes called robots.txt "an access preference protocol."

Keeping one AI crawler out while allowing others means targeting that crawler's user-agent token. Google's list of common crawlers documents Google-Extended as the token for managing whether content Google crawls may be used for training future Gemini models, for grounding in Gemini Apps and on Vertex AI. The token does not affect inclusion in Google Search, is not a separate crawler and has no HTTP user-agent string of its own. A draft Illyes co-wrote for the Internet Engineering Task Force would add a Content-Usage rule to robots.txt for stating usage preferences, but it remains a working group document and has not been approved. A standard way to express a training preference separately from crawl access is, for now, available only as a draft.

What did the Cloudflare-OpenAI pilot change?

Cloudflare and OpenAI said a research pilot helped OpenAI discover and index hundreds of millions of new web pages, giving ChatGPT Search more information to draw on. The pilot drew on approximately 1% of Cloudflare's network, using signals that help the OpenAI crawler identify pages, bypass non-existent pages and detect updates.

The Oct. 15 blog post, co-authored by two OpenAI employees, states that finding new pages "can initially increase crawling" and does not report how often the newly identified pages are cited or how many visits they generate. The post also stated: "A signal about a page doesn't grant permission to access it." Because OpenAI ranks pages on its own, discovery through the pilot does not guarantee a citation.

OpenAI's crawler documentation splits search from training. A site can use robots.txt entries such as Allow for OAI-SearchBot and Disallow for GPTBot to signal that its content should not be used for training purposes.

Are raw and encoded brackets the same URL?

No. John Mueller said on Bluesky on Oct. 19 that square brackets in a URL and their percent-encoded forms, %5B and %5D, create different URLs. He added that he would have assumed they were equivalent and "probably gave people the wrong answer on this in the past." Mueller referenced RFC 3986, the URI syntax standard, which reserves brackets and plus signs.

If a site returns the same content for either version, Google will crawl each, eliminate duplicates and select one. Mueller's advice: "pick one," and use it consistently across links, canonical tags, sitemap entries and structured data. Existing sites using raw brackets throughout should keep working, he said, though he "wouldn't set up a new site like this."

Can you serve JPEG XL images?

Not to Google Images or Discover previews. Mueller shared on Bluesky on Oct. 15 that JPEG XL is not currently supported in either. His post followed JPEG XL support in Chrome 155, rolled out widely on Oct. 6, and in Firefox 158 on Oct. 13. Google's image SEO documentation lists BMP, GIF, JPEG, PNG, WebP, SVG and AVIF as supported formats for an img element's src attribute, and did not include JPEG XL as of Oct. 22.

Asked about fallbacks in a picture element, Mueller said the element includes an img element, "so that would work for any search engine." Serving JPEG XL as a picture source, with a JPEG, WebP or AVIF file in the img element, keeps a supported format in src. Google's Discover guide points to og:image and schema.org markup for preview images, so those should point to one of the listed formats too.

What did Pichai say about AI agents?

Google CEO Sundar Pichai told Alex Heath on the Sources podcast that he expects AI agents to reach websites both by browsing pages like people do and through protocols designed specifically for agents. "I think you have both ways and they're going to develop," Pichai said. He pointed to momentum behind agent and payment protocols, including the Universal Commerce Protocol that Google backs, and called protocols "a far more scalable approach" that will take longer to develop. He also said he expects agents to provide a unifying experience across Search and Gemini. Notably, Google's own hosted agents browse as Google-Agent, one of the user-triggered fetchers that Google says generally ignore robots.txt rules.

What does this change for practitioners?

Each story this week concerns a signal a crawler or agent receives about a site and where it may reach. A 503 tells Google the outage is temporary, within the documented one-to-two-day window. A disallowed crawl directive keeps compliant crawlers from crawling page content, but later use depends on controls like Google-Extended and GPTBot. A Cloudflare discovery signal grants no access permission. Mixed raw and encoded brackets determine whether Google sees one URL or two. Server logs show which crawler has been making requests, but which controls govern crawling and later use remains documented by each company separately. For now, the practical rule is simple: verify each signal against its vendor's own documentation before assuming it covers use, not just access.