Cloudflare Pilot Uncovered Hundreds of Millions of New Pages for ChatGPT Search

By Tom Whitfield

Cloudflare and OpenAI say a research pilot has helped OpenAI discover and index hundreds of millions of new web pages. The pilot drew on a participating cohort of about 1% of Cloudflare's total network. The companies shared the early results on Oct. 15 in a Cloudflare blog post coauthored by two OpenAI staff, more than three months after Cloudflare announced the pilot on July 8.

The post does not report how often the new pages were cited in ChatGPT or how many visits they sent. It also concedes that "finding new pages can initially increase crawling as those pages are evaluated and indexed."

How do Cloudflare's signals help OpenAI find pages?

OpenAI uses Cloudflare's network signals to steer its search crawler toward pages with value. The blog post cites examples:

• The user guide of a particular washing machine
• Current passport requirements and fees
• A restaurant's menu and hours

These pages "can be valuable even if few other sites link to them," the post states. The July 8 announcement listed the signals in play: how recently a page was updated, the quality of traffic to a website, and what updates were actually made. The information comes from participating websites, though neither announcement says who participated or whether the new pages sit on their sites.

The takeaway: network-level traffic data, not links, now decides which pages AI search can see.

What do the signals do to crawling?

The Oct. 15 post says the signals also help OpenAI's crawlers avoid unnecessary requests. Bots repeatedly request files that no longer exist on servers. Some websites redirect nearly any address to their homepages. Other addresses serve duplicate copies of the same content. Change signals tell the crawler whether a page warrants a revisit or holds no value.

The companies are also testing signals that flag harmful pages. Their example: a fake login form an attacker planted on one page of a legitimate site. The goal is to flag that page, not the whole domain. Early results suggest the signals can identify pages to keep out of the index, the post says.

The stated aim is to reduce unnecessary load for Cloudflare customers. The post provides no figures on crawl volume or server load.

Do crawler settings still decide access?

Yes. The Oct. 15 post states plainly: "A signal about a page doesn't grant permission to access it." The researchers use metadata from the web to improve search and discovery, not page content for model training, the blog says.

OpenAI's crawler documentation splits search and training into two robots.txt directives:

• OAI-SearchBot gathers information for ChatGPT's search results. Block it and a site drops out of those results, though it may still appear as a navigational link.
• GPTBot collects content that may be used for training. Allow both bots and OpenAI may use data from one crawl for both purposes.

Allowing OAI-SearchBot while blocking GPTBot keeps a site in ChatGPT search results while excluding its content from training. OpenAI offers this exact configuration as an example on its crawler page. Publishers can monitor which AI services hit their sites through Cloudflare's AI Crawl Control, available on every plan.

Why does this matter for marketers?

Once a page enters OpenAI's index, ChatGPT has another source to pull from. The companies state that OpenAI independently evaluates and ranks pages, so discovery through the pilot guarantees no citation.

Measurement remains fragile. Tracking ChatGPT visits depends on a UTM tag or a referrer, and some ChatGPT links lacked both earlier this month, as Search Engine Journal reported Oct. 15. One analyst said a fix began rolling out that day.

What comes next?

The next stage covers three tracks: improving how an AI selects a web page, testing faster and more reliable change indicators, and measuring results when the AI finds new content. Cloudflare says it is working with OpenAI to determine how improved content discovery could translate into business outcomes.

A July 1 Cloudflare blog post said the company intends to provide broad access to the capability at some point during the remainder of 2026. The Oct. 15 announcement named no date. Until then, OpenAI's crawler activity on any site shows up in server logs and in AI Crawl Control.