AI bots scrape thousands of publisher pages for every single human reader they send back, according to Cloudflare. That exchange rate defines the core problem in the dispute between publishers and AI companies over content access.

Lara Cohen, VP of strategic partnerships for media, creators and AI at Cloudflare, laid out the imbalance this week on the AdExchanger podcast "Making AI Pay Its Way, With Cloudflare."

"That's not what publishers agreed to when they started letting search engines crawl their sites in exchange for traffic," Cohen said.

What broke the original bargain?

The original bargain was simple. Publishers let search engine bots crawl their sites. In return, search engines sent human readers to publisher domains. Those readers viewed ads, bought subscriptions and sustained publisher revenue.

AI crawlers (software programs that copy web content to train or feed large language models) have inverted that bargain. They take content at massive scale. They return almost no traffic, because AI products answer questions without sending users to the source page.

Cloudflare's measurement puts a number on the gap. For every human reader the AI ecosystem returns, thousands of pages get scraped. The ratio is not marginal. It is orders of magnitude worse than what publishers accepted from search engines.

Why does Cloudflare sit at the center of this?

Cloudflare operates infrastructure that carries a large share of web traffic. That position lets the company see crawler behavior at scale, which is why its ratio claim carries weight beyond a single publisher's server logs.

The same position gives Cloudflare leverage. Publishers cannot easily block AI bots one by one. Cloudflare can act as a gatekeeper between crawlers and publisher content at the network level.

Cohen's title itself signals the company's direction. Her remit covers strategic partnerships across media, creators and AI, and the podcast segment frames the question as making AI "pay its way" rather than shutting it out entirely.

What does this mean for the traffic-for-content model?

The implications for publishers are structural, not cosmetic:

• The traffic-for-crawling model no longer balances. Input (scraped pages) and output (returned readers) differ by a factor of thousands.
• Publishers lack individual leverage against AI crawlers. Blocking at scale requires infrastructure-level action.
• Compensation, not traffic, becomes the plausible exchange. If AI companies will not send readers, they must pay for content.

Cohen's framing makes clear the publishers never signed up for this. The deal was traffic. The deal is not being honored.

The question the industry now faces is what replaces it. Cloudflare's position between publishers and AI companies suggests payment-for-content mechanisms, enforced at the network layer, will be the next test of whether the web's content economy can be rebuilt around AI rather than against it.

By Elena Vasquez