Publishers Take AI Bot Fight Transatlantic With Disclosure Bills

Scraping accounted for 70.9% of malicious bot traffic analyzed by DataDome in its September report, and it rose 185.2% over the study's 12-month period. Publishers who once fought the flood with technical blocks now want laws that force AI crawlers to identify themselves — on both sides of the Atlantic.

How big is the scraping problem?

The numbers from individual publishers are stark. Reach CEO Piers North told Digiday his company has seen over 100 million successful scrapes of its content in the last six months. People Inc. now allows through just 38 bots a day among "tens of millions" of AI crawler requests that are blocked, chief innovation officer Jon Roberts told delegates at Digiday's Media Strategies London event.

Roberts described an escalating cat-and-mouse game that has outrun voluntary defenses. "They come in with a named crawler; we block it. They come in with an anonymous crawler; we block it. They pretend to be human; we block it. Then they try to be Google's crawler… we block that, and then they come through people's home proxies," he said. "This is where governments start getting involved."

At the same event, Financial Times md of customer revenue Fiona Spooner said the paywalled title is plagued by bots daily and dedicates resources to separating bad bots from good ones. One account alone tried to access about a million articles in eight hours from roughly 700,000 IP addresses. "We are being scraped and content taken and attempted all the time, as well as humans trying to get around it and access it in different ways," said Spooner.

Who is really knocking on publishers' doors?

Identity fraud among crawlers is widespread. A recent Piano analysis of Cloudflare data found 70% of bot traffic is unverified. Centinel, examining crawler identity claims across the news and media sites it protects from late June through Sept. 21, found crawlers routinely impersonate other bots:

• On a typical site, one in seven requests claiming to be Googlebot failed verification.
• 84% of requests claiming to be GPTBot were identified as fake.
• 99% of requests claiming to be Perplexity-User were identified as fake.
• 94% of fake Googlebot requests copied Google's actual crawler version string.

The findings build on Digiday's April reporting, which documented a surge in AI bot activity and a growing ecosystem of third-party scrapers harvesting and selling publishers' content without permission or compensation.

What would the legislation change?

The News/Media Alliance organized more than 300 publishing executives to lobby Congress in September in support of the federal Stealth Bot Prohibition Act, which would force stealth crawlers to clearly identify themselves and their purpose. U.K. publishers are now pushing for similar legislation.

On Oct. 15, a coalition of British and American publishers — including The Times, Financial Times and Condé Nast — urged the U.K. government to back a comparable bill, spearheaded by the U.K.'s News Media Association. News/Media Alliance CEO Danielle Coffey told Digiday the U.K. bill is modeled after the Stealth Bot Prohibition Act.

"We work closely with our U.K. counterparts," Coffey said, adding that coordinated efforts are building international momentum on unfettered AI bot scraping. The Alliance also helped draft and promote a New York bill cracking down on stealth crawlers.

A joint statement from top execs at major U.K. and U.S. media companies, published Oct. 14, made the ask explicit: "We urge you to back Damian Hinds MP's Automated Online Software (Access and Transparency) Bill or introduce equivalent provisions in this parliamentary session. This is a narrow, practical measure that would require bots to identify who operates them and why they are accessing UK websites. With bots now accounting for more than 50% of global web traffic, such measures are vital to protect the open web, British interests, and creators' ability to control and commercialize their content."

Do publishers' own defenses work?

Only partially. Reuters adopted a block-by-default approach earlier this year, but Phil Andraos, gm for digital at Reuters, told the Digiday Publishing Summit in September: "We do our best to block crawlers and bots unless they identify themselves and we authorize them. We do our best to block those that are not authorized. It's not always 100% foolproof."

People Inc. first blocked known AI crawlers while allowing partners. When crawlers hid or changed identity to evade the blocks, one-by-one blocking failed. The company switched to a default-block approach in January. "It has to be the CDN level. If you just have a list of like: 'these are the names we can't come through,' that is as useful as a chocolate teapot. Robots.txt does nothing," said Roberts.

Why the money angle matters

Roberts argued there is strong investor appetite for businesses that package and sell access to online content while the publishers who created it receive little or nothing. "That is the broken dichotomy we're living with at the moment," he said. He cited Exa, which raised $85 million at a $900 million valuation last September, then a $250 million Series C at a $2.2 billion valuation this May. "There are about 80 of these Napsters of content," he said.

He put the imbalance bluntly: "If I go to a venture capitalist and say I want to sign a publisher because AI uses good information, I will get laughed out there with a door slammed in my face. If I go to a venture capitalist and say 'I want to steal the world's information and sell it as a service; no, I'm not an AI company, but please can I have a quarter of a billion dollars', they're like, 'where do I sign'?"

Yet the VC money proves a "huge market and demand for content" that did not exist a year ago, Roberts said. "Now we just sort of wire it up. First step of wiring it up is do not give it away for free: have a block."

What comes next?

Lawyers expect the pressure to intensify. "I think that many content rightsholders around the world are looking to implement a stronger framework than the current one, which is largely based on voluntary measures like robots.txt directives," said Aaron Rubin, partner in the strategic transactions & licensing group at Gunderson Dettmer. He sees transatlantic precedent: "There is certainly precedent for parallel movement on these kinds of digital-economy issues on both sides of the Atlantic. For instance, GDPR was followed by a wave of U.S. state privacy laws."

Gary Kibel, partner at Davis + Gilbert, which advises media and advertising clients, expects the issue to keep growing. With bot-blocking measures struggling to keep pace and licensing leverage at stake, the push to replace voluntary signals with mandatory crawler identification is likely to shape the next round of publisher-AI negotiations.