Meta Strips AI Training From Crawling Clause — Lawyers Aren't Buying It

By Amara Osei

Meta removed the word "train" from its Commercial Terms crawling clause on Oct. 17, narrowing — but not closing — a path that critics argued let the company scrape publisher websites for AI model development. The revised terms take effect Oct. 30.

The change strips explicit AI training permission from language permitting Meta to crawl sites "associated with" a business's products or services. The permission itself stays. Access is unchanged.

What did the clause say before?

The original clause, reported by Nieman Lab on Oct. 15, allowed Meta to use crawled content "to provide, develop, improve, research, train, and maintain the integrity of" its products. The word "train" sat at the center of publisher alarm.

Two days later, Nieman Lab flagged legal expert warnings that the wording could support AI training claims. The European Publishers Council objected before the revision. The UK's News Media Association condemned it in an Oct. 16 statement.

Meta's promise of clarity arrived only after two trade bodies went public with objections.

What does the revised clause say?

The new clause limits crawling's stated purpose to "providing, developing, improving, researching, and maintaining advertising and business products." A parenthetical defines those as products that help a business create and deliver ads and connect with interested users. An integrity purpose survives, now worded as "the integrity and security of" Meta's products.

Two phrases changed. The scope did not. Meta retained its prior access rights, including automated software and "test credentials" used to gather information, plus periodic retrieval of website updates.

The revised clause drops a reference to robots.txt — a standard site instruction telling crawlers which pages to skip. As of Oct. 19, the UK English preview for Meta's Self-Serve Ad Terms still contained the original "may use any content ... train" language. The U.S. English version did not.

Why does Meta say it changed the wording?

Meta spokesperson Daniel Roberts told The Verge the consolidated crawling terms "only applied to our ads and business products, and we've clarified that in the terms themselves." He said the update "does not change how Meta uses web crawlers" and added: "This is not about scraping news sites."

Meta also told Nieman Lab the first version "wasn't clear" that robots.txt preferences are respected. The company said the updates do not give it "new rights that supersede those preferences."

Are the lawyers convinced?

Three legal specialists said no. Their caveat: Meta can still use crawled material for purposes that look like training under a less explicit label.

Matt Topic, a media and intellectual property attorney at Loevy & Loevy, told Nieman Lab: "I think 'develop' is broad enough to include 'train.'" He flagged that "business product" is not a defined term.

Cornell Law School professor James Grimmelmann said the edit does not restrict the "associated with your products or services" language, which he said could let Meta scrape entire sites. He called the purposes language "still broad enough to drive a truck through."

University of Wisconsin law professor Benjamin Sobel pointed to a separate limit: robots.txt is voluntary. Following it is a choice, not a legal obligation. A publisher that blocks Meta's crawlers cannot, on his reading, recover damages if Meta crawls anyway.

What does a publisher actually control?

Meta's crawler documentation lists separate crawlers — one for AI training, one for Meta AI search. A site can therefore disallow one and allow the other through robots.txt.

The revised terms do not state whether those granular controls override what a business grants by accepting Meta's terms. That gap is the dispute's center of gravity.

• Original scope: "any content" used for development, research, training, and integrity
• Revised scope: content used for advertising and business products, plus integrity and security
• Access rights: unchanged, including automated crawling and periodic re-fetching
• Robots.txt reference: dropped from Commercial Terms preview; UK Self-Serve Ad Terms preview still carries earlier language

The revision tightens the wording. It does not answer whether crawled data feeds model training under a different label.

Meta's terms take effect Oct. 30. Check the clause again before that date — and audit your robots.txt file against each named Meta crawler.