Google Broadens Mediapartners-Google Crawler Guidance Beyond AdSense
Google generalized its Mediapartners-Google crawler docs beyond AdSense, citing Google Ad Manager as an example but omitting a full list of affected ad products.
Updated
The brief
- Google updated Mediapartners-Google crawler documentation to cover multiple ad-related Google products, not just AdSense.
- The new wording names Google AdSense and Google Ad Manager as examples but does not list all affected services.
- The documentation does not specify whether Google Marketing Platform, Local Services Ads, or Google AdMob are affected.
Google has updated its crawler documentation to clarify the role of the Mediapartners-Google crawler, long associated exclusively with AdSense. Site owners, publishers, and SEOs who reference the Mediapartners-Google user agent in robots.txt or elsewhere should review the change.
What changed
Google described the revision as a "generalization": the documentation is now less specific to AdSense and more contextual to the wider category of Google Ads products.
The original wording advised:
"Crawling preferences addressed to the Mediapartners-Google user agent affect Google AdSense. The AdSense crawler visits participating sites in order to provide them with relevant ads."
The updated wording now reads:
"The Mediapartners-Google crawler visits participating sites in order to provide them with relevant ads. Crawling preferences addressed to the Mediapartners-Google user agent affect various ad-related Google products (for example, Google AdSense and Google Ad Manager)."
Why the wording matters
The new text expands the set of Google services that Mediapartners-Google touches — but stays opaque about which services those actually are. Google lists two examples, AdSense and Google Ad Manager, without providing the full list.
The documentation does not say whether the crawler affects Google Marketing Platform, Local Services Ads, or Google AdMob.
For publishers and SEO teams, the practical implication is straightforward: blocking or adjusting crawl preferences for the Mediapartners-Google user agent may now influence more than AdSense ad delivery. Any robots.txt rules built on the assumption that this user agent maps solely to AdSense deserve a second look.
The updated list of Google's special-case crawlers is available on Google's developer documentation.
Based on Search Engine Journal; developers.google.com
Filed under google, google-ads, seo, crawler, robots-txt
More from Amara Osei
Show full bio
News editor covering industry trends and analytics at Marketing Herald.
54 articles