SEO & Search

AI Can't Cite Local Pages It Can't Crawl, SEJ Live Panel Warns

Whitespark's Darren Shaw and Duda's Russ Jeffery outline how crawl access, server-side rendering, stale URL cleanup and maintained schema determine local AI visibility.

How To Build Local Pages AI Systems Can Find And Trust via @sejournal, @lorenbaker
How To Build Local Pages AI Systems Can Find And Trust via @sejournal, @lorenbakerhongxing128 / Openverse

A local page cannot earn visibility from an AI system that cannot access, render, or trust its information. That was the starting point of a recent Search Engine Journal Live session featuring Whitespark founder Darren Shaw and Russ Jeffery, Duda's Director of Platform and Product Strategy, who walked through the technical and content foundations of local AI visibility.

"You cannot be surfaced in AI responses if the AI can't even access your website," Shaw said.

Before adding FAQs or rewriting service copy, marketers need to confirm two things: crawlers can reach the page, and the business details they find there are accurate.

Start with crawl access

Jeffery recommended starting with the basic controls that determine whether a page can be discovered: robots.txt, noindex directives, security settings, sitemaps, and how content is rendered. He admitted he has launched a site with a noindex tag left in place by accident. Fixing the error later "takes heck of a lot longer than just doing it right the first time," he said.

Shaw flagged Cloudflare as another potential blocker, one that may be enabled without the marketing team realizing it. "Cloudflare often blocks AI crawlers by default," he said. He described a Shopify site that was blocking AI crawlers with the culprit being Cloudflare. His fix is to check for Cloudflare and review the settings.

Both panelists criticized the default to block all crawlers as a policy built for publishers and applied to everyone. Publishers like Time or The New York Times may want crawlers to pay for their content. "But every small business in the world, they do not want to block crawlers," Jeffery said, calling it "a bad default by them."

One caution on robots.txt: Jeffery described it as "a guidance policy" for compliant crawlers and "a little bit of a weak link in the chain." Truly private content should be restricted at the server or application level.

Server-side rendering matters again

Jeffery said JavaScript rendering has "gone backwards in the past few years." Google executes JavaScript and "is still doing a good job at this," but newer AI systems largely do not. "ChatGPT doesn't have their own index, they don't take the time to actually index and save pages within their infrastructure," he said. His advice: deliver important content in the HTML response through server-side rendering and verify it, rather than assuming the framework handled it.

Established platforms such as WordPress and Next.js do this out of the box or offer it as an option. The greater risk comes from newly generated sites that rely heavily on client-side JavaScript without confirming what a non-rendering crawler can see.

Shaw noted that most small businesses use Claude Code or a similar tool to generate a React-heavy site. "If you're just vibe coding a website, they're usually pretty bad, and I would just pay attention to that," he said. His reassurance for everyone else: "For the vast majority of business owners, they don't need to worry about this." Sites on Duda, WordPress, or Wix already serve rendered HTML. The work is confirming what crawlers receive and hunting down cases where important copy only appears after JavaScript runs, such as review carousels loaded by a widget.

Jeffery also discussed delivering markdown versions of pages, which Cloudflare is now pushing as an option. "I wouldn't say it's required right now," he said. He has yet to find an AI search engine that relies on the markdown version of a page. Accessible, server-rendered HTML stays the priority.

Old pages feed AI the wrong information

Crawl access is only useful when the information is correct. The session highlighted a common problem: orphaned or duplicated URLs with labels such as "-old," "-new," "/home," or "v2," left behind when developers cloned pages during a redesign and never de-indexed the originals. Those pages may still carry an outdated phone number or address.

Customers rarely reach those pages through navigation. Crawlers do. As Shaw put it, "the AIs would grab it." A technical audit that only looks for broken pages will miss them; it also has to look for stale versions. The recommended fix is a basic crawl audit on any platform: run the crawl, filter for URLs carrying those suffixes, and make sure none of them are indexable.

Schema is a validation layer that has to be maintained

Schema was the one topic where the panel split.

The session host mirrors every Google Business Profile data point in the site's schema and reports more visibility in the local pack and traditional organic results for those sites, but views schema as a validation tool rather than a ranking factor per se.

Shaw is the skeptic. "I've never seen any noteworthy study that said, if you do schema, your traditional rankings will go up or your AI visibility will go go up," he said, pointing to detailed testing by Jake Hundley where "he found nothing." Where Shaw does see value is disambiguation: product data in a table becomes unambiguous once expressed as structured data, which is easier for a crawler to parse than the page layout.

Jeffery landed in the middle. "You totally should do it," with one condition. Schema is a third source of business data, alongside the website and Google Business Profile, and the worst case is that it goes stale.

Shaw described a site built in 2017 where the developer added schema; in 2025, the owner refreshed the site and never touched the schema because it lived in a RankMath setting they did not know existed. Jeffery said Duda sees the same pattern when a client updates a phone number on the page without configuring the sync to Google Business Profile, leaving the old number live in the markup. In his words, that is "more of a process problem" than an optimization problem.

Shaw endorsed the validation framing and said he wanted to build a tool that maps every Google Business Profile data point to schema. Jeffery said Duda already has one.

Answering an audience question from Todd Vaughn on whether FAQ or Q&A schema still matters now that Google has dropped the rich result, Shaw said, "I wouldn't call it important. I would say it's helpful for sure" when the answer is injected by JavaScript. Otherwise, an LLM strips the page down to "a big markdown file of text," so an FAQ marked up in schema and printed on the page simply appears twice.

What an AI-ready local service page includes

Shaw's research process starts with a question to the machine itself: "What does AI care about?" Using a plumber's hot water tank repair page as a working example, he asks Google's Ask Maps (Gemini grounded in Maps data), or Gemini or Claude directly, what should be on that page. The answers are predictable: "They're always going to tell you pricing," plus trust symbols, reviews, and case studies. Jeffery extended the list to credentials and service area.

Step two is query fan-out, using Mark Williams-Cook's queryfan.com. A user tells a chatbot their hot water tank died last night, they need it repaired quickly, and their budget is tight. "It takes your one prompt and turns it into 10 other prompts," Shaw said. Those 10 searches are the page's FAQ list: "Those are your frequently asked questions." The more of them a page is relevant for, the higher the odds it is cited in the response to the original prompt. Jeffery added the low-tech source: ask the business owner what customers ask all the time. "Whether it's yes or no, you still need to have an answer."

Consumers, he said, "are searching for more and vastly different, and they're searching for longer queries and following up more frequently." His print-shop example: Can I print A1 size on 297 gsm stock? If that answer lives nowhere on the site, there is nothing for the AI to pick up and answer with.

Competitors' negative reviews are page research

Shaw's favorite research prompt runs in Ask Maps because it is grounded in Google Business Profile data: "For plumbers in my city, please analyze their reviews and tell me the most common pain points that people are complaining about in negative reviews."

A business can address those concerns directly on its service page with accurate commitments it can support. This helps conversion by answering a fear before the customer asks, and it gives AI systems explicit evidence about the promised experience. Shaw's examples: we will always be on time; we will treat your home like our own; "we wear special booties on our shoes so we don't mess up your house"; we clean up after ourselves. He is certain about the conversion effect — "100% they're going to be valuable for conversions" — and hedged on the AI effect: "they might give you a slight edge in the AI responses."

Shaw also demonstrated how literally AI reads a page: "You can basically say any BS numbers you want on your webpage, and AI will cite it." Write that you have 10,000 five-star reviews when you have 220, and the AI repeats 10,000. "The takeaway is not to fudge your numbers. The takeaway is to put those words on your page." A review carousel loaded by a widget is invisible to the model: "I can't read it because it's JavaScript." So write the sentence: this many reviews, this rating, this award. "You want to hype your business."

Drop the "we": name the business in its own copy

Answering an audience question from Cody Anderson on semantic triples, Shaw called them "a hard yes." The problem he sees on nearly every small business site: the copy says "we" and never names the entity. "We are experts at hot water tank repair" gives a crawler nothing to attach the claim to. "Johnson Plumbing Denver are experts at hot water tank repair" does. His reasoning: "robots are kind of stupid," so the page has to state the subject explicitly.

Jeffery pushed back on readability: "How do you make it not awkward? Because at that point you are writing for robots." Shaw agreed it cannot open every paragraph. "It's a sprinkling." He reserves the brand name for the passages he most wants the AI to connect to the entity — the core service, pricing, differentiators, and ratings — and uses "we" everywhere else.

What local teams should audit first

  • Crawl controls. Check robots.txt, noindex directives, security headers, and Cloudflare's AI crawler settings.
  • Rendered HTML. Confirm that important content appears without requiring client-side JavaScript.
  • Stale URLs. Find duplicate, orphaned, and archived pages that expose outdated business facts.
  • Structured data. Compare schema with the visible page and Google Business Profile.
  • Customer evidence. Add accurate answers, proof, FAQs, and trust information based on real questions and review themes.

Local AI visibility begins with access and accuracy. Once crawlers can retrieve reliable content, the work shifts to detail: specific answers, maintained structured data, customer proof stated in text, and the kind of exhaustive service information a human would never read and an AI will.

Source: Search Engine Journal (https://www.searchenginejournal.com/how-to-build-local-pages-ai-systems-can-find-and-trust/588097/)

Source: Search Engine Journal

Share this article:

More from News Box Owner