The metric an SEO team rewards its AI agents for will matter more than the model the team chooses — and research from MIT and Stanford, read together, points to that risk and to fixes that fit next quarter's plan.

The first piece is an interview Joshua Miller of The Boston Globe ran in his Camberville newsletter on September 17. His guest was Dylan Hadfield-Menell, an associate professor of electrical engineering and computer science at MIT on the faculty of artificial intelligence and decision-making. Hadfield-Menell studies how goals get set for AI systems and how that process goes wrong. He opened with an example every SEO will recognize.

The vacuum that fed itself

Researchers once trained a robot vacuum with reinforcement learning, rewarding it every time it picked up dirt. The vacuum learned to pick up dirt, dump it back on the floor, and pick it up again. It hit the target and defeated the purpose.

Hadfield-Menell connects that story to a 1970s management paper titled "On the Folly of Rewarding A, While Hoping for B." Its classic case is the university professor who gets promoted for publishing research while being expected to teach. Pay for one behavior, and you get that behavior, whatever you were hoping for.

What has changed, he says, is scale. Since early 2025, developers have applied reinforcement learning at much larger volume on top of language models, and it strengthens some behaviors nobody wants. He pointed to a recent incident involving OpenAI systems and Hugging Face, where models that judged a task too hard went looking for ways to cheat the test. He compared it to breaking into a professor's office to steal the exam.

His worry is not that machines wake up with goals of their own. Systems are handed a goal, adopt subgoals along the way, and keep pushing toward completion in a way he called "sticky."

SEO is the profession best placed to understand this problem — and, arguably, slow to admit it applies to the field. SEOs have spent more than 20 years optimizing proxies. Rankings, traffic, domain scores, and now AI visibility scores all stand in for a business result nobody can measure directly. A human team games a proxy slowly and with hesitation. An agent does it faster and without any.

The scoreboard is shakier than vendors admit

Stanford's 2026 AI Index shows why leaning on published scores is risky. The report says AI keeps improving quickly: on SWE-bench Verified, a coding benchmark, performance rose from 60% to near 100% in a single year, and 88% of organizations now use AI.

The same report, in its technical performance chapter, cites a review that found invalid-question rates on popular benchmarks ranging from 2% on MMLU Math to 42% on GSM8K. It also notes research suggesting that a model's standing on the Arena leaderboard may partly reflect adaptation to the platform rather than general capability.

Michelle Kim of MIT Technology Review summarized the report in April. She adds that models trained on benchmark test data can learn to score well without getting smarter, and that the top models now sit very close together and compete on cost, reliability, and real-world usefulness. Yolanda Gil, a University of Southern California computer scientist who coauthored the report, told Kim that when a company leaves out its results on certain benchmarks, particularly the responsible-AI ones, the omission "maybe says something."

That should change how an SEO team shops for tools. If the leading models sit within a few points of each other, and the scores themselves can be flawed or gamed, a vendor's benchmark slide says little about how the product will treat your pages, your queries, and your clients. One test on your own site is worth more than any leaderboard.

Where the returns actually come from

MIT Sloan's Betsy Vereckey reported in August on what separates companies that profit from AI from those that don't. George Westerman, a senior lecturer at MIT Sloan and a digital fellow at the MIT Initiative on the Digital Economy, says the answer is not better algorithms. The winners redesigned how work gets done. At the MIT Enterprise AI Forum in May, he told the audience that technology delivers little until the business itself operates differently.

He put the share of AI pilots that never scale at somewhere between 70% and 95%, a range the Sloan article attributes to studies without naming them. Pilots are easy to launch and hard to spread.

Westerman's sharpest test for leaders is about governance. "Is your governance more the steering wheel or is it more the brakes?" he asked.

HCA Healthcare shows what the steering-wheel version looks like. A committee reviews the risks, business case, and feasibility of every AI use case, then asks its questions again before a pilot at a small number of hospitals and again before the project scales. It also checks periodically that its models are still holding up. The risk questions point the team toward what to investigate rather than stopping the work.

Marketing appears in Westerman's case studies too. Dentsu Creative has pushed AI across planning, creative, market research, and campaign work.

Many SEO teams running AI pilots may be headed for that 70% to 95% failure band. A pilot that doesn't change the brief, the review step, or the reporting is a tool trial, whatever the slide deck calls it.

How to apply this to your SEO strategy

Four moves follow from the three pieces of research.

Pair every proxy with an outcome the agent can't touch. List the metrics your AI-assisted workflows are judged on, from pages published to schema deployed to brand mentions in AI answers. Then attach a second measure a human owns, such as qualified leads, pipeline, or branded search demand. Citation Share of Voice is useful, but it is still a proxy. If a content agent is judged on how often your brand shows up in AI answers, expect it to find the cheapest route there. Check a sample of those citations by hand each month and see whether they send anyone to a page that converts.

Test tools on your own pages. Pull a set of real queries from Search Console, run each candidate tool against your own content, and have an editor grade the results without knowing which tool produced them. Repeat it every quarter, because the leaderboards will move and you won't know why.

Gate the agents the way HCA gates use cases. Add review points before design, before the pilot, and before scale. Pilot on one directory or one language market, and decide in advance what result ends the pilot. Keep agent permissions narrow, so drafting doesn't quietly turn into publishing or editing templates. Westerman's advice is to change or drop a project that isn't producing the results you expected.

Rewrite one workflow, not the tool stack. Before buying anything, name the step in your process that will be different after the pilot — briefing, QA, or reporting — and say who loses a task because of it. Then tell the team what changes and what training comes with it. Westerman notes that silence lets people imagine the worst.

The next model release will not decide who wins in AI search. The metric used to judge the agents will, because the agents will find it first.