By James Calloway

Most SEO tests can't actually answer the question they were built for, because the tester never defined what a wrong answer would look like. That's the core warning from a new Search Engine Journal column by an SEO practitioner with an academic research background, who trains teams to "think like a scientist" when testing titles, internal linking, content formats, templates and now AI-generated changes.

Her argument is blunt: you can make a change, watch a metric move, and still have almost no evidence the change caused it. Equally, you can see no movement and wrongly conclude your change did nothing.

What separates a real hypothesis from a wish?

The author identifies six recurring mistakes. The first: skipping a falsifiable hypothesis.

She writes that "good research always starts with a question, not a hypothesis" — ideally a commercial uncertainty such as why product-page traffic dropped last quarter, or whether AI mentions fell in the past month.

Her model example shows the level of specificity required: "Updating the titles to include a country of reference on 700 pages will generate an additional 10% clicks over a four-week period, as compared to the four weeks preceding the change."

Under that hypothesis, a 12% uplift confirms it; 9% falsifies it. "Before you run the test, ask yourself: What result would prove my hypothesis wrong? If there isn't one, you're not really testing it," she writes. Vague statements like "improving page titles will improve SEO performance" fail the test because nothing in them can be contradicted.

Why did the metric move? You may not know

The second mistake is failing to isolate variables. Websites aren't sterile labs: while you test titles or a template change, product ships a sitewide feature, marketing launches a campaign, Google rolls out an update, search demand shifts, or developers deploy code on the same pages.

Researchers call these confounding variables, and the author admits she fell into the trap herself. In a recent study on whether people respond differently to AI-generated versus human-created content, she realized she couldn't attribute the differences to content origin — the materials also differed in messaging and participant familiarity. She scrapped the design and started over.

"Painful? Yes," she writes. "But considerably less painful than confidently getting to the wrong conclusion."

Her practical fix: keep a log of major changes across SEO, engineering, product and marketing, and check that control groups are genuinely comparable.

Are you defining success after seeing the numbers?

The third mistake is moving the goalposts post hoc. The author invokes the old SEO joke: if traffic goes up, it's your work; if it drops, it was a Google update. She points to research's answer — preregistration — and says SEO teams should borrow the principle, not the paperwork. Commit in writing, before launch, to the metric, the primary outcome, the period and the comparison.

Otherwise, she warns, "if you give yourself enough metrics and enough freedom after the fact, something will eventually look positive."

Can a seven-day sprint prove anything?

Mistake four: overreaching conclusions. Too-short windows, small samples, crawl-and-index delays and seasonal behavior can all mean a test simply lacked the power to detect an effect. "Seven days sound tidy, but doesn't necessarily mean you've given the test seven meaningful days of exposure," she writes.

Her key principle: "Absence of evidence isn't necessarily evidence of absence." Not finding a difference and proving there is no difference are two different results. More data won't rescue a badly designed experiment, but it can help a well-designed one.

What does 20 tests tell you that one can't?

Mistake five is treating each test as isolated. One experiment shows an uplift on one set of pages, in one period — it doesn't prove "Google definitely prefers this" and shouldn't become SEO law rolled out across every market. She recommends logging the question, hypothesis, treatment, sample, dates, context, result, interpretation and limitations for every test, including the inconclusive ones.

"One test gives you a result, but 20 properly documented tests start telling you something about your website, your audience, your markets, and which assumptions repeatedly survive contact with reality," she writes.

When is a win not worth rolling out?

The final mistake: testing without a decision attached. She suggests finishing this sentence before launch: "Depending on what we find, we will…" — scale, revert, change direction, or do nothing.

Doing nothing counts. A change that lifts organic clicks by 0.3% but demands six months of engineering time may have "worked" statistically while still justifying a pass. "Value isn't measured in volume, but in learning and actions we take as a result," she writes — a direct hit on teams that treat experiment count as a maturity marker.

Her closer reframes the whole discipline: "The output of an experiment isn't the chart, but a decision." For SEO teams, that's the shift that turns a folder of before-and-after graphs into a body of evidence worth acting on.

Tags: SEO testing, A/B testing, hypothesis, experimentation, search marketing