Three AI Models Shopped the Same 20 Products. None Agreed.

Three Claude models given identical Shopify product lists and identical instructions never agreed on the top product in any of six searches, according to a controlled test of 1,980 model picks run through Shopify's Shop MCP, the keyless catalog connection the company opened to AI agents on Oct. 13, 2026.

The tester, Slobodan Manic, ran each search once through Shop MCP as an anonymous U.S. buyer, saved the top 20 results, and then asked Claude Haiku 4.5, Sonnet 5 and Opus 5 to pick one product, 50 times each per search. Each call was fresh, with no memory, no tools and no web access. The models saw only the title, store, price, star rating, review count, the catalog's one-line summary and the first three listed features, numbered 1 to 20.

Shopify says Shop MCP "lets your agent or app search Shopify Catalog, build carts, and complete Shop Pay checkout through a single MCP connection." It covers over a billion products and requires no API key (MCP, or Model Context Protocol, is a standard connection that lets AI agents query external systems). The logic behind the test: a list is not a purchase, and when an agent runs on a language model, the model is what picks.

Which store gets the order may depend on which model the shopper's agent runs on

Pairwise agreement was near-total failure. Haiku and Sonnet agreed on three searches (backpack, coffee beans, water bottle), Sonnet and Opus on two (wireless earbuds, yoga mat), and Haiku and Opus on one (running shoes). Each model was consistent with itself — Sonnet's most-picked product averaged 94% of its picks, Opus's 89%, Haiku's 67% — but not with each other. On three of six searches, the different pick came from a different store: on water bottle, Haiku and Sonnet favored nobl's bottle while Opus picked Owala's all 50 times. The catalog's own ranking mattered less than expected: Haiku picked the catalog's first-ranked product 8% of the time, Sonnet never, Opus 20%.

When a model picks on a number the store's own page does not show

Haiku and Sonnet favored the product with the highest review count. On water bottle, the catalog listed nobl's Stainless Steel Water Bottle at 5.0 stars from 447,221 reviews. Haiku picked it 45 times out of 50 and cited the figure every time; Sonnet picked it 39 times, citing it in 33. But nobl's product page showed 5.0 from 6,537 reviews when checked on Oct. 24, and the catalog's record for the Blush color said 4.2 from 1,674. The 447,221 figure appears nowhere on nobl's site and is roughly 68 times the product page's count — though it is live: a later search on Oct. 24 returned 447,227. Neither model could verify it from what it was given. Opus never picked that bottle.

Opus called its pick "safe" in 222 of 300 picks

Given no shopper context, Opus 5 built a default shopper and bought for that person, calling its choice "safe" or "safest" in 222 of 300 picks. Haiku did so once, Sonnet 11 times. "With no stated preferences, a medium roast is the safest crowd-pleaser," Opus said on coffee beans. On a mineral sunscreen search, it ruled out "tint, powder, or stick" formats in 17 of 100 reasons — a format the shopper never mentioned.

The test also caught models relaying store claims as their reasoning. "Clinically proven" appeared in 63 reasons across eight skincare searches, straight from listings in 61 of them. All 22 reasons repeating that a retinal product works "11x faster" than retinol came from Haiku, nearly all word for word from listings. A claim in a listing is now a claim an agent makes on the store's behalf, unchecked.

Position in the list moved picks, and no model said so

On eight skincare searches, reversing the list — moving the catalog's top product from first to last — changed outcomes. Across 80 picks per version, Haiku's picks of the top product fell from 28 to 11, Opus's from 24 to 10, Sonnet's from 15 to 11. On mineral sunscreen, all three models almost stopped picking it when it was last. In 480 reasons, not one mentioned position.

Shopper wording carried weight too. Adding "under $25, for oily skin" to "mineral sunscreen SPF 50" moved all three models off the catalog's $48 top-ranked product in 298 of 300 picks. Sonnet picked ATTITUDE's $22.95 sunscreen all 100 times. Opus split between ATTITUDE (67) and Sun Bum's $17.49 lotion (33) — the lowest-rated of the six under $25 at 4.2 stars, but the only listing saying "without clogging pores."

What the test does not show

Manic is explicit about the limits: three Anthropic models only; no real shopping apps like ChatGPT or Gemini, which wrap the model in their own instructions; fixed lists from two days (Oct. 15 and 24, 2026); nothing bought. The decision layer between shopper and store remains a black box — unlike Google rankings, which a store can at least monitor. His advice to merchants: audit what the catalog shows agents, write listings in the customer's own words, keep claims true, and run cheap competitive tests across models. A store now competes for a model's preference, and there is no ranking to check.