ChatGPT names a different brand first in three of four identical questions, New York agency BusySeed matched 172 624 answer pairs for model and prompt, brands still buy AI rankings the machine cannot reproduce

Images

bnonews.com

Ask ChatGPT the same buying question twice and roughly three times in four it will name a different brand first, according to a study that compared 172,624 pairs of answers collected between April 27 and October 16. The brand placed on top stayed the same in only 25.4 percent of pairs. New York marketing agency BusySeed ran the comparison across 172 product and service categories, with every pair matched for model version, web-search setting and prompt wording, and the study attributes none of the churn to OpenAI updating its models.

The gap between answers explained almost nothing: pairs given less than a week apart held the same leading brand 23.9 percent of the time, while pairs more than four weeks apart held it 25.9 percent. Among the 143 categories with enough data to compare, the share of repeat answers naming a different leading brand ranged from 34.3 to 92.8 percent, with a median of 77.3, and no single category made up more than 1.7 percent of the data. A separate 15-week BusySeed study covering ChatGPT, Claude and Gemini found the same pattern on all three assistants: the first-named company changed in 56.4 percent of week-to-week comparisons, and only 85 of 480 tracked question series kept the same leader throughout.

A growing market of tools meanwhile reports a single AI ranking for a company, the equivalent of a search-engine position. The data says such a number describes one answer drawn from a distribution, not a position a brand holds. A brand that pays for visibility on Tuesday has no reason to expect it on Wednesday.

The study has limits its authors state plainly: it used OpenAI's developer interface rather than the logged-in app, and it cannot separate randomness inside the model from genuine changes in the web pages the model reads. BusySeed, founded in 2013, publishes its methodology and underlying data alongside the search, advertising and lead-generation services it sells. Its founder's conclusion is that what counts is "how often you show up across many asks, not where you placed once."

"If you check where you rank in ChatGPT on a Monday," chief executive Omar Jenblat said, "you've learned something about Monday."