Field notes
02
We asked 11 AI models which watch brand they would wear on a first date, 50 times each. Some gave the same answer every time. Others seemed to roll dice. The 546 answers held 245 different top-five lists, and not one matched the final ranking. Pooled, they settle fast: 10 answers per model get the top five right nine times in ten. Asking more changes the long tail, not the top of the list.
28 Sept 2026 · 7 min read
tap to read →
How many answers it takes before an AI ranking of brands stops moving, and which models make it wobble.
The experiment
Ask an AI assistant which watch to wear on a first date and it gives you one confident answer. Ask again and you may get a different one. Ask a different assistant and you probably will.
So how many answers do you need before you know where a brand stands? Most of our category studies ask each model about 10 times. For watches we went five times further: the same question, 50 times, to each of 11 models from 8 labs.
Each answer is a ranked top five. Across all 550, Omega was the #1 pick 39% of the time, Cartier 26%, Seiko 14% and Rolex 13%. Every other brand shared the last 8%.
Watch brands: share of 550 answers that put each brand #1. Rolex took another 13%.
Finding 1
Line up each model's 50 answers side by side, one square per answer coloured by its #1 pick, and the personalities jump out.
Grey squares are answers that came back empty.
The metronome. mistral-medium-3.5 put Rolex first in all 50 answers, and gave the exact same top five in 47 of them. The panel as a whole ranks Rolex fourth.
The fence-sitter. gpt-6-astra, OpenAI's newest model on the panel, couldn't choose between Cartier and NOMOS Glashütte. For its first 14 answers it swapped between the two almost every time. Over all 50, Cartier won 36 to 13.
The dice. deepseek-v4.1-flash named seven different #1s, and its favourite, Omega, won only 14 times. Its first ten answers made it look like a Tudor and Grand Seiko fan. Forty answers later, it looked like an Omega fan.
The shuffler. qwen3.8-max almost never repeats itself. Take any two of its answers: out of 1,176 possible pairs, only 6 gave the same top five in the same order. For mistral-medium-3.5, it was 1,082 out of 1,225.
Ask deepseek-v4.1-flash ten times and you'd call it a Tudor fan. Ask it fifty times and it's an Omega fan.
Finding 2
Is it the lab that makes a model steady or erratic? We scored each model's consistency across all 11 categories: how often its answer named its own usual #1.
Anthropic
OpenAI
One model per lab
The first 10 answers per model in each category. Each lab's two models sit 6 to 8 points apart.
Most consistent: mistral-medium-3.5 and gemini-3.8-flash, which stick with their usual #1 96% of the time. Least consistent: deepseek-v4.1-flash, at 69%. It is the only model that never gave the same #1 in all 10 answers, in any category.
Every lab with two models on the panel has a steadier one and a looser one, 6 to 8 points apart. That is a wider gap than the one between the labs' averages. Knowing the lab doesn't tell you how steady the model is.
It doesn't tell you what the model thinks either. A strong position with one model doesn't carry over to another model from the same lab. OpenAI's two models agree that Cartier is #1 and disagree on much of the rest:
Each brand's score out of 100 with each model, across 50 answers.
Anthropic's two models don't even share a #1: claude-opus-5 picks Seiko and claude-haiku-4.5 picks Omega. Cartier scores 46 with Opus and 2 with Haiku. Google's two Geminis both pick Omega, but Cartier scores 82 with one and 34 with the other.
The sources differ too. gpt-5.6-luna cites Hodinkee in every answer; gpt-6-astra, OpenAI's newer model, cites it in only a third and leans on brands' own websites instead. So what got you cited by one model may not work for its successor.
There is a second kind of wobble, further down the list. Ask a model to name its top 5 brands, 10 times over. A perfectly steady model ends up with the same 5 brands. A restless one ends up with many more.
claude-haiku-4.5 usually agrees with itself about its #1. Below that, it roams: asked for 5 brands 10 times, it ends up with about 10 different brands, twice as many as it was asked for. In watches it named 18 different brands in 50 answers, plus three generic answers like "fashion watch brands". mistral-medium-3.5 named 7.
5 is the minimum: the same 5 brands every time. Average of 11 categories.
So is wobble a bell curve, with most models somewhere in the middle? No. Take every model in every category, 121 pairs in all. In 57% of them, the model gave the same #1 all 10 times. The wobble sits in a thin tail, and one model is in it more than any other: deepseek-v4.1-flash accounts for 4 of the 16 pairs at 6 in 10 or below.
Answers out of 10
Finding 3
With models this different, no single answer is a good guide to the ranking. Of the 550 watch answers, 546 came back with a top five. The other four were empty: three from deepseek-v4.1-flash and one from qwen3.8-max, where the model spent its whole reply reasoning and never wrote the list. The 546 held 245 different top-five lists. Even the first answer from each of the 11 models was a different list. Not one answer listed the final top five in order: Omega, Cartier, Seiko, Rolex, Tudor. Only 69 even named the same five brands.
0 of 546
answers gave the panel's final top five in order
How can that be? The final ranking isn't an answer any model gave. It's an average of all of them. Each brand scores points for where it lands in each answer, and the points are added up. It works like "the average family has 2.4 children": the average is accurate, but no single family matches it.
Out of 546 answers with a top five.
Tudor is why the last step fails. It came 5th overall, but hardly any answer put it 5th. It appears in 170 answers and sits at #5 in only 8 of them. Usually it's #3 or #4, mostly from kimi-k3, deepseek-v4.1-flash and gemini-3.8-flash. When Tudor is in an answer, Rolex is usually pushed to #5: the most common list after mistral's is Omega, Cartier, Tudor, Seiko, Rolex. And when an answer has Rolex at #4, #5 goes to Casio, Hublot, Richard Mille or others, never Tudor. A third of answers rank Tudor fairly high and the rest leave it out. That averages to 5th.
The ranking comes from pooling the answers, and pooling is where more runs pay off. We redrew each model's answers at random, 1,000 times at each size, and checked how often the pooled top five came out in the final order.
Answers per model, 11 models
The #1 settles almost at once. The rest takes longer. With one answer per model, Omega came out on top in 98% of redraws, but the whole top five was in the right order only 37% of the time. At 10 answers per model it's 90%, and at 20 it's 99%. The erratic models cancel out because they wobble in different directions, but it takes a few rounds for that to happen.
The second is the long tail. Every new batch of answers turned up brands nobody had named before: Breitling in run 7, Patek Philippe in run 15, Sinn in run 39, Oris in run 45. Each was named once, and three of the four came from claude-haiku-4.5. By 50 runs, the models had named 28 brands between them, more than twice as many as after one.
Runs per model
Each model's answers in the order they were collected. Generic answers, such as 'fast-fashion brands', aren't counted.
Finding 4
People don't act on an average of 11 models. They ask one assistant. So the question that matters is how many times you need to ask one model before you know what it thinks.
We ran each model 50 times on watches. From those runs we worked out how often 10 runs get the right answer, and extrapolated what 100 runs would give.
95%
the chance that 10 runs name a model's own #1 correctly, for 8 of the 11 models
Margins are for the typical gap between two brands in a model's own top five, on the 0 to 100 score. Every figure comes from simulated batches drawn from each model's 50 real answers.
Ten runs tell you what most models prefer. Fifty tell you their top three. Model by model, it looks very different:
Share of 4,000 simulated batches, drawn from each model's 50 real answers, that named the same #1 as all 50. We did not run any model 100 times.
A tie is the honest answer in that last case. When a model scores two brands within a few points of each other, no number of runs will split them. Report it as a tie, not a winner.
What it means
Your position isn't one number. It's a separate answer from every model your buyers use.
01
Ten runs give you a solid read on each model's #1. If your brand sits between #2 and #5, more runs, up to 50 per model, give you a much clearer picture of where it stands.
02
Models from the same lab answer differently. A strong position with gpt-5.6-luna doesn't carry over to gpt-6-astra, and the sources that got you cited by one may not work for the next. Check each model on its own.
Where does your brand stand across the models? Every BrandRanking.AI study asks a panel of models the same question about 10 times each and shows where they agree and where they don't. Browse the category studies or request a study for your own market.
BrandRanking.AI © 2026 · Study data: brandranking.ai/categories