Your Brand has an AI reputation and it's shaping purchase decisions right now.

Swipe up to continue

Category · Business Class airlines

Which airline would you recommend for business-class travel?

4 AI models · 5 answers per model · ranked by score

ordered by brand score: 100 pts for rank 1 → 20 for rank 5 · no mention = 0

  1. #1Qatar Airways
    100
    brand score
    100first choice
    12345
  2. #2Singapore Airlines
    80
    brand score
    50first choice
  3. #3ANA
    48
    brand score
    28first choice
  4. #4Emirates
    41
    brand score
    25first choice
  5. #5Cathay Pacific
    14
    brand score
    10first choice
Show 4 more
  1. #6Delta Air Lines
    9
    brand score
    9first choice
  2. #7Japan Airlines
    4
    brand score
    3first choice
  3. #8EVA Air
    2
    brand score
    2first choice
  4. #9Turkish Airlines
    2
    brand score
    2first choice

Each bar covers the middle half of one model’s answers (Q1–Q3); the line inside it is that model’s median rank for the times it recommended the brand. The badge under a brand says how far apart the models are on it, and “first choice” scores rank 1 far above the rest — a brand can place well overall and still rarely lead an answer.

Swipe left for the full report

Category · Business Class airlines

Model by model

Median rank each model gave each brand.

claude-opus-5gemini-3.7-flashgpt-6-astrakimi-k3
Qatar Airways1.01.01.01.0
Singapore Airlines2.02.02.02.0
ANA4.03.03.03.0
Emirates3.04.05.04.0
Cathay Pacific5.04.05.0
Delta Air Lines5.05.0
Japan Airlines4.0
EVA Air5.0
Turkish Airlines5.0
1st
2
3
4
5th

A dash means that model never named the brand in any of its runs. Deeper blue is better: rank 1 is the brand a model would recommend first.

Category · Business Class airlines

Where the models disagree

0 of 9 brands split the panel. Each mark is one model's score for the brand, on the same 0–100 scale as the ranking.

  1. #4Emirates

    gpt-6-astra 12 claude-opus-5 60

  2. #5Cathay Pacific

    claude-opus-5 0 gpt-6-astra 40 · 1 of 4 never named it

  3. #3ANA

    claude-opus-5 24 gpt-6-astra 60

  4. #6Delta Air Lines

    gpt-6-astra 0 claude-opus-5 20 · 2 of 4 never named it

A hollow mark is a model that never named the brand in any of its runs, which scores 0. Agreement is not endorsement — a brand every model ignores equally agrees just as tightly as one they all rank first.

Category · Business Class airlines

Named, and named first

Across → how frequently the panel names the brand at all.
Up ↑ how often the answers that name it put it first.

the default answera narrow favouritelisted, rarely ledthe long tail12345678
  1. 1Qatar Airways100/100
  2. 2Singapore Airlines100/0
  3. 3ANA90/0
  4. 4Emirates90/0
  5. 5Cathay Pacific45/0
  6. 6Delta Air Lines45/0
  7. 7Japan Airlines10/0
  8. 8EVA Air10/0

The horizontal line sits at 20% — the rate a named brand would lead at if the models were picking one of its 5 slots at random. Above it they are choosing it first on purpose. Both figures average across models, so a thinly sampled model counts the same as a heavily sampled one.

Category · Business Class airlines

What the models actually said

Models
4
Answers
20

Qatar Airways leads with a score of 100 of 100, ranked by 4 of 4 models.

The category shows strong consensus at the top and widening disagreement further down. All four models place Qatar Airways at a median rank of 1 and Singapore Airlines at 2, with no model dissenting on either. The splits emerge among the middle and lower tiers, and much of the divergence is about how far a carrier trails rather than the direction of the judgment.

Where the models agree

Qatar and Singapore are locked. Every model lands on 1 and 2 respectively, and the reasoning converges too: the Qatar Qsuite is treated as the benchmark cabin, and Singapore's second place is consistently attributed to seat privacy and diagonal sleeping position rather than any soft-product weakness. The recalled sources reinforce the agreement — Skytrax World Airline Awards, The Points Guy, and One Mile at a Time appear across all four models for both carriers.

The one wrinkle at the top is framing rather than placement: gpt-6-astra shares Qatar's first rank but qualifies it, flagging that the Qsuite is not fleet-wide.

Qsuite is not available on every aircraft, so confirm the scheduled seat configuration before booking.

— gpt-6-astra

Where the models split

The clearest divergence is on Emirates. claude-opus-5 rates it highest at a median of 3, gemini-3.7-flash and kimi-k3 sit at 4, and gpt-6-astra is the low outlier at 5. All four cite the same tension — strong A380 and ground experience against the older 2-3-2 777 seating — but weigh the seating variability very differently. gpt-6-astra lets that flaw set the ceiling, while claude-opus-5 treats the new 777 suites as a mitigating factor.

It ranks fifth among these options because the seating experience varies substantially: some older Boeing 777 cabins lack direct aisle access for every passenger and offer less privacy than the leading alternatives.

— gpt-6-astra

ANA shows a milder split: three models place it at 3, but claude-opus-5 alone sits at 4. Notably, claude-opus-5 rates Emirates above ANA (3 vs 4), while gpt-6-astra reverses that ordering (ANA 3, Emirates 5). The underlying complaint for ANA is uniform — The Room is exceptional but flight-dependent — so the disagreement is about how heavily to penalize limited availability versus Emirates' seat inconsistency.

Cathay Pacific splits by degree among the three models that ranked it: gpt-6-astra at 4, gemini-3.7-flash and kimi-k3 at 5. kimi-k3 pushes it lowest on post-pandemic service and catering recovery, a concern the other models do not foreground.

Single-model tails

Several carriers were ranked by only one model, so they carry no cross-model signal:

  • Japan Airlines (rank 4) and Delta (rank 5) — Delta ranked by claude-opus-5 and gemini-3.7-flash, both at 5, but no other model placed it.
  • EVA Air (rank 5) — gpt-6-astra only.
  • Turkish Airlines (rank 5) — kimi-k3 only.

These reflect differences in which brands each model chose to recall as much as differences in evaluation. That claude-opus-5 surfaced Japan Airlines while kimi-k3 surfaced Turkish Airlines, for instance, may be a coverage difference rather than a disagreement about relative quality — but the data does not let us test that, since the other models did not rank these carriers at all.

On the sources

The recalled source base is highly uniform across the category: Skytrax World Airline Awards (57×), The Points Guy (45×), and One Mile at a Time (39×) dominate every brand. This makes it difficult to attribute any specific model's split to a distinctive source. One plausible pattern — and it is a hypothesis, not something the data confirms — is that the US-inflected placements lean on region-specific sources: Delta's ranking draws on J.D. Power North America Airline Satisfaction Study, which appears nowhere else in the category. Whether that source drives claude-opus-5 and gemini-3.7-flash to surface Delta at all, versus merely accompanying a judgment reached otherwise, cannot be determined from these numbers.

Category · Business Class airlines

What shaped the answers

35 sources across 293 references, grouped by site from 53 recalled names. The top 5 carry 77% of them.

Skytrax World Airline Awards×75 · 26%

worldairlineawards.com

Sources are what each model recalled as having shaped its view — not verified citations. A model without web access reconstructs a reference from memory, so a link may not lead where the model thought it did.

Category · Business Class airlines
#1

Qatar Airways

100
brand score
100first choice
12345

Summary

Qatar Airways occupies an unambiguous top position in this category, holding a median rank of 1 (Q1 1, Q3 1) across all 20 ranked answers from 4 models, with each individual model also landing on a median rank of 1. Agreement is complete: no model placed it below first, and the reasoning behind that consensus is consistent, centering almost entirely on the Qsuite business-class product. Across claude-opus-5, gemini-3.7-flash, and kimi-k3, the Qsuite is repeatedly described as the benchmark for the cabin — fully enclosed suites with doors, double-bed and quad configurations, and dine-on-demand service — supplemented by the Al Mourjan lounge at Doha's Hamad International and repeated Skytrax World's Best Business Class wins. The recalled sources under these themes converge as well, with Skytrax World Airline Awards, The Points Guy, and One Mile at a Time appearing across all four models, alongside Business Traveller and Qatar's own site.

Its Qsuite business class offers fully enclosed suites with doors and quad configurations, widely regarded as the best business class product in the world.

— claude-opus-5

The one point of nuance comes from gpt-6-astra, which shares the top ranking but qualifies its endorsement by flagging that the flagship product is not universally deployed, advising travelers to verify the aircraft and seat map before booking.

Qsuite is not available on every aircraft, so confirm the scheduled seat configuration before booking.

— gpt-6-astra

Per model summary

  • claude-opus-5
    brand score
    100
    first choice
    100
    median rank
    1.0
    present in
    5/5

    Consistently centers on the Qsuite as the benchmark business-class product (enclosed suites with doors, quad configurations), alongside top-rated dining, the Doha/Al Mourjan lounge, global connectivity, and repeated Skytrax World's Best Business Class wins.

  • gemini-3.7-flash
    brand score
    100
    first choice
    100
    median rank
    1.0
    present in
    5/5

    Repeatedly frames Qatar Airways as setting the industry benchmark via its Qsuite (privacy doors, double-bed configurations), combined with dine-on-demand fine dining and world-class lounge/ground service at Doha's Hamad International.

  • gpt-6-astra
    brand score
    100
    first choice
    100
    median rank
    1.0
    present in
    5/5

    Recommends Qatar Airways as the top choice for long-haul business class based on Qsuite's privacy, flexible layouts, and dining, while consistently cautioning that Qsuite is not on every aircraft and advising travelers to check the seat map.

  • kimi-k3
    brand score
    100
    first choice
    100
    median rank
    1.0
    present in
    5/5

    Describes the Qsuite as the benchmark for business class (enclosed suites, doors, double-bed/quad options, dine-on-demand), reinforced by consistent fleet-wide quality, the Doha hub/lounge, and Skytrax awards.

Sources per model

Each model’s own references for this brand, grouped by site. Tile area is that site’s share of the model’s references; tap one for the pages behind it.

claude-opus-54 sites · 15 references
Skytrax World Airline Awards×5 · 33%

worldairlineawards.com

gemini-3.7-flash5 sites · 15 references
Skytrax World Airline Awards×5 · 33%

worldairlineawards.com

gpt-6-astra5 sites · 15 references
Skytrax World’s Best Business Class Airlines×5 · 33%

worldairlineawards.com

kimi-k33 sites · 15 references
One Mile at a Time×5 · 33%

onemileatatime.com

Sources are what each model recalled as having shaped its view — not verified citations. A model without web access reconstructs a reference from memory, so a link may not lead where the model thought it did.

Category · Business Class airlines
#2

Singapore Airlines

80
brand score
50first choice
12345

Summary

Singapore Airlines occupies a remarkably stable position across all four models, holding a median rank of 2 (Q1 2, Q3 2) over 20 ranked answers. Every model — claude-opus-5, gemini-3.7-flash, gpt-6-astra, and kimi-k3 — landed on the same median of 2, indicating broad agreement that the carrier is a near-top choice but not the single strongest recommendation. The shared themes are consistent: polished, world-renowned cabin service, exceptionally wide business-class seats, and the Book the Cook dining program, with several models also crediting the Changi hub experience. These recurring praises draw on sources such as the Skytrax World Airline Awards, One Mile at a Time, The Points Guy, and Condé Nast Traveler.

The disagreement is less about placement than about the reason it sits at second rather than first. Both gpt-6-astra and kimi-k3 explicitly frame it as trailing Qatar's Qsuite, citing seat privacy and sleeping configuration as the deciding factors, while claude-opus-5 and kimi-k3 emphasize the manual flip and diagonal sleeping position of the seat as its recurring drawback. Those hardware caveats, drawn from reviewer sources including One Mile at a Time and The Points Guy, temper otherwise strong soft-product assessments.

Extremely wide business class seats, excellent Book the Cook dining and famously polished cabin crew make it a benchmark carrier.

— claude-opus-5

It loses the top spot mainly because most of its business seats lack privacy doors and require sleeping diagonally, making it slightly less private than Qatar's Qsuite.

— kimi-k3

Per model summary

  • claude-opus-5
    brand score
    80
    first choice
    50
    median rank
    2.0
    present in
    5/5

    Consistently praised for polished cabin service, very wide business seats, and Book the Cook dining, while repeatedly noting the drawback that some seats require manual flipping to lie flat and older configurations lack direct aisle access.

  • gemini-3.7-flash
    brand score
    80
    first choice
    50
    median rank
    2.0
    present in
    5/5

    Emphasizes world-renowned cabin crew hospitality, wide lie-flat seating, the Book the Cook dining program, and consistency, along with the Changi hub experience.

  • gpt-6-astra
    brand score
    80
    first choice
    50
    median rank
    2.0
    present in
    5/5

    Regards it as an excellent all-round choice for polished service, dining, and spacious seats, but consistently ranks it just behind Qatar because seat privacy and sleeping comfort vary by aircraft and trail the Qsuite.

  • kimi-k3
    brand score
    80
    first choice
    50
    median rank
    2.0
    present in
    5/5

    Highlights exceptionally wide seats, polished service, and Book the Cook dining, while noting that seats require sleeping diagonally and often lack privacy doors, keeping it slightly behind Qatar's Qsuite.

Sources per model

Each model’s own references for this brand, grouped by site. Tile area is that site’s share of the model’s references; tap one for the pages behind it.

claude-opus-56 sites · 15 references
Skytrax World Airline Awards×5 · 33%

worldairlineawards.com

gemini-3.7-flash6 sites · 15 references
Skytrax World Airline Awards×5 · 33%

worldairlineawards.com

gpt-6-astra5 sites · 15 references
Skytrax World’s Best Business Class Airlines×5 · 33%

worldairlineawards.com

kimi-k35 sites · 15 references
Skytrax World Airline Awards×5 · 33%

worldairlineawards.com

Sources are what each model recalled as having shaped its view — not verified citations. A model without web access reconstructs a reference from memory, so a link may not lead where the model thought it did.

Category · Business Class airlines
#3

ANA

also named ANA (All Nippon Airways), All Nippon Airways, ANA All Nippon Airways

48
brand score
28first choice
12345

Summary

ANA lands consistently in the upper-middle of the field, with a median rank of 3 across 18 ranked answers from 4 models (Q1 3, Q3 4). Three of the four models — gemini-3.7-flash, gpt-6-astra, and kimi-k3 — each place it at a median of 3, while claude-opus-5 sits slightly lower at a median of 4 (Q1 4, Q3 4). The models agree closely on the underlying reasoning: the standout attraction is The Room, described as one of the widest, door-equipped business suites in commercial aviation, paired with attentive Japanese hospitality and strong dining. The recurring caveat that keeps ANA from ranking higher is limited availability — the best seat appears only on select aircraft and routes, making the experience flight-dependent — alongside a network described as more Japan- or Asia-centric.

That tension between an exceptional hard product and inconsistent access runs through every model's assessment. gpt-6-astra ties its third-place ranking directly to that limitation, while kimi-k3 and claude-opus-5 point to both fleet inconsistency and a thinner global network. The recalled sources cluster around a small set of aviation-review outlets, most frequently Skytrax World Airline Awards, One Mile at a Time, and The Points Guy, which appear under both the praise for The Room and the notes on route availability.

ANA pairs attentive service and excellent Japanese dining with an outstanding seat on aircraft equipped with The Room. It ranks third because that exceptionally spacious, enclosed configuration is available only on selected aircraft, making the recommendation more flight-dependent.

— gpt-6-astra

It ranks third because the product is only on select aircraft and routes, and its global network is thinner than the Gulf and Singapore carriers.

— kimi-k3

Per model summary

  • gemini-3.7-flash
    brand score
    56
    first choice
    32
    median rank
    3.0
    present in
    5/5

    Repeatedly emphasizes 'The Room' as one of the widest, most private business suites on select 777-300ERs, paired with Japanese hospitality and refined dining, with the main caveat being limited availability of the top product.

  • gpt-6-astra
    brand score
    60
    first choice
    33
    median rank
    3.0
    present in
    5/5

    Focuses on attentive service, excellent Japanese dining, and the standout spacious seat on aircraft with The Room, while ranking ANA third due to that cabin's limited availability making the experience flight-dependent.

  • kimi-k3
    brand score
    52
    first choice
    30
    median rank
    3.0
    present in
    5/5

    Consistently praises 'The Room' on 777s as one of the widest, door-equipped seats with strong Japanese service and catering, but notes fleet inconsistency, limited route availability, and a thinner network outside Asia-Pacific.

  • claude-opus-5
    brand score
    24
    first choice
    15
    median rank
    4.0
    present in
    3/5

    Consistently highlights 'The Room' as arguably the most spacious business seat with doors, plus excellent Japanese service and catering, while noting its limitation to select aircraft/routes and a more Japan/Asia-centric network.

Sources per model

Each model’s own references for this brand, grouped by site. Tile area is that site’s share of the model’s references; tap one for the pages behind it.

gemini-3.7-flash7 sites · 15 references
Skytrax World Airline Awards×4 · 27%

worldairlineawards.com

gpt-6-astra6 sites · 15 references
One Mile at a Time×4 · 27%

onemileatatime.com

  • ×3/
  • ×1named without a URL
kimi-k33 sites · 12 references
Skytrax World Airline Awards×5 · 42%

worldairlineawards.com

claude-opus-54 sites · 9 references
One Mile at a Time×3 · 33%

onemileatatime.com

Sources are what each model recalled as having shaped its view — not verified citations. A model without web access reconstructs a reference from memory, so a link may not lead where the model thought it did.

Category · Business Class airlines
#4

Emirates

41
brand score
25first choice
12345

Summary

Emirates settles in the upper-middle of the field, with an overall median rank of 4 (Q1 3, Q3 4) across 18 ranked answers from 4 models. Agreement on placement is fairly tight but not uniform: claude-opus-5 puts it highest at a median rank of 3 (Q1 3, Q3 3), gemini-3.7-flash and kimi-k3 both land at a median of 4, while gpt-6-astra is the outlier with a median of 5 (Q1 5, Q3 5). The narrative behind the numbers is remarkably consistent across all four — the brand is credited for its A380 onboard bar and lounge, ICE entertainment, chauffeur service, Dubai ground experience and vast network, then held back by one recurring flaw.

That flaw is the older 2-3-2 seating on much of the 777 fleet, which lacks direct aisle access and drives the perception of product inconsistency. This tension is what separates the more generous placements from gpt-6-astra's fifth-place ranking, where the seating variability weighs heaviest.

The A380 business cabin's 1-2-1 layout is good but older 777 cabins in 2-3-2 are a weak point, though the new 777 suites are much improved.

— claude-opus-5

It ranks fifth among these options because the seating experience varies substantially: some older Boeing 777 cabins lack direct aisle access for every passenger and offer less privacy than the leading alternatives.

— gpt-6-astra

The recalled sources cluster around industry review and awards bodies that underpin both the praise and the caveat: Skytrax World Airline Awards appears under every model, alongside One Mile at a Time and The Points Guy, with claude-opus-5 and gpt-6-astra also citing Emirates directly, and gemini-3.7-flash and kimi-k3 leaning on Business Traveller.

Per model summary

  • claude-opus-5
    brand score
    60
    first choice
    33
    median rank
    3.0
    present in
    5/5

    Emirates is praised for its A380 onboard bar/lounge, strong entertainment, excellent Dubai ground experience and vast network, but consistently criticized for the older 2-3-2 777 business cabins lacking direct aisle access, which newer retrofits are improving.

  • gemini-3.7-flash
    brand score
    44
    first choice
    27
    median rank
    4.0
    present in
    5/5

    Emirates is highlighted for its glamorous A380 onboard bar/lounge, ICE entertainment, chauffeur service and extensive Dubai network, with the recurring caveat that inconsistent 2-3-2 seating on older 777s keeps it from ranking higher.

  • kimi-k3
    brand score
    48
    first choice
    28
    median rank
    4.0
    present in
    5/5

    Emirates is noted for its glamorous A380 onboard bar/lounge, strong soft product, chauffeur service and Dubai network, but repeatedly flagged for inconsistency due to older 2-3-2 777 seats without direct aisle access.

  • gpt-6-astra
    brand score
    12
    first choice
    12
    median rank
    5.0
    present in
    3/5

    Emirates is valued for its extensive network, entertainment and A380 onboard lounge, but ranks fifth because seating varies substantially by aircraft, with older 777 cabins offering less privacy and no direct aisle access for every seat.

Sources per model

Each model’s own references for this brand, grouped by site. Tile area is that site’s share of the model’s references; tap one for the pages behind it.

claude-opus-56 sites · 15 references
Emirates×4 · 27%

emirates.com

gemini-3.7-flash8 sites · 15 references
Business Traveller×4 · 27%

businesstraveller.com

kimi-k35 sites · 13 references
Business Traveller×4 · 31%

businesstraveller.com

gpt-6-astra4 sites · 9 references
One Mile at a Time×3 · 33%

onemileatatime.com

  • ×2/
  • ×1named without a URL

Sources are what each model recalled as having shaped its view — not verified citations. A model without web access reconstructs a reference from memory, so a link may not lead where the model thought it did.

Category · Business Class airlines
#5

Cathay Pacific

14
brand score
10first choice
12345

Summary

Cathay Pacific occupies a mid-table position across the three models, landing at a median rank of 4 (Q1 4, Q3 5) over 9 ranked answers. The models agree on the underlying strengths — comfortable reverse-herringbone seating, the newer Aria Suite, and highly regarded Hong Kong lounges such as The Pier — but diverge on placement. gpt-6-astra holds it steadiest at a median of 4 across 5 answers, while gemini-3.7-flash and kimi-k3 both place it at 5. The gap is one of degree rather than direction: all three cite the same qualities, but weigh the shortcomings differently.

For gpt-6-astra, the ceiling is set by cabin generation, noting that older business-class cabins offer less privacy than leading enclosed suites and that the Aria Suite is not fleet-wide. kimi-k3 pushes it lower on service and recovery concerns, pointing to an aging hard product and inconsistent post-pandemic performance. The recalled sources cluster around Skytrax World Airline Awards, The Points Guy, and One Mile at a Time, with Cathay Pacific's own site and outlets like Business Traveller and Executive Traveller feeding the seat, lounge, and dining themes.

Its newer Aria Suite strengthens the offering, but varying cabin generations keep it below the top three for an unspecified itinerary.

— gpt-6-astra

It ranks last here because the hard product is aging, and service and catering have been inconsistent during its post-pandemic rebuild compared with the carriers above.

— kimi-k3

Per model summary

  • gpt-6-astra
    brand score
    40
    first choice
    25
    median rank
    4.0
    present in
    5/5

    Consistently frames Cathay Pacific as a well-rounded long-haul option with comfortable seating and excellent Hong Kong lounges, ranking it fourth because older cabin generations offer less privacy than leading suites and the newer Aria Suite is not fleet-wide.

  • gemini-3.7-flash
    brand score
    4
    first choice
    4
    median rank
    5.0
    present in
    1/5

    Emphasizes the well-designed reverse-herringbone seats and new Aria Suite, along with top-rated Hong Kong lounges like The Pier and quality bedding and dining.

  • kimi-k3
    brand score
    12
    first choice
    12
    median rank
    5.0
    present in
    3/5

    Notes comfortable reverse-herringbone seats and excellent Hong Kong lounges like The Pier, but ranks it last/fifth citing an aging hard product, inconsistent post-pandemic service, and slower recovery compared with rivals.

Sources per model

Each model’s own references for this brand, grouped by site. Tile area is that site’s share of the model’s references; tap one for the pages behind it.

gpt-6-astra6 sites · 15 references
Cathay Pacific×4 · 27%

cathaypacific.com

gemini-3.7-flash3 sites · 3 references
Executive Traveller×1 · 33%

executivetraveller.com

kimi-k33 sites · 7 references
Skytrax World Airline Awards×3 · 43%

worldairlineawards.com

Sources are what each model recalled as having shaped its view — not verified citations. A model without web access reconstructs a reference from memory, so a link may not lead where the model thought it did.

Category · Business Class airlines

How this was measured

The question

  • Unaided brand recommendation question: “Which airline would you recommend for business-class travel?
  • The prompt asks each model to return exactly 5 brands, ranked 1 to 5, and for each one a reason for the recommendation and the sources that informed it.

Sampling

  • Models are not deterministic, so the question is asked over and over — 5 answers per model
  • Spellings of the same brand are normalized to the most commonly used form before counting

Measures of Position: Median and Quartiles

  • Median value indicates that in 50% of answers the brand held this position or higher.
  • Quartiles help visualize the spread of rankings per brand. Q1 indicates that 25% of answers had this rank or higher. Q3 means that 75% of answers ranked the brand as X or better.
  • Medians and quartiles are computed only from answers where the brand was present.

Coverage

  • Count of answers in which the given brand was named, as a percentage.

Brand score - Normalized Borda score

  • The brand score is calculated using a normalized Borda score. Borda count is a voting method: each ballot awards points by position instead of naming one winner. Our ranking responses from the LLM are always fixed to 5 answers, so we assign 100 points for rank 1, then 80, 60, 40, 20 — and 0 if the brand is missing.
Brand score=100ni=1n6ri5\text{Brand\ score} = \frac{100}{n} \sum_{i=1}^{n} \frac{6 - r_i}{5}

where

ri={1,,5if the brand appears6if the brand is not mentionedr_i = \begin{cases} 1,\dots,5 & \text{if the brand appears} \\ 6 & \text{if the brand is not mentioned} \end{cases}
  • This way we can calculate a common score for all brands mentioned across model responses. The score reflects both how high the brand was ranked and how frequently it was mentioned.

First choice score - adjusted MRR - top-rank indicator

  • The first choice score is based on Mean Reciprocal Rank (a common search-engine measure), but adapted to measure the position of a specific brand across repeated ranked responses.
  • Each occurrence receives a reciprocal-position score: 100 for 1st, 50 for 2nd, 33.3 for 3rd, 25 for 4th, 20 for 5th, and 0 when the brand is not mentioned. The scores are averaged across all responses.
First choice score=100ni=1nsi\text{First\ choice\ score} = \frac{100}{n} \sum_{i=1}^{n} s_i

where

si={1/riif the brand is present0if the brand is absents_i = \begin{cases} 1/r_i & \text{if the brand is present} \\ 0 & \text{if the brand is absent} \end{cases}
  • No rank below 1st place gets more than 50, so the score is heavily driven by first places.
Your category

Ask your own question of every model

Tell us what to ask and how wide to sample it — we run it and send back the report.

Sample size
Display results
0/1000

Sampling every model takes up to 24 hours once we start.