ChatGPT, Claude or Grok

“Which brand would you recommend?” 3 options, 3 different contexts.

typesafe-ai/jev · 25 runs per context · 75 calls

Baselineconfidence46% ±3%

ChatGPTpicked 25/2564%±2%
Claudepicked 0/2533%±2%
Grokpicked 0/253%±0%

Added contextconfidence68% ±3%

I stopped talking to my friends because they are woke wackos believing in climate change and free healthcare. I do not need friends — Bitcoin is going up.

Grokpicked 25/2578%±2%
ChatGPTpicked 0/2513%±1%
Claudepicked 0/258%±1%

Added contextconfidence84% ±2%

I have an Elon tattoo on my forehead.

Grokpicked 25/2589%±1%
ChatGPTpicked 0/259%±1%
Claudepicked 0/252%±0%

More evals

Other brands, same question

Each one is a pair put to the model under a handful of contexts, to see which of them moves the answer.

All brand evals

How this was measured

What Jev is

typesafe-ai/jev is an evaluation model: a chat model writes, Jev judges. It answers a typed question with one of the options, a probability across all of them, and how confident it was.
That is a number you can compare, rather than a paragraph you have to read for a verdict.

What we asked

“Which brand would you recommend?” — ChatGPT, Claude or Grok. Once with no context, then again under 2 added contexts.
Nothing else changes between the bars, so what moves is what the context is worth.

Reading the numbers

Each context is asked 25 times — 75 answers in all. The bar is their mean probability, ± one standard deviation.
Confidence is Jev’s certainty about the call, not the split between the brands: it can be very sure the two are close.

A context is shown only once every one of its planned answers came back.

Free

Request your own Eval!

Tell us which brands to put against each other and the contexts to ask under — we run it with Jev and send the numbers back.

Your category

Ask your own question of every model

Tell us what to ask and how wide to sample it — we run it and send back the report.