Apple iPhone or Android

“Which brand would you recommend?” 2 options, 3 different contexts.

typesafe-ai/jev · 25 runs per context · 75 calls

Baselineconfidence61% ±3%

Apple iPhone81%±1%Android19%±1%

picked 25/25picked 0/25

Added contextconfidence98% ±0%

I already own a MacBook, an iPad and AirPods.

Apple iPhone99%±0%Android1%±0%

picked 25/25picked 0/25

Added contextconfidence10% ±6%

I drop my phone at least once a week.

Apple iPhone45%±3%Android55%±3%

picked 1/25picked 24/25

More evals

Other brands, same question

Each one is a pair put to the model under a handful of contexts, to see which of them moves the answer.

typesafe-ai/jev is an evaluation model: a chat model writes, Jev judges. It answers a typed question with one of the options, a probability across all of them, and how confident it was.
That is a number you can compare, rather than a paragraph you have to read for a verdict.

“Which brand would you recommend?” — Apple iPhone or Android. Once with no context, then again under 2 added contexts.
Nothing else changes between the bars, so what moves is what the context is worth.

Each context is asked 25 times — 75 answers in all. The bar is their mean probability, ± one standard deviation.
Confidence is Jev’s certainty about the call, not the split between the brands: it can be very sure the two are close.

A context is shown only once every one of its planned answers came back.

Free

Tell us which brands to put against each other and the contexts to ask under — we run it with Jev and send the numbers back.

Your category

Tell us what to ask and how wide to sample it — we run it and send back the report.