Nike or Adidas

“Which brand would you recommend?” 2 options, 3 different contexts.

typesafe-ai/jev · 25 runs per context · 75 calls

Baselineconfidence74% ±2%

Nike87%±1%Adidas13%±1%

picked 25/25picked 0/25

Added contextconfidence10% ±6%

I care about how it looks, not how it performs.

Nike55%±4%Adidas45%±4%

picked 23/25picked 2/25

Added contextconfidence96% ±0%

I'm training for my first marathon.

Nike98%±0%Adidas2%±0%

picked 25/25picked 0/25

More evals

Other brands, same question

Each one is a pair put to the model under a handful of contexts, to see which of them moves the answer.

typesafe-ai/jev is an evaluation model: a chat model writes, Jev judges. It answers a typed question with one of the options, a probability across all of them, and how confident it was.
That is a number you can compare, rather than a paragraph you have to read for a verdict.

“Which brand would you recommend?” — Nike or Adidas. Once with no context, then again under 2 added contexts.
Nothing else changes between the bars, so what moves is what the context is worth.

Each context is asked 25 times — 75 answers in all. The bar is their mean probability, ± one standard deviation.
Confidence is Jev’s certainty about the call, not the split between the brands: it can be very sure the two are close.

A context is shown only once every one of its planned answers came back.

Free

Tell us which brands to put against each other and the contexts to ask under — we run it with Jev and send the numbers back.

Your category

Tell us what to ask and how wide to sample it — we run it and send back the report.