Slack or Microsoft Teams

“Which brand would you recommend?” 2 options, 3 different contexts.

typesafe-ai/jev · 25 runs per context · 75 calls

Baselineconfidence33% ±3%

Slack67%±2%Microsoft Teams33%±2%

picked 25/25picked 0/25

Added contextconfidence99% ±0%

Our company already pays for the Microsoft 365 suite.

Slack0%±0%Microsoft Teams100%±0%

picked 0/25picked 25/25

Added contextconfidence8% ±7%

I'm a designer at a 10-person startup and half the team is on personal Gmail.

Slack54%±4%Microsoft Teams46%±4%

picked 24/25picked 1/25

More evals

Other brands, same question

Each one is a pair put to the model under a handful of contexts, to see which of them moves the answer.

typesafe-ai/jev is an evaluation model: a chat model writes, Jev judges. It answers a typed question with one of the options, a probability across all of them, and how confident it was.
That is a number you can compare, rather than a paragraph you have to read for a verdict.

“Which brand would you recommend?” — Slack or Microsoft Teams. Once with no context, then again under 2 added contexts.
Nothing else changes between the bars, so what moves is what the context is worth.

Each context is asked 25 times — 75 answers in all. The bar is their mean probability, ± one standard deviation.
Confidence is Jev’s certainty about the call, not the split between the brands: it can be very sure the two are close.

A context is shown only once every one of its planned answers came back.

Free

Tell us which brands to put against each other and the contexts to ask under — we run it with Jev and send the numbers back.

Your category

Tell us what to ask and how wide to sample it — we run it and send back the report.