Agree on “good” before asking AI to score it.
Published Apr 29, 2026
What it saysPeople can use the same scoring rule and mean different things. MultEval lets a team compare disagreements, revise rules together, and keep the examples behind each decision.
Why it mattersAn AI checker cannot resolve a quality standard your team has never agreed on.
Try thisAsk two colleagues to review the same answers separately. Start the discussion with their disagreements.
How to use it
Suggested steps
- Choose examples that split opinion.
- Ask what each person understood the rule to mean.
- Rewrite the rule together; save a clear example beside it.
| Vague rule | A clearer question |
|---|---|
| “Be helpful” | Did the answer resolve the user’s request? |
| “Be accurate” | Which claim does the source support? |
Keep in mind A study of collaboration and a prototype, not proof of a universal accuracy gain.
A quick check for yourself
What does reviewer disagreement reveal?
Show answer
A rule may need clarification, not just a different AI model.
Source & why it’s here
Explains the human work needed before Netflix’s approach can work.