Getting the order right does not make the scores right.
Published August 18, 2026
What it saysAutomatic scoring ranked the compared systems in the same order as the human jury, but compressed the differences between weak and strong results.
Why it mattersA score can pick the right winner and still set the wrong pass mark.
Try thisCompare AI and human scores for weak, average, and strong answers.
How to use it
Suggested steps
- Compare AI and human scores for weak, average, and strong answers.
- Check the size of the gaps as well as the order.
- Review examples near your release pass mark with people.
| Check | What to look for |
|---|---|
| Who performed better? | Compare the ranking |
| How much better? | Compare the scoring scale |
Keep in mind The study uses competition submissions. Your product needs its own human comparison.
A quick check for yourself
What is the practical lesson?
Show answer
A score can pick the right winner and still set the wrong pass mark.
Source & why it’s here
The IOL-AI Challenge (opens in a new tab)
Original issue reading order, preserved from the archive.