A steady score can still miss a mistake.
Published August 25, 2026Read first
What it saysThe researchers test two abilities: ignoring edits that do not change quality, and noticing edits that do. Judges could look stable while missing meaningful differences.
Why it mattersConsistency alone does not prove that a checker catches the problem you care about.
Try thisMake two copies of an answer: rephrase one and add a factual mistake to the other.
How to use it
Suggested steps
- Make two copies of an answer: rephrase one and add a factual mistake to the other.
- Check that rephrasing leaves its accuracy score unchanged.
- Check that the introduced mistake lowers the score.
| Check | What to look for |
|---|---|
| Only the wording changes | Accuracy score stays the same |
| A fact becomes wrong | Accuracy score gets worse |
Keep in mind The result depends on the kinds of edits tested and the chosen scoring rule.
A quick check for yourself
What is the practical lesson?
Show answer
Consistency alone does not prove that a checker catches the problem you care about.
Source & why it’s here
A Judge Should Know What Changed (opens in a new tab)
Original issue reading order, preserved from the archive.