A judge needs tests built for its exact rule.
Published Oct 20, 2026Read first
What it saysSANE reads a judge’s specification and creates controlled changes that should preserve or move its score. Across five criteria it produced 77 validated test families versus 21 for a fixed-library baseline. Its faithfulness tests ranked 11 judges similarly to a human-labeled benchmark (Spearman correlation 0.852 versus 0.100).
Why it mattersAverage agreement can hide brittle behavior. A targeted test tells you whether a judge reacts correctly when the exact thing it claims to measure changes—or stays the same.
Try thisChoose one judge. Write one change that should preserve its verdict and one that should reverse it, then run both before the next release.
How to use it
Suggested steps
- State the judge’s criterion and output contract plainly.
- Create paired examples with an expected same, higher or lower score.
- Validate the changes independently before counting a judge failure.
| Controlled change | Expected behavior |
|---|---|
| Paraphrase only | Score stays stable |
| Add grounded support | Faithfulness score rises |
| Repeat the same input | Verdict remains consistent |
Keep in mind Generated recipes and cases were validated by model panels, not people. Human alignment was tested only through judge rankings on faithfulness, not case-by-case or across the other four criteria.
A quick check for yourself
What makes a useful judge stress test?
Show answer
A controlled input change with an independently justified expected score relation.
Source & why it’s here
SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges (opens in a new tab)
Read first: the most direct and reusable method this week for testing whether an AI judge behaves like a measurement instrument.