Pinning a judge does not make every verdict stable.
Published Sep 27, 2026
What it saysAcross four frontier judges and three benchmarks, identical temperature-zero requests still changed verdicts on roughly 5% of items on average and about 40% of close calls. The authors recommend at least five judge reruns plus uncertainty around adjacent rankings.
Why it mattersA small score difference may reflect the measurement process, not a better agent. A fixed model name and temperature are not enough evidence of repeatability.
Try thisRerun your judge at least five times on borderline release comparisons. Report how often the verdict flips.
How to use it
Suggested steps
- Hold candidate answers and prompts fixed across judge reruns.
- Report item flip rate and confidence intervals, not only a point score.
- Repeat with a second judge family and keep human agreement as a separate test.
| Repeated result | What to report |
|---|---|
| Same verdict each run | Stable on this sample |
| Verdict flips | Flip rate and interval |
| Judges disagree | Separate profiles; do not average blindly |
Keep in mind All judges used one enterprise cloud; provider caching was not controlled, aliases were not dated snapshots, and some long answers were truncated. The study tests consistency, not agreement with people.
A quick check for yourself
Does temperature zero guarantee a repeatable judge verdict?
Show answer
No. Measure repeatability with identical reruns.
Source & why it’s here
Third: an inexpensive repeatability check for close release or leaderboard decisions.