Test the judge, then test the evidence.

What it saysThe authors crossed ten reasoning models with different prompts and scoring scales. Average answer-grading accuracy was 96.5%, yet switching models changed the share of errors that were overly forgiving by as much as 56.1 percentage points. Human ratings anchored the sentiment and toxicity tests.

Why it mattersTwo judges can report similar accuracy while sending very different mistakes to production. An aggregate score can hide whether a checker is too harsh or too forgiving.

Try thisRun two judge models on the same trusted examples. Compare false acceptances and false rejections, not just accuracy.

What it saysTwo security experts found fabricated or unverifiable claims in about 60% of correct vulnerability verdicts. The proposed checker turns free text into a structured record, then applies deterministic and semantic checks across eight failure types. It exposed 87% of errors missed by a free-text AI judge.

Why it mattersA final pass can be lucky. A compact record of evidence, state changes and assumptions makes hidden gaps testable without trusting persuasive prose.

Try thisFor one critical workflow, require a short structured record of evidence, actions and state changes. Check the fields separately from the final answer.

What it saysThe study held arithmetic and numbers fixed while moving citations between table cells with the same value. The checker sometimes accepted the wrong financial role and sometimes withheld valid alternatives. On 36 fresh source pages, 25 of 53 reviewer-supported citations were withheld at the chosen cutoff.

Why it mattersMatching a value does not prove that its source supports the claim. Period, unit and business role need their own check, while valid alternative evidence must not be rejected blindly.

Try thisTake one passing evidence-based answer. Swap its citation for a same-value cell with a different role, then test whether your checker notices.