Test the score before trusting it.

What it saysAdobe researchers compute reference answers with code at test time, then compare individual facts rather than wording. On synthetic data, this agreed with three expert reviewers better than written instructions for computing the answer.

Why it mattersA stored answer can become wrong when prices, availability, or customer records change. The checker needs the same data state as the agent.

Try thisPick one changing-data question. Write a small, independently reviewed calculation for its expected answer.

What it saysIn restaurant-recommendation tests, more detailed accounts of the agent’s work made some AI judges reject more correct answers. Labeling which restaurant each review described helped, but did not remove every false alarm.

Why it mattersCatching bad answers is only half the job. Rejecting good ones creates needless rework and human review.

Try thisGive your judge known-good answers with short and detailed work records. Count incorrect rejections separately.

What it saysThis benchmark tests AI-written scoring rules on questions whose demanded conclusion cannot be supported. Tailored answers can earn high scores while violating the evidence. A later audit also found that failure rates depend strongly on the verifier.

Why it mattersRewarding confidence or completeness can penalize an honest limit. But an automated check of dishonesty needs validation too.

Try thisAdd one deliberately unanswerable question. Check whether a supported explanation of the limit beats an invented answer.