Screen broadly. Confirm with real outcomes.

What it saysAdobe researchers compute reference answers with code at test time, then compare individual facts rather than wording. On synthetic data, this agreed with three expert reviewers better than written instructions for computing the answer.

Why it mattersA stored answer can become wrong when prices, availability, or customer records change. The checker needs the same data state as the agent.

Try thisPick one changing-data question. Write a small, independently reviewed calculation for its expected answer.

What it saysIn restaurant-recommendation tests, more detailed accounts of the agent’s work made some AI judges reject more correct answers. Labeling which restaurant each review described helped, but did not remove every false alarm.

Why it mattersCatching bad answers is only half the job. Rejecting good ones creates needless rework and human review.

Try thisGive your judge known-good answers with short and detailed work records. Count incorrect rejections separately.

What it saysThis benchmark tests AI-written scoring rules on questions whose demanded conclusion cannot be supported. Tailored answers can earn high scores while violating the evidence. A later audit also found that failure rates depend strongly on the verifier.

Why it mattersRewarding confidence or completeness can penalize an honest limit. But an automated check of dishonesty needs validation too.

Try thisAdd one deliberately unanswerable question. Check whether a supported explanation of the limit beats an invented answer.

What it saysNetflix uses AI to check explanations of show recommendations. Experts supply examples and reasons; the team adjusts the checker, then keeps comparing its decisions with fresh human reviews.

Why it mattersA checker that worked at launch can become unreliable as the content changes. Matching a human’s verdict is not enough if the reason is wrong.

Try thisReview a small sample of recent answers. Record where people and the checker disagree, including why.

What it saysPeople can use the same scoring rule and mean different things. MultEval lets a team compare disagreements, revise rules together, and keep the examples behind each decision.

Why it mattersAn AI checker cannot resolve a quality standard your team has never agreed on.

Try thisAsk two colleagues to review the same answers separately. Start the discussion with their disagreements.

What it saysAn AI assistant can say it completed a task without completing it. Anthropic recommends combining software checks, AI judgments, and human review, chosen for the task.

Why it mattersA convincing “done” message tells you little about whether a booking, file edit, or support request succeeded.

Try thisFor one task, define the evidence that would prove it worked. Check that evidence directly.

What it saysWhen comparing systems that find documents, many results overlap. This method saves earlier judgments and reviews only new documents, then compares every system using the updated shared collection.

Why it mattersEach round of testing can make the next round cheaper.

Try thisCompare the documents returned by two search versions.

What it saysThis test gives assistants conflicting information from users, stored knowledge, and tools. The tested models did not reliably handle every kind of conflict.

Why it mattersMentioning a new fact is not enough. The assistant must use it in its next action.

Try thisCreate a test where an old availability result conflicts with a fresh one.

What it saysIn a study of editing server configuration files, apparently successful edits could change the wrong thing. The proposed approach separates deciding what to change from applying the exact edit.

Why it mattersA correct request can still fail during execution.

Try thisChoose one action where the assistant changes a saved record.

What it saysThe researchers test two abilities: ignoring edits that do not change quality, and noticing edits that do. Judges could look stable while missing meaningful differences.

Why it mattersConsistency alone does not prove that a checker catches the problem you care about.

Try thisMake two copies of an answer: rephrase one and add a factual mistake to the other.

What it saysAI judges were compared with people reviewing voice conversations. Reliability depended on the industry, the question being scored, and the setup; some checks were unsuitable for full automation.

Why it mattersA checker that handles one question well may fail at another.

Try thisCompare human and AI reviews for each quality question separately.

What it saysA score for one quality can be influenced by another. This paper measures that problem and tests removing irrelevant evidence from the checker’s explanation.

Why it mattersSeparate score names do not guarantee separate judgments.

Try thisRephrase an answer without changing any facts.

What it saysThe study separates producing possible answers, recognizing a correct one, and choosing the final response. Changing the selection rule improved results without generating new answers.

Why it mattersFind the failing step before changing the model.

Try thisSave the alternatives your assistant considered for one failed task.

What it saysThis work makes file state explicit. What an assistant can read, edit, and submit affects its measured performance; different representations can otherwise refer to different versions.

Why it mattersA version mismatch can look like an AI mistake or hide one.

Try thisRecord which file version the assistant reads and edits.

What it saysAutomatic scoring ranked the compared systems in the same order as the human jury, but compressed the differences between weak and strong results.

Why it mattersA score can pick the right winner and still set the wrong pass mark.

Try thisCompare AI and human scores for weak, average, and strong answers.

What it saysThis benchmark follows assistants through hundreds of decisions in a long simulation. All tested models finished, yet the quality of their outcomes differed and became clearer late in the task.

Why it mattersA completion count can hide weak decisions along the way.

Try thisChoose one workflow with several linked decisions.

What it saysThis study compares an assistant with professional estimators on real building plans. It checks how much material was identified separately from how accurate the quantities were.

Why it mattersFinding more items and measuring them correctly are different abilities.

Try thisSplit your overall score into the qualities that matter.

What it saysJudges with similar overall accuracy differed sharply in detecting unwanted side effects. Catching more problems also sent more work to human reviewers.

Why it mattersA good average can hide the very failures you need to catch.

Try thisList your most important failure types.

What it saysIn analytics tasks, a stronger configuration answered more questions using real data while still showing unreliable choices during its work. The final answer alone missed these issues.

Why it mattersCheck how the assistant reached an answer as well as the answer itself.

Try thisSave the data and tool calls used for a task.

What it saysChanging the tools and control logic around a coding model changed its cost and failure patterns, even when task success rates were fairly close.

Why it mattersA model score is also a score for the setup around it.

Try thisRecord the model, instructions, tools, retry limits, and stopping rules for each test.

What it saysModels changed position across different agent benchmarks. A broad score can mix reasoning, tool use, and other abilities in ways that hide what improved.

Why it mattersChoose tests that reflect the work your users need done.

Try thisName the specific ability you need to measure.

What it saysWhen the same substance was written in different styles, judges changed their preferences. A style-aware checking step helped separate the content from its presentation.

Why it mattersAn answer should not get credit for sounding better when its facts are unchanged.

Try thisWrite two versions of an answer with the same facts but different styles.

What it saysThis study builds scoring rules from professional deliverables. Those rules aligned better with specialized work than rules created from a task prompt alone.

Why it mattersReal successful work gives you a stronger starting point than imagined standards.

Try thisCollect examples that experts agree are good work.

What it saysThe paper describes deciding whether to accept, reject, continue testing, or withhold a decision before every test has run. Some tasks reached a conclusion much sooner than others.

Why it mattersA small sample is useful only if it supports the decision you need to make.

Try thisSet the smallest improvement that would change your decision.

What it saysThis work describes maintaining a test collection by the abilities its cases cover. New examples can be added, replace weaker ones, or go to human review.

Why it mattersA growing collection should improve coverage without filling up with duplicate tests.

Try thisTag each example with the ability and failure type it tests.