Screen broadly. Confirm with real outcomes.

What it saysNubank compared agent versions through multi-turn synthetic customers and mocked tool responses. A configuration selected after more than 16,000 simulated conversations later improved self-service by 8.82 percentage points in a live test, with no significant change in customer satisfaction.

Why it mattersThis adds a useful layer between hand-written tests and exposing real customers. Simulation finds weak candidates cheaply; customer behavior still confirms the release.

Try thisChoose one failure hypothesis. Run the incumbent and candidate through the same 100 simulated journeys before a small live test.

What it saysIn the main ten-judge test, shared errors reduced the independent information to roughly 3.5 judges. In up to 28% of system comparisons, a claimed significant difference disappeared after the analysis accounted for those shared mistakes.

Why it mattersAdding more judges can create false confidence when they fail on the same examples. Using models from different providers does not guarantee independent evidence.

Try thisRun every judge on about 100 trusted examples. Look for cases that fool several judges before choosing a voting rule.

What it saysA cheap first-pass judge accepted confident decisions and sent uncertain ones to a stronger judge. On pooled public tasks, one threshold escalated 34% of cases and reached 91.3% accuracy versus 91.7% for always using the stronger judge, at 47% of its fee.

Why it mattersA calibrated escalation rule can reduce evaluation cost without treating every case as equally easy. Confidence is useful for routing, not proof that a verdict is right.

Try thisTest a confidence threshold on local human-labeled examples. Track both accuracy and how many cases require escalation.