Case-specific rules can outperform a generic safety label.
Published Oct 21, 2026
What it saysOn 236 human-labeled responses, three rule-guided judges reached 89.4%–95.3% agreement, compared with 80.5%–87.3% for generic prompts. Each prompt’s scoring guide separated prohibited behavior, allowed exceptions and usefulness instead of asking for one broad safe-or-unsafe verdict.
Why it mattersA named policy category is not enough guidance for borderline cases. Concrete rules and exceptions make disagreements easier to diagnose and reduce the chance that caution simply replaces usefulness.
Try thisFor one high-risk intent, write the prohibited behavior, valid exceptions and minimum useful response as separate checks.
How to use it
Suggested steps
- Translate the policy into observable yes-or-no items.
- Keep safety and usefulness as separate outputs.
- Compare the judge with expert labels, including borderline exceptions.
| Separate check | Question |
|---|---|
| Prohibited behavior | Did the response enable the named harm? |
| Allowed exception | Does a policy exception apply? |
| Usefulness | Did the safe response still help? |
Keep in mind The prompts were synthetic and the tested systems were locally served open-weight models without production safeguards. The study covers dialogue, not tool-using agents, and model-generated rule specifications may miss rare risks.
A quick check for yourself
Why score safety and usefulness separately?
Show answer
A refusal can be safe yet unhelpful; one label hides that trade-off.
Source & why it’s here
Second: strong human-alignment evidence for turning policy language into specific evaluation criteria.