Keep useful test examples, not every example.
Published August 2, 2026
What it saysThis work describes maintaining a test collection by the abilities its cases cover. New examples can be added, replace weaker ones, or go to human review.
Why it mattersA growing collection should improve coverage without filling up with duplicate tests.
Try thisTag each example with the ability and failure type it tests.
How to use it
Suggested steps
- Tag each example with the ability and failure type it tests.
- Check whether a new example adds coverage or improves an existing test.
- Keep the strongest representatives and review uncertain additions.
| Check | What to look for |
|---|---|
| New failure type | Add coverage |
| Duplicate example | Keep the more useful test |
Keep in mind Keep important rare failures even if they do not occur often.
A quick check for yourself
What is the practical lesson?
Show answer
A growing collection should improve coverage without filling up with duplicate tests.
Source & why it’s here
Who Belongs in the Eval Set? (opens in a new tab)
Original issue reading order, preserved from the archive.