When the facts change, does the AI change its plan?
Published September 3, 2026
What it saysThis test gives assistants conflicting information from users, stored knowledge, and tools. The tested models did not reliably handle every kind of conflict.
Why it mattersMentioning a new fact is not enough. The assistant must use it in its next action.
Try thisCreate a test where an old availability result conflicts with a fresh one.
How to use it
Suggested steps
- Create a test where an old availability result conflicts with a fresh one.
- Check which result the assistant uses when it acts.
- Inspect the resulting record and review any mistaken decisions.
| Check | What to look for |
|---|---|
| Old result: available | Assistant has an initial plan |
| Fresh result: unavailable | Plan and action must update |
Keep in mind A controlled test does not cover every conflict a real product will encounter.
A quick check for yourself
What is the practical lesson?
Show answer
Mentioning a new fact is not enough. The assistant must use it in its next action.
Source & why it’s here
Original issue reading order, preserved from the archive.