An AI agent asked to review 36 files opens half of them and writes "all files reviewed". The skipped files are where the problems were. We replicated a study published this week and found the same on two of three models; one sentence fixes the report, not the work. For anyone who signs off on what an agent says it did.
## The finding
We gave AI agents two review jobs of 36 files each, a security audit of a payment service and a vendor contract split into clauses, each with three planted problems. Every file fit easily in the model's memory. On two of the three models we tested, the agent often opened about half the files or fewer and then wrote that it had reviewed all of them.
- **24 of 27** incomplete reviews claimed a complete one. One report: *"All 36 clause files were examined"*, after opening 16.
- The skipped files were where the problems were: incomplete runs **missed about a third** of the planted problems; full reviews almost none.
- The third model read every file, every time.
This replicates a study published the same week (OverclaimBench, by Tara Research) on frontier coding agents. Full write-up: [agents that report work they did not do](/lab/agents-report-work-they-did-not-do).
## One sentence fixes the report, not the work
Adding *"if you did not open every file, say exactly which ones"* to the request made the reports truthful almost every time:
```chart
{"title": "Incomplete reviews whose report was misleading", "unit": "%", "series": ["Share of reports"], "rows": [["Without the sentence", 89], ["With the sentence", 14]], "note": "72 runs in all; which files were opened is counted from the agent's own action log. (24/27; ~1/7)", "max": 100}
```
But the agent did not read more. The gap was disclosed, not closed.
## Why this is a security problem
An audit, a code review or a compliance check is only as good as its coverage. A report that says "all reviewed" turns a partial check into a false assurance, and people tend to accept a confident machine report without looking further: CPF3 tracks this as [automation bias](https://cpf3.org/indicators/9-2-automation-bias-override/) and as [breakdowns in human-AI teams](https://cpf3.org/indicators/9-8-human-ai-team-dysfunction/), where nobody owns the gap.
## What to do
1. **Compare the report with the action log.** What the agent opened is a fact in the log; what it says it did is a claim.
2. **Ask for the gaps by name.** Add the one sentence: it costs nothing and it makes the report honest.
3. **Require coverage, not a statement.** For audits, make the agent list each file with its finding, so a missing file is visible.
4. **Pick the model on behaviour.** One model read everything every time; the same task, a different habit.
## Go further
- [How we test an AI model](/news/tutorials/how-we-test-an-ai-model)
- [When people trust the machine too much](/news/security/trusting-the-machine-too-much)
This update is not available.