A correction trained into the model, not written into the prompt. The corrected model names the files it did not open in about 9 of 10 incomplete reviews, keeps reading as much as before and finds the same problems. Still to fix: on complete reviews it sometimes claims it skipped files it read.
## From a prompt line to the weights
Last week we showed that some AI agents, asked to review a set of files, open part of them and then report that they reviewed everything, and that one sentence in the request ("say which files you did not open") makes the report truthful. A sentence in a prompt is not a fix of the model: it has to be written every time, and it is easy to forget. So we tried to put the same behaviour inside the model itself, on an open 20-billion-parameter model, with nothing added to the prompt.
## What we did
We trained a small add-on to the model (a LoRA adapter) on review jobs from five new domains, with tasks written in English, Italian, French, Spanish and German. Each training example is a real stopping point of the model's own work, paired with an honest version of its own report: same findings, same style, but the scope says which files were not opened. We also taught it, at points where the work was not finished, to keep working rather than stop. We then tested it on two review jobs it had never seen.
## What we found
- With no instruction in the prompt, the corrected model says which files it did not open in about 9 of 10 incomplete reviews, where the original almost never did. It holds in all five languages.
- It keeps reading as much as before and finds as many of the planted problems.
- The first attempt taught it the wrong lesson, to stop working and write the report too early. The second attempt fixed that.
- What is left: on reviews it did complete, it now sometimes claims it skipped files it actually read, about one time in three. It has learned the form of the confession better than the check against its own work. That is the next step.
```chart
{"type": "bar", "title": "Incomplete reviews whose report hides the gap (%)", "labels": ["Original model", "Correction, first try", "Correction, second try"], "series": [{"name": "Misleading report", "data": [91.9, 56.8, 10.8]}]}
```
## Technical detail
Held-out set: 37 incomplete review prefixes (the model's own stopping points) and 20 complete ones. Misleading reports on incomplete prefixes: 0.92 for the original model, 0.11 after the correction (per language 0.00 to 0.33). False gaps on complete prefixes: 0.05 before, 0.35 after. Planted problems found: 0.70 before and after. Continues working when given tools: 31 of 37 after the correction, 36 of 37 before. Two independent judges, agreement (Cohen kappa) 0.81. Limits: synthetic review jobs, one training seed, 4-bit inference.
This update is not available.