← All lab updates
Why models invent facts, and how to catch them at it
Language models invent books, sources and figures with complete confidence. It is not a glitch: it follows from how they are trained. Here is why it happens, what our own tests found (one open model invents a specific answer to an unanswerable question about one time in three), and four checks you can run on any assistant.
## Why it happens
A model is trained to predict the next word of real text ([how a transformer is trained](/news/tutorials/how-a-transformer-is-trained)). Real text rarely says "I don't know": it states things. So the model learns to continue fluently, and a fluent, specific answer is what a question seems to call for, whether or not the model has the fact. Later rounds of training reward answers people like, and people tend to rate a confident answer above an honest "I'm not sure".
## What we found
**Unanswerable questions.** We built 16 pairs of questions in five languages: one answerable, and a twin built on a false premise, an invented name or an unknowable fact. Both open models answered every real question correctly. On the twins:
```chart
{"title": "Unanswerable questions answered with a specific invented answer", "unit": "%", "series": ["Share of answers"], "rows": [["Larger open model", 33], ["Smaller open model", 14]], "note": "Approximate rates from 16 question pairs in five languages; the larger model about 1 in 3, the smaller about 1 in 7.", "max": 100}
```
Changing only the wording of the question moved the result more than asking the same question twice ([the write-up](/lab/knowing-when-not-to-answer)).
**Summaries.** When a model summarises a report, the writer's doubt often disappears: a plain summary turned "it may have been" into "it was" about 4 times in 10 for one model and 3 in 10 for the other ([summaries drop the doubt](/lab/summaries-drop-the-doubt)).
**Reading it from the text.** Inside one answer, our reading ranks the invented sentence above a correct one about 7 times in 10, because made-up content is written in a recognisable way. It does not check whether a fact is true: that needs a source ([the write-up](/lab/hallucination-reading-on-human-labelled-answers)).
## Four checks for any assistant
1. **Ask about something that does not exist,** a book, a regulation, a product code. Does it say so?
2. **Rephrase the same question** three ways. Does the answer change?
3. **Ask for the source** of each figure, then open it.
4. **Summarise a document with doubts in it,** and see whether the doubts survive.
## Go further
- [RAG explained](/news/tutorials/rag-explained): answering from your own documents, and its limits
- [Addressing hallucinations in large language models](https://xbe.at/llms-and-genai/addressing-hallucinations-in-large-language-models/) on xbe.at
This update is not available.