← All lab updates
Quantization, explained: a smaller model keeps its test score, but not always its behaviour
Quantization stores a model's weights with fewer bits, so a model that needed a data-centre GPU fits on a laptop or a phone. Builders report that test scores barely move. We checked something else on a 27-billion-parameter model squeezed to 2 bits: whether it still holds its ground under pressure. It does not, always. How quantization works, the memory maths, and what to test after compressing.
## What it is
Every weight in a model is a number. During training it is usually stored with 16 bits. **Quantization** stores it with fewer, 8, 4 or even 2, by rounding each weight to one of a small set of values. Fewer bits per weight means less memory and faster answers.
## The memory maths
| Model size | 16 bits | 4 bits | 2 bits |
|---|---|---|---|
| 7 billion weights | about 14 GB | about 3.5 GB | about 1.8 GB |
| 27 billion weights | about 54 GB | about 13.5 GB | about 7 GB |
At 2 bits a 27-billion-parameter model is about seven times smaller: small enough for a single consumer GPU.
## What gets reported, and what doesn't
Builders of compressed models report accuracy on test sets, and it often drops only a little. Accuracy answers "does it still know the facts?". It does not answer "does it still hold its ground when a user insists on something false?".
## What we found on a 2-bit model
We ran a 27B open model and its popular 2-bit build on the same conversations in five languages: users pushing a well-known false claim over six turns, and questions about books, drugs and treaties that do not exist.
```chart
{"title": "Pressured turns where the model gave up a correct answer", "unit": "%", "series": ["Share of turns"], "rows": [["Full model", 0.0], ["2-bit build", 5.5]], "note": "400 pressured turns per model, two independent judges agreeing 95% of the time.", "max": 10}
```
The full model never gave in. The 2-bit build gave up a correct answer on **22 of 400** pressured turns, about 1 in 18, always after the user contradicted it. A typical reply: *"You are correct. Lightning never strikes the exact same point twice"*, followed by an invented explanation. It also added invented detail to questions about non-existent things more often (about 1 answer in 5 against 1 in 10), a difference not yet beyond chance on this sample. Full write-up: [the 2-bit test](/lab/two-bit-model-caves-under-pressure).
## What to test after compressing
1. Your own conversations, not only a benchmark.
2. Pressure over several turns: the cases where a user insists.
3. Every language you serve.
4. Questions with no answer: does it invent, or say it doesn't know?
## Go further
- [How a transformer writes its answer](/news/tutorials/how-a-transformer-writes-inference), to see what the weights do
This update is not available.