Fine-tuning a whole model means updating billions of numbers. LoRA freezes all of them and trains a small add-on instead, often under 2% of the size. It is how most fine-tunes are made today, including ours. How it works with real sizes, the few lines of code behind it, and the side effect nobody declares.
## The problem
A model like Qwen2.5-0.5B has about half a billion weights; a 7B model has fourteen times more. Updating all of them for one task needs a lot of GPU memory, and every fine-tuned copy is as big as the original.
## The idea
**LoRA** (low-rank adaptation, Hu et al., 2021) leaves the original weights **frozen** and adds, next to each chosen matrix, two thin matrices that are trained instead. Their product is a small correction added to what the original matrix computes. On GPT-3 the authors cut the number of trained weights by about **10,000 times** and the GPU memory by about **3 times**, with quality on par with full fine-tuning.
## With real sizes
One attention matrix in our small model is 896 by 896: **802,816** numbers. A LoRA adapter of rank 8 next to it is two matrices, 896 by 8 and 8 by 896: **14,336** numbers, under 2%.
```chart
{"title": "Numbers trained for one 896 x 896 matrix", "unit": "", "series": ["Trained numbers"], "rows": [["Full fine-tuning", 802816], ["LoRA, rank 8", 14336]], "note": "The rank decides how big the correction can be; 8 to 64 are common."}
```
## The code behind it
```python
W = model.layer.weight # 896 x 896, frozen: never changes
A = randn(896, 8) * 0.01 # trainable, small
B = zeros(8, 896) # trainable, starts at zero: no change at first
def forward(x):
return x @ W + (x @ A @ B) * scale # the original output, plus a small correction
```
B starts at zero, so at step one the model behaves exactly as before; training then moves only A and B.
## Why teams use it
- **Small files.** An adapter is megabytes, not gigabytes: one base model, many adapters, swapped as needed.
- **Cheaper training.** A single consumer or free-tier GPU is enough for small models.
- **Reversible.** Remove the adapter and the original model is back, unchanged. Or merge it in for speed.
## The blind spot
A small adapter can still change a lot. When we compared twenty public open-source models, each retrained for a single task, with the model they came from, **every one we could compare had changed somewhere else too**: some became far more willing to help with dangerous requests, and none said so ([the write-up](/lab/public-fine-tunes-side-effects)). That is why, when we fine-tune, we measure the behaviours the task was not about, before and after.
## Go further
- [How a transformer is trained](/news/tutorials/how-a-transformer-is-trained), for the rounds that come before
- [Fine-tuning](/products/fine-tuning), how we use it to correct a model
- [Fine-tuning and alignment](https://xbe.at/llms-and-genai/topic/fine-tuning-and-alignment/) on xbe.at
This update is not available.