A model is not programmed, it is trained: billions of tiny corrections to its weights, one guessed word at a time. Then two more rounds turn a text predictor into an assistant. Here is the whole path in plain words, with the few lines of code at its heart, and the point where habits like flattery creep in. Part 3 of our series on how transformers work.
## Nobody writes the rules
In [part 1](/news/tutorials/how-a-transformer-reads-attention) and [part 2](/news/tutorials/how-a-transformer-writes-inference) we saw a model read and write. Every number it used, the 896-wide vectors, the attention matrices, the 4,864 detectors per layer, was **learned**, not written by anyone. Training is how. It happens in three rounds.
## Round 1: pre-training, guessing the next word
The model is shown huge amounts of text, cut into tokens, and asked one thing over and over: **what comes next?** At first its guesses are random. After each guess it is told how wrong it was (the **loss**), and every weight is nudged a tiny bit in the direction that would have made the guess better. That nudge is computed by **backpropagation**: blame for the error is passed back through all the layers, weight by weight.
Repeat this over trillions of tokens (Qwen2.5, the family in our examples, was pre-trained on up to 18 trillion) and something remarkable happens: to predict text well, the model has to pick up grammar, facts, styles, even some reasoning. Nobody teaches them; they are what helps the next guess.
```python
for batch in data: # billions of sentences, cut into tokens
logits = model(batch.tokens[:, :-1]) # guess every next token
loss = cross_entropy(logits, batch.tokens[:, 1:]) # how wrong was each guess
loss.backward() # who is to blame, weight by weight
optimizer.step() # nudge every weight a little
optimizer.zero_grad()
```
These few lines, run on thousands of GPUs for weeks, are the whole of pre-training.
## Round 2: instruction tuning
A pre-trained model completes text; it does not answer you. In the second round it is trained on many examples of a request followed by a good answer, in the same way: guess the next token, measure the error, nudge the weights. Far less data, far fewer steps. Afterwards it follows instructions and keeps the format of a conversation.
## Round 3: preference tuning
Now people (or a model standing in for them) compare pairs of answers and say which is better, and the model is tuned toward the preferred ones. This is where it learns to be helpful, polite and careful, through methods with names like RLHF and DPO.
## Where bad habits come from
The same round plants the habits we read for. People tend to prefer answers that agree with them and sound confident, so a model can learn that **agreeing pays** ([sycophancy](/news/tutorials/ai-sycophancy-explained-five-minute-check)) and that a fluent invented answer scores better than "I don't know". Nothing in the loss says "be honest when the user pushes back"; the model learns whatever the ratings reward.
## Fine-tuning: a fourth, small round
Anyone can run one more round on top: a few thousand examples, often through small add-on weights (adapters such as LoRA) instead of the full model. That is how companies adapt a model to their domain, and also how a careless tune can erode what earlier rounds built. It is also our tool: we fine-tune on examples of the **right** behaviour, and then check that nothing else got worse.
## Go further
- Animated explainer: [How a model is trained](/llm-training) and [How we fine-tune a model](/fine-tuning)
- [Fine-tuning and alignment](https://xbe.at/llms-and-genai/topic/fine-tuning-and-alignment/) on xbe.at
This update is not available.