Ask a model to complete "The Eiffel Tower is in the city of" and it answers "Paris", 85% sure. Where does that answer come from? We follow it up through the 24 layers of a real model: a notebook that every layer writes in, a bank of detectors, and an answer that only appears near the top. Part 2 of our series on how transformers work.
## The setup
*The Eiffel Tower is in the city of...* A small open model, Qwen2.5-0.5B, completes it with **"Paris", 85% sure**. In [part 1](/news/tutorials/how-a-transformer-reads-attention) we saw attention, how words share information. Attention is one step, repeated in 24 layers. Here is what happens across them, with the model's own numbers.
## 1. The residual stream: a notebook passed from desk to desk
Each token carries a vector of 896 numbers up through the layers. No layer replaces it: each one **reads it and adds something**. Nothing is erased, every desk writes a little more, which is why the vector grows as it climbs.
## 2. Inside one layer: two parts
Every layer has the same two parts. First **attention**, where the token looks at the others and takes what it needs. Then the **feed-forward block**, where each token works on its own. For the last word of our sentence, the feed-forward block writes more than attention in every layer.
## 3. The feed-forward block: a bank of detectors
It stretches the 896 numbers into **4,864**, each one a detector checking for a pattern, then folds them back into 896. In layer 22, for the word "of", only **39** of the 4,864 detectors fire strongly; the rest stay almost silent. Research suggests that much of what a model knows, like facts about places, is stored in these blocks.
## 4. Watching the answer form
We can read the notebook at any layer as if it were the last, and ask: what would the model say now? Early on the guess is noise, fragments of words, with "Paris" thousands of places down the list. Then, near the top, it climbs fast: **position 162 at layer 21, first place at layer 22, 85% at layer 24**. The answer is not looked up in one place. It is built, layer by layer.
## 5. From vector to word
The final vector is multiplied by the vocabulary matrix: one score for each of 151,936 rows. Softmax turns the scores into probabilities. "Paris" gets 85%; next come "Le", "Mar", "Lou" and "France", all under 2%. A setting called **temperature** decides how strictly the model takes the top word or samples among the others.
## 6. One word at a time
The chosen word is added to the sentence and the whole stack runs again: Paris, a comma, France, a full stop. **Every word is a full trip through all 24 layers.** To save work, the model keeps the keys and values it already computed for earlier words (the "KV cache") and only computes them for the new one. That cache is why long conversations cost memory, and why speed drops as a chat grows.
## The whole model in code
```python
x = embed(tokens) # tokens become vectors of 896 numbers
for layer in layers: # 24 times
x = x + layer.attention(x) # share information between tokens
x = x + layer.feed_forward(x) # each token works on its own
probs = softmax(x[-1] @ vocab.T) # one score per word in the vocabulary
next_token = pick(probs) # then add it and run again
```
## Why this matters for behaviour
A model's answer is the last step of a long build-up, and its habits live along the way: in which layer an answer settles, in which detectors fire. That is where we read a model when we have access to it, and where a correction in the weights takes effect. Next: [how a model learns all this in the first place](/news/tutorials/how-a-transformer-is-trained).
See it animated in our explainer [How a model writes its answer](/llm-inference).
This update is not available.