In "the animal didn't cross the street because it was too tired", you know "it" is the animal. A language model works it out with a mechanism called attention. We follow it step by step inside a real open model, Qwen2.5-0.5B, with the numbers it actually computes. Part 1 of our series on how transformers work.
```youtube
x1eBqygpswg
```
## The question
*The animal didn't cross the street because it was too tired.* What does "it" refer to? You know it is the animal. A language model has to work it out, and the way it does it is called **attention**. Below are the seven steps, with real numbers from Qwen2.5-0.5B, a small open model.
## 1. Tokens
A model never reads letters. It cuts the text into **tokens**, whole words or pieces of words, and each token is a number in a vocabulary of 151,665 entries. "Didn't" becomes two tokens: "didn" and "'t".
## 2. Vectors
Each token becomes a list of **896 numbers**, a vector. Words with similar meanings sit close together in that space. But the vector for "it" only says "a pronoun": nothing in it says which thing. To find out, "it" has to look at the other words.
## 3. Query, key and value
Every token makes three new vectors, by multiplying its own by three matrices learned in training. The **query** asks "what am I looking for?", the **key** says "this is what I contain", the **value** is what the token hands over if chosen. Think of a library: the question you bring, the labels on the books, and what is written inside.
## 4. Position
Query and key are rotated by an angle that depends on where the token sits. Without it, "the dog bites the man" and "the man bites the dog" would look the same.
## 5. Scores
"It" multiplies its query with the key of every earlier word: one number per word. "Animal" scores highest by far. Each score is divided by 8 (the square root of 64, the size of these vectors) to keep numbers stable, and the words after "it" are masked out: a model that writes can only look at what is already written.
## 6. Softmax
Softmax turns the scores into shares that add up to 100%, and exaggerates the differences: a score of 3.84 against 1.55 becomes **79% against 8%**. So "it" takes 79% of what it needs from "animal", 8% from "didn", 3% from "street".
## 7. Mixing the values
Each word hands over its value, scaled by its share, and the pieces are added up into a new vector for "it". That vector points almost exactly where "animal" points (a similarity of **0.98** out of 1) and almost nowhere near "street" (0.08). From here on, "it" means the animal.
## Many heads, many layers
That was one **head**. This layer has 14 working in parallel, each looking for something different: one finds what "it" refers to, one looks at the word just before, one rests on the first token as a way of saying "nothing for me here". Nobody programmed these roles; the model learned them. And there are 24 layers like this, stacked.
## The whole mechanism in code
```python
q, k, v = x @ Wq, x @ Wk, x @ Wv # query, key, value
q, k = rotate(q, pos), rotate(k, pos) # stamp in the word order
scores = q @ k.T / 8 # 8 = square root of 64
scores = scores.masked_fill(future, -inf) # no looking ahead
weights = softmax(scores, dim=-1) # shares that add up to 100%
out = weights @ v # mix the values
```
These few lines run for every head, in every layer, for every token, each time a model reads or writes a single word.
## Why we care
When we read a model, these are the places we look: which words a reply leans on, and where a behaviour lives inside the stack. Next in the series: [how the layers turn that into an answer](/news/tutorials/how-a-transformer-writes-inference).
More videos on our channel, [@SiliconPsyche](https://www.youtube.com/@SiliconPsyche). Deeper reading: [Transformer architecture](https://xbe.at/nlp-and-transformers/topic/transformer-architecture/) on xbe.at.
This update is not available.