← All lab updates
Context windows, explained: why long chats get forgetful, slow and expensive
Every time a model answers, it rereads the whole conversation from the start: your instructions, every turn, every pasted document. That space is the context window. It explains why long chats slow down, why instructions given at the start fade, and why facts in the middle of a long text get missed. How it works and what to do about it.
## What the context window is
A model has no memory between replies. To answer turn 20, it reads the system instructions, turns 1 to 19, any documents and turn 20, all at once. The maximum it can read is the **context window**, measured in tokens (roughly three quarters of a word each in English).
## Why long chats get slow and expensive
Every new word attends to every word before it ([how attention works](/news/tutorials/how-a-transformer-reads-attention)). Models keep the keys and values of earlier words in memory (the KV cache, see [how a transformer writes its answer](/news/tutorials/how-a-transformer-writes-inference)), so they do not recompute them, but that memory grows with every turn. A long conversation costs more memory, more time and, on paid APIs, more money per reply.
## Why things get lost
- **The middle is a blind spot.** Researchers found that models use information at the beginning and at the end of a long input much better than information in the middle (Liu et al., *Lost in the Middle*, 2023). A key clause on page 12 of 30 is easier to miss than one on page 1.
- **Instructions fade.** The system prompt is a few lines at the top; every turn after it adds text and pressure from the other side. This is one reason behaviour drifts over long conversations, the thing we read turn by turn ([caving across turns](/lab/reading-caving-across-turns)).
- **Truncation.** When a chat outgrows the window, something gets dropped, often the oldest turns, sometimes the instructions themselves.
## What to do
1. **Put what matters at the start or the end,** not in the middle.
2. **Restate key rules** in long sessions, or start a new conversation with a short summary.
3. **Retrieve instead of pasting everything:** give the model the relevant passages, not the whole archive ([RAG explained](/news/tutorials/rag-explained)).
4. **Test long conversations,** not just the first reply: behaviour at turn 15 is not behaviour at turn 1.
This update is not available.