← All lab updates
Embeddings, explained: how meaning becomes a list of numbers
An embedding turns a piece of text into a list of numbers, placed so that texts with similar meaning sit close together. It is how a search finds "broken item" when you typed "damaged goods", how RAG picks passages, and the first thing a language model does with every word. How it works, five lines of code, and its limits.
## The idea
Computers compare numbers, not meanings. An **embedding** is a way to turn text into a list of numbers (a vector) so that **closeness in numbers means closeness in meaning**. "Refund policy for damaged goods" and "can I return a broken item?" share almost no words, yet their vectors end up near each other.
## Where you meet them
- **Inside every language model.** The first step turns each token into a vector, 896 numbers in the small model of our series ([how attention works](/news/tutorials/how-a-transformer-reads-attention)).
- **Search and RAG.** Documents and questions are embedded, and the closest documents are fetched ([RAG explained](/news/tutorials/rag-explained)).
- **Recommendations and clustering:** similar products, similar tickets, duplicates.
## How closeness is measured
The usual measure is the **cosine similarity**: how much two vectors point in the same direction, from -1 to 1. Close to 1 means same meaning.
```python
a = embed("refund policy for damaged goods")
b = embed("can I return a broken item?")
c = embed("quarterly revenue grew 4%")
cosine(a, b) # high: different words, same meaning
cosine(a, c) # low: unrelated
```
## How they are learned
An embedding model is trained on pairs of texts that belong together (a question and its answer, a title and its article) and pairs that do not, and learns to pull the first close and push the second apart. Nobody decides what each of the numbers means.
## The limits
- **Similar is not true.** Embeddings capture topic and phrasing, not correctness: "the drug is safe" and "the drug is not safe" can sit very close.
- **Negation and numbers** are weak spots: "above 5%" and "below 5%" look alike.
- **Domain matters.** A general model may place two legal terms that mean opposite things side by side.
- **Bias travels.** Whatever associations were in the training text end up in the distances.
## Go further
- [Vector databases and embeddings](https://xbe.at/llms-and-genai/topic/vector-databases-and-embeddings/), practical articles on xbe.at
- [Context windows](/news/tutorials/context-windows-long-chats): what happens when there is too much to fit
This update is not available.