How LLMs Actually Generate Text — The Full Inference Pipeline
Prompt → answer, stage by stage — including the positional encoding most diagrams skip. As you read, ask: where would a guardrail go?
1
INPUT
👤Why is the sky blue?
2
TOKENIZER text → integers
Whyistheskyblue?
[8241] [318] [262] [6766] [4171] [30]
3
EMBEDDING + POSITIONAL ENCODING the step everyone skips
token embeddings
position signal
sinusoidal / learned / RoPE
sinusoidal / learned / RoPE
(seq_len × d_model)
d_model = 4096
d_model = 4096
Attention is order-blind — without position, “dog bites man” = “man bites dog”. Modern models (Llama, Qwen) rotate Q/K by position (RoPE) instead of adding vectors.
4
TRANSFORMER BLOCK × N layers (32–120)
A) Multi-head self-attention
Q
K
V
attention scores
causal: no future
causal: no future
softmax(QKᵀ/√d_k)·V → Add & LayerNorm
KV cache (per layer)
time →
K
V
past K,V stored — only the new token is computed
B) FFN
Linear
d_model × d_ff→ →Linear
d_ff × d_model →Add & LayerNorm ×96 layers in GPT-4-class models
d_model × d_ff→ →Linear
d_ff × d_model →Add & LayerNorm ×96 layers in GPT-4-class models
5
LM HEAD linear + softmax
Linear
d_model × 128K→
→softmax→
d_model × 128K→
1logits128K
1128K
P(next token)
The model’s entire output = one probability per vocabulary token. Nothing else.
6
SAMPLING picking one token
Greedy
pick argmax
Top-K
sample from top K
Top-P
smallest set ≥ P
Temperature
flat = creative
7
SPECULATIVE DECODING speed trick
draft model
small, fast → Theskylooksgreen → target model
verifies in 1 pass → ✓✓✓✗
small, fast → Theskylooksgreen → target model
verifies in 1 pass → ✓✓✓✗
accepted prefix kept, wrong tail regenerated — same distribution, fewer slow steps
8
DETOKENIZER integers → text
[464]→The
[6766]→sky
[3073]→looks
[4171]→blue
[780]→because
9
STREAMING OUTPUT then loop back to step 4
The sky looks blue because sunlight scatters ▌
one token at a time, appended to the context, until a stop token