How LLMs Actually Generate Text - The Full Inference Pipeline

Prompt → answer, stage by stage - including the positional encoding most diagrams skip. As you read, ask: where would a guardrail go?

Visual In depth How training works →

INPUT - the user prompt

Why is the sky blue?

TOKENIZER text → integers

Whyistheskyblue?

Token IDs: [8241] [318] [262] [6766] [4171] [30]

The model never sees letters or words - only these integers, from a fixed vocabulary of ~32K-256K subword pieces.

EMBEDDING + POSITIONAL ENCODING the step everyone skips

Each token ID selects a row of the embedding matrix → a dense vector (d_model ≈ 4096-16K dims).

Position must be injected - attention alone is order-blind

Without it, “dog bites man” = “man bites dog”. Classic models add a position vector to each embedding (learned or sinusoidal). Modern models (Llama, Qwen, Mistral) use RoPE: they rotate the Q/K vectors inside attention by an angle that encodes each token’s position.

TRANSFORMER BLOCK × N layers (32-120)

A) Multi-head self-attention. Every position builds Query, Key, Value vectors and looks at all previous tokens:

Attention(Q,K,V) = softmax(QKᵀ / √d_k) · V

KV cache (per layer)

Keys and Values of past tokens are stored so each new token only computes its own - no recomputation of the whole prompt at every step.

B) Feed-forward network. Linear → SwiGLU/ReLU → Linear (d_ff ≈ 4×d_model), position-wise. Each sub-block wrapped in residual Add & Norm (LayerNorm/RMSNorm).

LM HEAD linear + softmax

The final hidden state is projected onto the whole vocabulary → logits (~128K numbers), then softmax → a probability for every possible next token. This distribution is all the model produces. There is no “answer” object - only next-token probabilities.

SAMPLING picking one token

Greedypick argmax

Top-Ksample from top K

Top-Psmallest set ≥ P

TemperatureT→0 deterministic, T→1+ creative

SPECULATIVE DECODING speed trick

A small draft model proposes k tokens; the large model verifies them in one parallel forward pass, accepting the matching prefix. Same output distribution, fewer slow steps.

DETOKENIZER integers → text

[464] → The [6766] → sky [3073] → looks [4171] → blue [780] → because …

STREAMING OUTPUT then loop back to step 4

The sky looks blue because sunlight scatters ▌

Each generated token is appended to the context and the decode loop repeats - one token at a time, until a stop token.