How LLMs Are Actually Trained — Raw Text → “Aligned” Model
CORPUS trillions of tokens
Deduplicated, filtered, mixed by recipe. The model will become a compressed statistical portrait of this text — nothing more, nothing less.
TOKENIZER TRAINING BPE — learned from the corpus
A subword vocabulary (~32K–256K pieces) is learned from the corpus itself: frequent strings become single tokens, rare words get split. Fixed forever after — inference must use the exact same vocabulary.
BATCHING self-supervision — no human labels
Text is packed into sequences of 4K–8K+ tokens. Nobody labels anything:
The target is the input, shifted by one token
input: The sky looks blue because …
target: sky looks blue because sunlight …
the target is the input shifted by one — every position is a training example: “guess the next token”
EMBEDDING + POSITIONAL ENCODING learned from random init
The embedding matrix starts as random noise and is learned like every other weight. Since attention is order-blind, position is injected: added position vectors (learned/sinusoidal) or RoPE rotations of Q/K inside attention — trained jointly with everything else.
TRANSFORMER STACK × N layers, causal mask
Same architecture as inference (attention + FFN + residual norms), with one crucial addition:
Causal mask — no peeking at the future
Position t may only attend to positions ≤ t. That’s what lets ALL positions in a sequence be trained in parallel in one forward pass — the reason training scales at all.
LOSS one number rules everything
Logits → softmax → cross-entropy against the true next token, averaged over the batch:
L = −Σ log P(next token | context)
The entire objective of pretraining is this single scalar: how well did you predict the next token? Truth, safety, helpfulness — none of these appear in the loss.
BACKPROP + ADAMW AdamW, LR schedule
Gradients flow back through every layer; AdamW nudges billions of weights a tiny step downhill. Warmup + cosine decay, mixed precision (BF16), gradient clipping.
Repeat ~10⁶ steps across thousands of GPUs
Data/tensor/pipeline parallelism, checkpoints, months of wall-clock. Result: the base model — a formidable next-token predictor. Not an assistant. It will happily continue anything.
POST-TRAINING where “alignment” happens — same mechanism, new data
SFT — supervised fine-tuning
Curated instruction→response examples teach the assistant format and tone.
RLHF / DPO — preference tuning
Human (or AI) preference pairs reshape the distribution toward answers people rate as helpful and harmless.
Both stages use the same mechanism as pretraining: adjust weights so preferred continuations become more probable. No rule is written anywhere — a probability landscape is tilted.
SHIP IT weights frozen, behavior emergent
The finished model is a frozen set of weights. Everything it does from now on — every answer, every refusal, every mistake — is emergent behavior of that tilted distribution under your prompt, decided token by token at inference time.