How LLMs Are Actually Trained - Raw Text → “Aligned” Model
CORPUS trillions of tokens
Deduplicated, filtered, mixed by recipe. The model will become a compressed statistical portrait of this text - nothing more, nothing less.
TOKENIZER TRAINING BPE - learned from the corpus
A subword vocabulary (~32K-256K pieces) is learned from the corpus itself: frequent strings become single tokens, rare words get split. Fixed forever after - inference must use the exact same vocabulary.
BATCHING self-supervision - no human labels
Text is packed into sequences of 4K-8K+ tokens. Nobody labels anything:
The target is the input, shifted by one token
input: The sky looks blue because …
target: sky looks blue because sunlight …
the target is the input shifted by one - every position is a training example: “guess the next token”
EMBEDDING + POSITIONAL ENCODING learned from random init
The embedding matrix starts as random noise and is learned like every other weight. Since attention is order-blind, position is injected: added position vectors (learned/sinusoidal) or RoPE rotations of Q/K inside attention - trained jointly with everything else.
TRANSFORMER STACK × N layers, causal mask
Same architecture as inference (attention + FFN + residual norms), with one crucial addition:
Causal mask - no peeking at the future
Position t may only attend to positions ≤ t. That’s what lets ALL positions in a sequence be trained in parallel in one forward pass - the reason training scales at all.
LOSS one number rules everything
Logits → softmax → cross-entropy against the true next token, averaged over the batch:
L = −Σ log P(next token | context)
The entire objective of pretraining is this single scalar: how well did you predict the next token? Truth, safety, helpfulness - none of these appear in the loss.
BACKPROP + ADAMW AdamW, LR schedule
Gradients flow back through every layer; AdamW nudges billions of weights a tiny step downhill. Warmup + cosine decay, mixed precision (BF16), gradient clipping.
Repeat ~10⁶ steps across thousands of GPUs
Data/tensor/pipeline parallelism, checkpoints, months of wall-clock. Result: the base model - a formidable next-token predictor. Not an assistant. It will happily continue anything.
POST-TRAINING where “alignment” happens - same mechanism, new data
SFT - supervised fine-tuning
Curated instruction→response examples teach the assistant format and tone.
RLHF / DPO - preference tuning
Human (or AI) preference pairs reshape the distribution toward answers people rate as helpful and harmless.
Both stages use the same mechanism as pretraining: adjust weights so preferred continuations become more probable. No rule is written anywhere - a probability landscape is tilted.
SHIP IT weights frozen, behavior emergent
The finished model is a frozen set of weights. Everything it does from now on - every answer, every refusal, every mistake - is emergent behavior of that tilted distribution under your prompt, decided token by token at inference time.