How LLMs Are Actually Trained - Raw Text → “Aligned” Model
Pretraining to RLHF, stage by stage - and why alignment is a tilt in the weights, not a rule.
1
CORPUS trillions of tokens
web · code · books · papers
→
dedup + filter
2
TOKENIZER TRAINING BPE - learned from the corpus
the
→the
→the
frequent pairs merge → vocab of ~128K pieces, frozen forever
3
BATCHING self-supervision - no human labels
input
Theskylooksbluebecause
target
skylooksbluebecausesunlight
the target is the input shifted by one - every position is a training example: “guess the next token”
4
EMBEDDING + POSITIONAL ENCODING learned from random init
random noise
structured embeddings
position signal
learned / sinusoidal / RoPE
learned / sinusoidal / RoPE
5
TRANSFORMER STACK × N layers, causal mask
N layers
attention + FFN
attention + FFN
causal mask
✕ = future hidden
✕ = future hidden
masking the future lets all positions train in parallel in one forward pass - the reason training scales
6
LOSS one number rules everything
model P(token)
true next token
the entire pretraining objective: predict the next token. Truth, safety, helpfulness - not in the formula.
7
BACKPROP + ADAMW repeat ~10⁶ steps
⟵ gradients flow back
every weight nudged downhill
every weight nudged downhill
learning-rate schedule
grad clipping 1000s of GPUs
data ∥ tensor ∥ pipeline
8
SCALING LAWS → the base model
loss falls predictably with scale
a next-token predictor - not an assistant, continues anything
9
POST-TRAINING where “alignment” happens - same mechanism, new data
SFT
curated instruction → response pairs teach the assistant format
RLHF / DPO
preference pairs push weights toward answers people rate helpful + harmless
“alignment” = the same curve, reshaped
unwanted behavior gets rarer - never impossible
unwanted behavior gets rarer - never impossible