How a language model is built

No API is needed here. The small Python example was run with Python 3.9 and its output is real. The claims about training follow the original papers, linked inline.

The transformer and attention

Almost every modern LLM is a transformer, an architecture introduced in the 2017 paper Attention Is All You Need. Its central idea is attention: when the model processes a token, it looks at every earlier token and decides how much each one matters.

Take the sentence “The cat sat because it”. To continue it, the model has to know what “it” refers to. Attention gives every earlier word a score, turns the scores into weights that add up to 1, and mixes in information mostly from the high-weight words. A toy version of that last step:

import math

# How much should "it" attend to each earlier word? (toy scores)
words = ["The", "cat", "sat", "because", "it"]
scores = [0.1, 2.5, 0.3, 0.2, 0.5]

exps = [math.exp(s) for s in scores]
weights = [e / sum(exps) for e in exps]
for word, w in zip(words, weights):
    print(f"{word:8} {w:.2f}")

The      0.06
cat      0.70
sat      0.08
because  0.07
it       0.09

In a real model the scores are not written by hand. They come from learned weights, and the model runs many attention “heads” in parallel across dozens of layers. The principle is the same: every token can draw on every earlier token, which is why the model can keep track of long text.

Stage 1: pretraining

The model starts with random weights. In pretraining it reads a huge amount of text, and at every position it predicts the next token, compares the prediction with the real token, and adjusts its weights a little. This repeats trillions of times.

The result is a base model. It knows a lot about language and the world, but it only continues text. Ask it a question and it may answer, or it may continue with three more questions, because that also looks like plausible text.

Most of what a model “knows” comes from this stage. It is also where the knowledge cutoff comes from: the model knows nothing that happened after its training data was collected.

Stage 2: supervised fine-tuning

To turn the base model into an assistant, it is trained further on example conversations: a question, and a good answer written or checked by people. This is supervised fine-tuning (SFT). The model learns the format of a conversation and the habit of answering what was asked.

The amount of data is tiny compared with pretraining. Fine-tuning changes how the model behaves much more than what it knows, which is why it works for style and format and not for adding knowledge.

Stage 3: preference training

An answer can be correct and still unhelpful, too long, or unsafe. The last stage teaches the model which of two answers is better:

• RLHF (reinforcement learning from human feedback): people rank several answers, a reward model learns their preferences, and the LLM is trained to get a high reward. Described in InstructGPT.
• DPO (direct preference optimization): trains directly on pairs of “preferred” and “rejected” answers, without a separate reward model. Described in the DPO paper.
• Constitutional AI: Anthropic’s approach, where a written set of principles guides AI feedback on the answers. Described in Constitutional AI.

What this explains

Two behaviors follow from these stages:

• For information newer than the cutoff, or private, you have to give the model the text, which is what RAG does.
• It was trained to produce answers people prefer, so it can sound confident even when it is wrong.