How Does It Work? (Part 3): Knowledge from Statistics
How Does an LLM Work?
How the model 'knows' that Paris is the capital of France — when nobody told it directly. Training, RLHF, and emergence. Full resolution of the anchor example.
1
Learning Material
5 pagesLesson 6 — How Does It Work? (Part 3): Knowledge from Statistics
Understanding the Complex: How Does an LLM Work?
We've covered the mechanism. You know how text becomes tokens, tokens become vectors, and vectors travel through attention layers that weight context. Now the hard part.
How does the model know that Paris is the capital of France?
Nobody told it. It wasn't given a geography textbook. It was trained on one task and one task only: predict the next token. And yet, if you ask it the capital of every country on Earth, it'll get almost all of them right.
Where does that knowledge come from?