An interactive walkthrough

How AI predicts the next word

Five short modules. No maths required; bring your scepticism.

← Back to catalogue

Module 01

Words become numbers

Myth we'll unpick: "There's a dictionary inside"

Two Darwins

Charles Darwin proposed that species evolve by natural selection.

Charles Darwin University opens semester two enrolments.

Same two words. How does a machine, which has never met either Darwin, tell them apart?

Computers only speak numbers

Darwin

→

token
#20482

→

[0.42, -0.17, 0.88,
0.03, -0.56, 0.71]

Real models use thousands of numbers per word; we're showing six. These values are illustrative, not real model outputs.

Where do those numbers come from?

Nobody typed them in. They are weights: adjustable dials inside the model, and they start out random. Training is trial and error at colossal scale.

The model practises on a sentence

The finch pecked at the photocopier

How wrong was that guess? (the error)

The dials for "finch" (our six numbers)

[0.91, 0.05, -0.62,
-0.44, 0.13, -0.09]

training steps: 0 (dials random)

Guess; measure the miss; nudge every dial a tiny bit in the direction that reduces the miss; repeat, billions of times across the whole training corpus. You already know this idea from Excel: a trendline is two numbers fitted to data by minimising error. This is the same idea with billions of numbers. The probabilities you'll meet in Module 3 are calculated from these same trained dials. Values illustrative.

The map of meaning

Add context and watch the same word move:

The word never changes. Its position does; and position is meaning.

Meaning is position. Nearby words behave alike. The model stores no definitions and no dictionary; it stores geometry, learned from patterns in text.

One honest admission

We can read every number inside the model. We still cannot fully explain why those numbers work. Working that out is a live research field called interpretability; we'll come back to it in Module 5.

Module 02

The scale of it

Myth we'll bust: "It's a database that looks things up"

Start with one book

0.5–1 MB

One paperback, as plain text

Text is remarkably compact. Every word in a full novel weighs less than one photo on your phone. Hold onto that unit; we're about to stack it.

Book and library sizes in this module are illustrative estimates for teaching; the web-scale and GPT-3 figures are cited on screen.

Climbing the ladder

Your phone

the whole thing, in your pocket

128 GB

A suburban library

about 50,000 books; as text, it fits on your phone twice over

≈ 50 GB of text

National Library of Australia

every print book it holds, as plain text (illustrative)

a few TB

Common Crawl

the open web-crawl corpus, collected since 2008; one monthly snapshot alone is roughly 2–2.5 billion pages, 350–400 TiB

9.5+ PB

The bars are on a logarithmic scale. Drawn honestly, with your phone at one centimetre, the Common Crawl bar would run about 740 metres.

Common Crawl figures: Baack, S. (2024) "Training Data for the Price of a Sandwich", Mozilla Foundation, mozillafoundation.org/en/research/library/generative-ai-training-data/common-crawl/ ; Ilyankou et al. (2024) arXiv:2405.11039.

What goes in

The ingredients; click one for detail

The filtered web crawl from the last screen: news, forums, blogs, documentation; the everyday written internet.

Large collections of digitised books, fiction and non-fiction. For GPT-3, OpenAI never disclosed exactly which books were included.

Research papers and preprints; the source of the formal, cited register it can imitate.

Public code repositories; this is where its programming languages come from.

Wikipedia and other encyclopaedias; tiny next to the web crawl but high quality, so trainers weight them up and the model sees them more often.

The quality filter; worked example, GPT-3

Raw web crawl: ~45 TB of compressed text

↓

Filtering: deduplication, quality screens, cleanup

↓

Kept: 570 GB ≈ 500 billion tokens; about 1% of the raw crawl

Brown et al. (2020) "Language Models are Few-Shot Learners", arXiv:2005.14165; Baack, S. (2024), Mozilla Foundation, as previous screen.

How much reading is that?

500 billion tokens ≈ 375 billion words

÷ 250 words a minute , 24 hours a day, no breaks

≈ 2,800 years of non-stop reading

Calculated from the GPT-3 token count; illustrative. To finish this morning, you would have needed to start around 800 BC, before Rome was founded.

Where did the pages go?

That 570 GB of training text is not inside the model. The model kept the patterns, not the pages; like you after a thousand books: educated, not a photocopier.

"So is it training on me right now?"

Training

Already happened

Done in the past, once, at vast expense
Read the enormous corpus you just climbed
Produced the frozen model: billions of fixed weights
Those weights do not change when you chat

Inference

Your conversation, now

Your prompt goes in; the model predicts the next words
Every reply is generated fresh, on the spot
Afterwards, the model is exactly as it was
Nothing you typed is woven into its weights

Whether a provider separately stores or reuses your prompts is a product and policy question; it is set by terms of service and settings, not by anything the model does on its own.

There is no database inside and nothing to "look up". Everything it says is generated fresh from patterns learned during training.

Module 03

Rolling weighted dice

Myth we'll bust: "It just copies and pastes"

Probabilities in this module are illustrative, hand-crafted for teaching; they are not real model outputs.

Build a sentence, one draw at a time

Click a word to place it. We skip the small connecting words to save time; in the real model every word, even "the", gets its own weighted draw.

The temperature dial

One setting controls how the dice are rolled: play it safe, or give unlikely words a chance.

Prompt: "The crocodile lurked in the …"

Same prompt, three outputs

Prompt: "The crocodile lurked in the …", medium temperature, three separate runs

It cannot copy and paste an answer it does not store; each sentence is a fresh weighted draw over the whole vocabulary.

Where do the percentages come from?

Module 1

Position on the map of meaning: "crocodile" lives near water, mangroves and the wet season.

→

Module 2

Patterns from the training text: how sentences like this one tended to continue, across billions of examples.

→

This module

The bars you just clicked: geometry plus patterns, run through the trained dials, set every percentage.

No lookup happens anywhere in this chain; there is no table of facts to consult.

Generation is weighted dice rolled over learned patterns. Uniqueness is built in; the same prompt produces different phrasings by design, which is why it cannot simply copy and paste.

Module 04

The confident machine

Myth we'll bust: "It knows when it's wrong"

Probabilities in this module are illustrative, hand-crafted for teaching; they are not real model outputs.

Same machine, harder question

Prompt: "Give me a journal citation on buffalo management in the Northern Territory."

This paper does not exist

Every fragment above is plausible on its own: the name, the journal, the volume, the DOI. The whole is fiction. The dice never checked a shelf, because there is no shelf.

Why did that happen?

Module 3

A right answer

Position on the map of meaning
Patterns from the training text
Weighted draw, one word at a time
Fluent output

This module

A fake citation

Position on the map of meaning
Patterns from the training text
Weighted draw, one word at a time
Fluent output

Read the two lists again; they are identical. The process that produced a correct answer and the process that produced the fake are the same process. There is no separate fact-checking step, and no slot anywhere in the machine that holds "true".

Why not just say "I don't know"?

The distribution always offers a next word. Watch how the honest answer competes:

"Certainly. One relevant paper is…"41%

"A useful reference on this topic…"33%

"Research in this area includes…"21%

"I cannot find a verified source."5%

Declining to answer is itself just a pattern, and it must be trained in; it competes with fluency and usually loses. Helpfulness training amplifies confident continuation; it is the amplifier, not the root cause. Values illustrative.

What helps

Ask for sources; then check them

A citation is a claim, not evidence. Search for the paper; if it does not exist, you have learned something important about the answer.

Prefer grounded modes

Tools that search or retrieve real documents before answering give the dice something solid to land on.

Fluency is not accuracy

Confidence and polish are properties of the writing, not the facts. The fake citation read beautifully.

Right answers and wrong answers come off the same production line. Checking is the human's job and the tooling's job; it is not the model's instinct.

Module 05

The black box

We built it; we can measure it; we cannot fully explain it.

What we can and cannot see

Measurable

Everything

Every weight; all billions of dials are readable
Every probability on every draw
Every input and every output

Still hard

The why

Why these particular numbers produce that behaviour
Which internal patterns carry which ideas
What the model will do on inputs nobody has tried

Working on the right-hand column is called interpretability research: reverse-engineering the features and circuits inside trained models. It is real, it is progressing, and it is early.

Why this matters

A tool this capable, and this incompletely understood, keeps human judgement in the loop by necessity, not by courtesy. You have seen the machine: no dictionary, no database, no fact-checker, no self-doubt. What it does have is patterns, at a scale no person can match. Use it accordingly.