Next-Token Prediction

It is not thinking about your question. It is computing what token most likely comes next — and every strength and every flaw grows out of that.

The last chapter finally turns to the machine itself. The previous five chapters were about using it: how to phrase the work, how to make it steady, where knowledge comes from, what it costs, which answers to trust. This one explains why all of that holds. It is not thinking about your question. It is deciding which character belongs in the next slot. That sounds like a put-down, but it is the only entry point that makes the thing add up — everything it can do and everything it gets wrong traces back to this one fact.

After reading, you should be able to answer:

• Why does the same question give you a different answer the second time?
• Why does telling it to think first actually help?
• Why does it keep writing after it has gone wrong, instead of backing up?

It only ever moves one slot forward

Give it a stretch of text and it enters a loop with no exit, repeating four moves each turn: read the whole current sequence; work out a probability distribution for the next token; draw one from that distribution; attach it to the end — and immediately start over. That chained hand-off is the core of the Transformer architecture, and every mainstream model today descends directly from it. transformer

Step three deserves a pause. It is not "take the highest-probability token" — it is sample. Ask the same question twice and the two versions diverge right here: the top candidate merely has the best odds, and the candidates behind it are not at zero, so whenever one of them wins, the sentence takes a different road. Sampling is also why both versions can sound equally sure of themselves. Nothing in the tone hints at the coin flip underneath.

Step one deserves a pause too. Every extra slot means the whole sequence gets read from the top again. There is no answer prepared in advance somewhere, and no draft. Watching characters surface one by one is not a finished answer being read aloud — that is the process itself.

The four things you assume it does, it does none of

Take off the "not human, but very human-seeming" filter and four everyday assumptions are left standing:

1. It understood your question — it only cut your text into tokens and computed a string of numbers.
2. It looked things up — there is no lookup step; everything it hands back comes from statistical patterns in the weights.
3. It worked out the answer before speaking — no such step; it decides as it writes.
4. It checked its work before replying — no such step, unless you make it run again.

The last two are the easiest to miss, because nothing in the output gives them away. Hence the judgment that matters: what comes back is the most plausible-sounding thing, not a verified thing. Nor will it flag which sentence deserves more suspicion — whether something holds up is not part of its computation. The practical consequence is narrow and blunt: numbers, dates, names, quoted passages — anything that has to be exact, verify before you rely on it.

Every failure traces back to the same mechanism

Turn the mechanism around and you are holding the full list of its weak spots. None of them qualify as defects; they are what this design necessarily produces. A newer generation can blunt them, never remove them.

Two costs that come from sampling. Two runs disagree because step three draws a sample rather than taking the top-scoring token, and the confident tone travelling alongside an uncertain result is just the same machine doing both at once. The other cost is fabrication: it is after the thing that most plausibly comes next, not the thing that happened, so wherever real material is missing at some position, the most plausible candidate gets drawn anyway — and reads exactly like the truth.

Two costs that come from a sequence that only grows. Whatever has already been produced is now part of the sequence, and it can only build on top of it. So when it spots its own earlier mistake it cannot back up; it smooths the story over, and the further it goes the further it drifts — last step's deviation becomes this step's starting point. The same fact says where your effort pays best: format, voice and limits all belong before it starts talking, because once the opening sets the direction it is very hard to bend back. It also shapes how you assign work: rather than waiting for it to wake up mid-task, cut the task small, let each piece be produced and checked on its own, and deviation accumulates more slowly.

One more comes from how digits are split. Numbers arrive broken into tokens, and place value is a weak statistical relationship, so counting and exact arithmetic are things it is bad at by construction. Where precision matters, do not let it do the sums in its head — hand that step to a tool outside it.

The capability is a by-product of guessing well

How does something that only ever continues text end up writing code, and reading the meaning hidden in half a question? The answer comes in a slight curve: getting the next character right, often enough, already demands that it pick up a whole pile of other skills along the way. Getting it wrong, after all, costs it something during training.

Getting it right has its own demands, and they come in layers.

The first is the structure of language itself: which kind of element may follow which, how clauses nest, which formats have to appear in matched pairs. The second is how an argument moves: the link from "if A holds, then B probably holds too" has a strong shape to it, and what the model picks up is the shape rather than logic proper — yet when the shape lines up, the conclusion tends to line up as well.

The third is register: the same point is worded differently for a customer than for a colleague, and that difference is just as countable from text.

Put those layers on real documents and it becomes concrete. What follows "this Agreement shall become effective upon" in a contract depends on the set phrasings for conditions and carve-outs and on where those normally sit in the document. Which figure and which unit follow "up year over year" in an earnings release depends on the pairing conventions between a number and its wording. What gets written after "chief complaint" in a clinical note depends on the order that kind of text always moves in. What it holds are these shapes and collocations, not contract, finance, or medical knowledge itself — but when the shape lines up, what it attaches looks convincing too.

Past a certain amount of scale, its behaviour spills outside the business of continuing text. Nobody trained it specifically to translate, summarize, or write code; those abilities jumped out in whole chunks as parameters and data grew — the paper's title says it outright: language models are few-shot learners. gpt3-paper

Writing out the intermediate steps is groundwork for itself

Why does "think it through first, then answer" genuinely work? The answer is still this mechanism: you make it write out a chain of intermediate steps, those steps stay in the sequence, and they become the basis for every step after them. It has not become smarter; it has simply laid better groundwork for its own continuation. Ask for more words, and you are buying more rounds of computation.

Look at the price at the same time. Every extra step has to be recomputed against a longer sequence, so both time and spend go up. "Think it through first" earns its keep on work with a long chain of reasoning, where one wrong step ruins everything; aimed at a request that one sentence would answer, piling on intermediate steps just buys nothing.

What this does not imply

"It is only predicting the next token" does not imply that it cannot work, and it does not imply that it only imitates. It genuinely produces combinations that never appeared in training, and it completes tasks nobody taught it — those emerged from scale, they were not designed in.

But it equally does not imply that it can make your judgments for you. This mechanism only guarantees that what it says is plausible, not that it is correct; that it can continue, not that it can hold. Which parts go to it and which stay with you is decided by something else entirely: whether you would notice a mistake and whether you could undo it. Where that line falls is the subject of It completes the implementation; it does not supply the judgment.

One more consequence is easy to miss: a thing assembled token by token can be interrupted at any point, and can be fed something new at any point. That is exactly why it suits streaming and suits being run inside a loop — and precisely because it is only a continuation machine, what happens once you put it in a loop needs its own engineering. That part lives in The Agent Loop.