How the next word gets chosen

A language model does not decide what to say. It produces a probability for every token it knows, and then a few lines of arithmetic pick one. Those lines are the difference between text that repeats forever and text that wanders off into nonsense.

Third in a series, after what the model reads and where the vocabulary comes from. The model below is tiny and runs in your browser; the sampling arithmetic is the real thing. 12 August 2026.

What actually comes out

The model on this page is about as simple as a language model gets: a tally of which token followed which, taken from 120 tokens of the opening of A Tale of Two Cities. Ask it what comes after it was the and it does not answer with a word. It answers with all 7 words it has ever seen there, and how often:

This is the only thing any language model produces. GPT-4o does the same thing over its 200,000-token vocabulary, conditioned on thousands of tokens rather than two, but the output is the same shape: a number for every token, adding to one. Everything after this point is a choice about how to read that list.

The obvious approach, and why nobody uses it

Always take the most likely token. Here that means ␣age, at 20.0%. It is deterministic, it is defensible, and it does this:

Once the model reaches a state it has seen before, the most likely continuation is the same as last time, so it produces the same token, which returns it to the same state. A loop is the correct behaviour of a rule that never varies. Every repetition you have seen a chatbot fall into is a version of this, and it is why nobody ships greedy decoding for open-ended text.

Temperature

So introduce chance: sample from the distribution instead of taking its maximum. Temperature controls how faithfully you sample. Every probability is raised to the power 1/T and the results renormalised — that is the whole operation:

At T = 0.5 the leading token gets more of the mass. At T = 2 the gap between best and worst narrows and the tail becomes reachable. At T = 0 the operation has no meaning — you cannot raise to the power of infinity — so implementations special-case it to mean greedy, which is why temperature zero is not really a temperature.

Cutting off the tail

Temperature has an unpleasant property: it never makes anything impossible. Raise it far enough and every absurd continuation the model has ever seen becomes reachable, because they all keep a sliver of probability. So samplers usually cut the list down first.

Top-k keeps the k most likely tokens and throws the rest away:

Top-p, or nucleus sampling, does something subtler: it keeps the smallest group of tokens whose probabilities add up past a threshold. The size of that group changes with the model's confidence — narrow when it is sure, wide when it is not:

That adaptiveness is why top-p is usually preferred to top-k. A fixed k of 40 is far too generous when the model is certain of the next token and far too mean when it is genuinely torn.

Try it

Train the model on any text and turn the knobs. Everything runs in your browser; the seed makes each run repeatable.

Context is the other knob

Sampling is only half of it. The other half is how much the model conditions on. Here is the same corpus, the same temperature and the same seed, with the model allowed to look back two tokens, one token, and none:

Two tokens of memory produce something that reads almost like the original. One token produces text that is locally plausible and globally adrift — each pair of words is fine, the sentence is not. Zero context is a bag of words shaken out in frequency order.

This is the axis along which real language models moved. They are not running a cleverer sampler than the slider above; they are conditioning on thousands of tokens with a mechanism that can weigh which of them matter. The arithmetic that turns their answer into a word is the arithmetic on this page.

What is different in a real model

Three things, none of which is the sampler. The distribution comes from a neural network rather than a tally, so it can generalise to contexts it has never seen instead of backing off to a shorter one. It is computed over 200,000 tokens instead of 61. And it conditions on the whole conversation, not two tokens.

But when a model gets stuck repeating itself, or produces a confident sentence with a wrong word in the middle, or gives you a different answer to the same question twice, the mechanism is the one you just turned by hand. It is worth knowing that the last step between a model and its output is this small.

The sampler in your browser and the Python that generated every figure here are separate implementations. The build checks them against each other on seven corpora, three model orders and seven sampler settings, including the random number stream itself — a seeded generator is worth nothing if the two languages disagree about 32-bit arithmetic. Read them at ngram.js and ngram.py. The corpus is the opening of A Tale of Two Cities (1859, public domain).