What the model actually reads

A language model never sees letters. Your text is first chopped into pieces drawn from a fixed vocabulary of about 200,000 — and almost everything strange these models do with spelling, arithmetic and non-English text begins right there.

Every figure below is generated from a real tokenizer, in your browser and at build time. 12 August 2026.

The strawberry problem

Ask a model how many times the letter r appears in strawberry and it may confidently tell you two. This gets passed around as a famous stupidity. It is closer to a reading problem.

Here is the word as the model receives it:

Three pieces. The model is handed the numbers 302, 1618, 19772 — and the letters are gone before it begins. There is no r anywhere in that input to count. Asking how many the word contains is like asking someone to count brushstrokes in a painting they only ever saw described by catalogue number.

Models often answer correctly anyway, because text about spelling appears in their training data — they have read that strawberry is spelled s-t-r-a-w-b-e-r-r-y. But that is recall, not perception. It is why the failure is so erratic: it holds for common words and collapses on rare ones.

Try it yourself

Type anything. This runs entirely in your browser — the text never leaves your machine, and there is no server to send it to.

The space before the word

Whitespace is not separate from the word. It is welded on. The same ten letters are one token or three depending on what sits in front of them:

strawberry — with the leading space — is a single token, because that is how the word almost always appears in running text. Strip the space and you get an unusual fragment the tokenizer has to build from three pieces.

Numbers do not have digits

Nothing forces a tokenizer to split numbers at sensible places, and this one does not:

1234567890 arrives as four chunks, not ten digits. Adding two numbers column by column is difficult when the columns are not there — the model must first reconstruct place value from pieces that cut across it. Add a comma and the split changes completely. This is a large part of why arithmetic is unreliable in a system that can otherwise write a proof.

Code is mostly whitespace

Watch what happens to that four-space indent. The line break fuses to the closing ): and becomes one token. Three of the four indent spaces form a second token. The fourth space is welded onto return. A single level of Python indentation is not one thing to the model — it is a boundary spread across three tokens, none of which line up with it.

Reindenting a file therefore changes its token count without changing a line of logic, and a model editing code has to reconstruct block structure from pieces that cut across it.

The language tax

Here is Article 1 of the Universal Declaration of Human Rights — the same sentence, the same meaning, in eleven languages, in the UN's own translations:

English costs 33 tokens. Japanese costs 71 — 2.15× as many for the same sentence. Because context windows are measured in tokens and API pricing is per token, a Japanese speaker fits less of their document into the same window and pays more to say the same thing. The tax is invisible, and every language in that table pays it.

Note the column that misleads. Chinese and Japanese have the two lowest characters-per-token ratios in the table — 1.23 and 1.18, barely one character per token — yet they land in completely different places: Chinese at 1.06× English, Japanese at 2.15×. The difference is compression in the writing system. Chinese says the whole sentence in 43 characters where English needs 170; Japanese needs 84 and gets no such discount. A bad ratio only hurts if you also need a lot of characters. What you are billed for is tokens, and neither characters nor words predict them reliably.

What changed between model generations

That tax used to be far worse. GPT-4 and GPT-3.5 used a vocabulary called cl100k_base; GPT-4o moved to o200k_base, twice the size, with far better coverage of non-Latin scripts:

Hindi went from 180 tokens to 54 — a 70% cut — while English prose did not move at all. Doubling the vocabulary bought almost nothing for English and an enormous amount for everyone else. Which tells you what the first vocabulary had been optimised for.

Why this is worth knowing

Tokenization is not a detail of the implementation that users can ignore. It sets what the model can perceive. A model cannot reliably count letters it was never shown, cannot align digits it received in clumps, and cannot charge a Japanese sentence the same as its English twin.

None of this is mysterious, and none of it requires trusting a claim about how these systems behave. It is a text-processing step you can run yourself — which is what the box above is for. Paste in something you have wondered about.

Next: Where a vocabulary comes from — these pieces were not designed by anyone. Watch the algorithm that invented them run, four merges at a time.

Token counts come from gpt-tokenizer (MIT), computed at build time and re-checked against the copy your browser runs. The translations are the UN's official texts of UDHR Article 1. If you spot an error, the whole point is that you can verify it — every figure here is reproducible by pasting the same text into the box above.