Where a vocabulary comes from

The pieces a model reads are not designed by anyone. They are counted into existence by an algorithm short enough to state in four lines — and you can watch it invent the word berry from nothing but tallies.

A sequel to what the model actually reads. Every figure is generated; the trainer below runs in your browser. 12 August 2026.

The problem

You need a fixed list of pieces that can spell any text at all. Two obvious answers both fail. Use single characters and everything is representable, but a paragraph costs hundreds of tokens and the model spends its attention assembling words instead of thinking. Use whole words and text gets short, but the list is never finished — new words, names, typos and other languages all fall off the end.

Byte pair encoding takes the middle. Start with single characters, then let the text itself decide which combinations deserve promotion to a single piece. Common things become short. Rare things stay spelled out. Nothing is ever unrepresentable.

The algorithm

1. Split the text into words, each still a string of characters.
2. Count every adjacent pair of symbols in the whole corpus.
3. Merge the most frequent pair everywhere, and record it as a new token.
4. Repeat until you have as many tokens as you wanted.

That is the entire method. There is no linguistics in it and no notion of what a word is. It is counting, repeated.

Watch it run

Here is a deliberately tiny corpus — 295 characters, 50 words, 29 of them distinct — small enough that every merge is explicable:

the strawberry and the raspberry and the blueberry and the blackberry are all berries. a strawberry is red. a raspberry is red. a blueberry is blue. the blackberry is black. berries grow on the berry bushes in the berry garden. she picked a strawberry, then another strawberry, then a raspberry.

The whole training set.

Starting from 23 distinct characters, the first 30 merges go like this:

#	Pair	Becomes	Seen	Vocab
1	`e` + `r`	`er`	16	24
2	`b` + `er`	`ber`	15	25
3	`ber` + `r`	`berr`	15	26
4	`berr` + `y`	`berry`	13	27
5	`h` + `e`	`he`	11	28
6	`␣` + `a`	`␣a`	10	29
7	`t` + `he`	`the`	9	30
8	`r` + `a`	`ra`	7	31
9	`␣` + `the`	`␣the`	7	32
10	`␣` + `b`	`␣b`	7	33
11	`␣b` + `l`	`␣bl`	6	34
12	`␣` + `i`	`␣i`	5	35
13	`␣` + `s`	`␣s`	4	36
14	`␣s` + `t`	`␣st`	4	37
15	`␣st` + `ra`	`␣stra`	4	38
16	`␣stra` + `w`	`␣straw`	4	39
17	`␣straw` + `berry`	`␣strawberry`	4	40
18	`␣a` + `n`	`␣an`	4	41
19	`c` + `k`	`ck`	4	42
20	`␣i` + `s`	`␣is`	4	43
21	`␣an` + `d`	`␣and`	3	44
22	`␣` + `ra`	`␣ra`	3	45
23	`␣ra` + `s`	`␣ras`	3	46
24	`␣ras` + `p`	`␣rasp`	3	47
25	`␣rasp` + `berry`	`␣raspberry`	3	48
26	`␣bl` + `u`	`␣blu`	3	49
27	`␣blu` + `e`	`␣blue`	3	50
28	`␣bl` + `a`	`␣bla`	3	51
29	`␣bla` + `ck`	`␣black`	3	52
30	`r` + `e`	`re`	3	53

␣ marks a space. Highlighted rows are the three worth stopping on.

Three things just happened

By merge 4, the token berry exists. Nothing told the algorithm that berry is a morpheme, or that English has suffixes. The letters e and r kept turning up together, then b in front of them, then y behind. Four tallies and a word-piece falls out.

At merge 6, a space welds itself onto a word. This is the mechanism behind the strangest fact on the previous page: leading spaces belong to the words that follow them. No rule imposes it. Words are overwhelmingly preceded by a space in real text, so  + a letter is always among the most frequent pairs going.

At merge 17, strawberry becomes a single token — assembled out of the berry learned at merge 4. Watch it come together:

After 0 merges — 11 pieces

·strawberry

After 1 merge — 10 pieces

·strawberry

After 2 merges — 9 pieces

·strawberry

After 3 merges — 8 pieces

·strawberry

After 4 merges — 7 pieces

·strawberry

After 16 merges — 2 pieces

·strawberry

After 17 merges — 1 piece

·strawberry

The same eleven characters, re-read after each merge.

Three fates

Every word ends up in one of three states, and which one depends entirely on how often it appeared:

Word	After 30 merges here	Under o200k (200,000 merges)
`␣strawberry`	`␣strawberry` (1)	`␣strawberry` (1)
`strawberry`	`s · t · ra · w · berry` (5)	`st · raw · berry` (3)
`␣blackberry`	`␣black · berry` (2)	`␣blackberry` (1)
`strawberries`	`s · t · ra · w · berr · i · e · s` (8)	`st · raw · berries` (3)
`␣kiwi`	`␣ · k · i · w · i` (5)	`␣kiwi` (1)

Left: this page's 30-merge vocabulary. Right: o200k_base, the real thing, from the same words.

strawberry is one token in both — a toy trained on four sentences and a production vocabulary trained on the internet agree, because they are running the same algorithm against the same statistical fact. Strip the space and both fragment. kiwi never appeared in these four sentences, so the toy shatters it into characters; o200k has seen plenty of kiwis and spends one token. That gap is the whole difference between this page and a real tokenizer: not the method, just how much text it counted.

Train one yourself

Paste anything — your own writing, code, another language. It runs in your browser and nothing is sent anywhere.

Training text

Merges

A link carries the training text and merge count in the URL after the #, which browsers never send to a server — so you can show someone exactly what you trained without it reaching me.

–vocabulary

–merges learned

–starting characters

Test a word against what it learned

The merges it learned

What changes at scale

A production tokenizer differs from the one above in three ways, none of them the algorithm. It starts from the 256 possible bytes rather than from characters, so that any input in any script is representable even if it never appeared in training. It uses a more careful rule for splitting text before counting, so that numbers and punctuation behave. And it runs for 200,000 merges over an amount of text no one reads.

Everything else is what you just watched. The vocabulary that decides whether your language costs twice as much as English is the output of counting pairs on a corpus, and the corpus is the argument.

Next: How the next word gets chosen — once a model has a probability for every token, a few lines of arithmetic decide which one you actually see.

The trainer in your browser and the Python that generated every figure above are two separate implementations. The build compares them on six corpora — including emoji, combining marks and text with no repetition at all — and fails if they disagree on a single merge. Read them at bpe.js and bpe.py.