Learning instead of looking up
Everything else on this site describes the outside of a language model — what goes in, where the vocabulary came from, how the output is picked, what it costs. This is the part in the middle, at the smallest size that still shows the one thing that matters: a model that has never seen your sentence can still answer it.
Fifth in the series. The network below trains in your browser, in about a second, with every derivative written out by hand. 12 August 2026.
Why a table was never going to work
The model on the sampling page is a tally: it looks up what followed this context before. Give it a context it has not seen and it has nothing, so it backs off to a shorter one and eventually to noise.
That failure is not rare, it is the normal case. This page trains on the same 295-character corpus as the vocabulary piece, and asks what follows each run of 3 characters. The text contains 124 distinct contexts. The number of contexts that could be asked about is 10,648.
1.2% of possible contexts appear in the training text
A lookup table cannot answer the other 98.8%. Not because it is small — because looking up is the wrong operation.
What replaces it
Instead of storing contexts, store a short vector for each character and learn a function of those vectors. Every character gets 8 numbers; the 3 characters of context are looked up and laid end to end; that runs through one hidden layer of 64 units and out to a probability for each of the 23 characters.
3 characters of context | look up a vector for each C (23 x 8) v 24 numbers | multiply, add a bias, squash W1 (24 x 64), b1 v 64 hidden units | multiply, add a bias W2 (64 x 23), b2 v 23 scores -> softmax -> probabilities
Nothing here is a lookup of a context. The context only ever appears as vectors being multiplied, which is exactly why an unseen combination is not a special case.
Watching it learn
All 3,279 numbers start random, so the model starts by predicting noise. Each step: run a batch forward, measure how surprised it was by the real next character, work out which direction every parameter should move to be less surprised, and take a small step that way.
loss 3.230.22 after 2,000 steps
After 0 steps — loss 3.37
ppppkptcdnygycle rb nepwpuclwdrhkoseidnli .idthip wpphptyl,y
After 50 steps — loss 1.00
blackberry is berry berry is berrie blackberry is redeberrie
After 200 steps — loss 0.38
berry and the berry and ther strawberry berry and the berrie
After 600 steps — loss 0.32
berry and the berry is blueberry garden. shes in the strawbe
After 2,000 steps — loss 0.27
blue. the blackberry and the berries grow on the berry and t
By 50 steps it has words. It has not been told that words exist, that spaces separate them, or that berry is a unit — only which character tended to follow which.
How I know the gradients are right
Every other page here is checked by running two independent implementations
and demanding identical output. That is not available for this one, and saying
so matters: training is thousands of floating-point operations deep, and
tanh, exp and log differ in their last
bits between engines. Two implementations that both merely train prove
very little.
So the check is different. Every derivative on this page is written out by hand, which is exactly the kind of code that is silently, plausibly wrong. For any parameter, its gradient claims to predict how the loss changes when you nudge it. That is testable: nudge it up, nudge it down, see what the loss actually did, and compare.
2.8e-08 worst relative error between the analytic gradient and finite differences
What it learned: vectors, not entries
The interesting parameters are the per-character vectors, because nothing told the model what to put in them. Characters that behave alike drift together, since the same nudges apply to both:
| Character | Nearest by cosine similarity |
|---|---|
a | r 0.34, g 0.33, u 0.23, . 0.19 |
e | w 0.60, i 0.52, h 0.47, c 0.41 |
b | u 0.51, d 0.49, , 0.39,
0.26 |
s | o 0.54, d 0.42, , 0.34, w 0.26 |
| k 0.29, y 0.25, g 0.21, t 0.20 |
The part that could not have worked before
Here is the whole argument in one table. Two contexts the training text contains, and two it does not:
| Context | In the training text? | Lookup table says | Network says |
|---|---|---|---|
err | yes | y×13, i×2 | y 0.91, i 0.09, e 0.00 |
raw | yes | b×4 | b 1.00, r 0.00, p 0.00 |
yst | never occurs | nothing at all | r 0.98, b 0.01, s 0.00 |
wby | never occurs | nothing at all | a 0.96,
0.01, 0.01 |
For yst the table has nothing and never will.
The network answers r
with 98% confidence, and it is right, because it
learned from elsewhere in the text what tends to follow those characters. It
generalises from the parts to a whole it never saw.
That is the property. Everything since — bigger models, attention, transformers — is a better answer to the same question: how do you turn a context into a prediction without having stored that context?
It always has an answer
The same table shows the cost. For wby, also
absent from the text, the network replies
a at
96% — just as confidently, with nothing to back
it up.
A lookup table can say it has nothing. This cannot. There is no state in it that means I have not seen anything like this: the arithmetic runs to completion on any input and always produces a distribution that sums to one. Confidence here is a number the model computes, not a measure of whether it should be trusted — and that is the same machinery underneath a large model stating something false in a fluent sentence.
Train one yourself
Paste any text. It trains in your browser — a second or two — and nothing is sent anywhere. Then ask it about a context your text does not contain.
What it writes
Ask it about a context
Type 3 characters. Try something your text does not contain.
What this is not
This is not a transformer and it would be a poor one. It sees a fixed 3 characters and cannot look further back, so it has no way to connect a pronoun to a name a paragraph earlier. Every position is treated identically; there is no mechanism for deciding that one earlier character matters more than another. That mechanism is attention, and it is the thing this model most conspicuously lacks.
What it does have is the part that made the rest possible: parameters learned by gradient descent, and representations that generalise instead of entries that are looked up. A modern model is this, scaled by eight orders of magnitude, with attention in the middle and a great deal of engineering around it.
Both implementations are readable: mlp.py and mlp.js. Neither uses an autodiff or matrix library — the derivatives are written out because they are the point. The build checks each one's gradients against finite differences and checks that the two agree on initialisation and the forward pass; see checkmlp.py.