Looking at the right thing

The model on the previous page reads a fixed window and wires every position to the output separately. That is the ceiling it hits. Attention removes it by choosing where to look, and unlike almost anything else inside a model, the choice is a number per character that you can read off.

Sixth in the series, and the last component. Everything here is one attention head, 1,618 parameters, gradients written out by hand. 12 August 2026.

A question a window cannot answer

Here is a task built to need memory rather than pattern. Each line pairs letters with digits and then asks for one of them again:

The fixed-window model tries hard

First, the model from the previous piece, given a window of 16 characters — the whole line, so it is not being starved of information. It has 9,570 parameters, and it learns a surprising amount:

It learns the format perfectly and narrows the answer to the right three candidates most of the time. Then it stops. Choosing between them requires finding which pair began with the queried letter, and the position of that pair changes from line to line. A flattened window has one weight per position, so the only rules it can express are of the form the character at offset seven matters. There is no offset that is right every time.

One head, and why it also fails

So: attention. Build a query from the current position, compare it against a key at every position, softmax the comparisons into weights, and take a weighted average of the values there. Which position matters is decided from the content, at run time.

x_i     = C[char_i] + P[i]        embedding + position
q       = x_last @ Wq             one query, from where we are now
k_i     = x_i @ Wk                a key at every position
v_i     = x_i @ Wv                a value at every position
score_i = (q . k_i) / sqrt(16)
w       = softmax(score)          how much to look at each position
context = sum_i w_i v_i
logits  = context @ Wo + b

I built that, trained it, and it scored 12%. Chance is 10%. It is worse than the fixed window it was supposed to beat.

What the second layer is for

Real transformers do this with two layers. The first one does something that sounds trivial: at every position, it copies information about the previous character forward. After that, a position holding 9 also carries a trace of the d that came before it — and now a query built from d has something to match. The second layer does the matching and reads off the value. The pair is called an induction head, and it is one of the few things inside a large model that has been pinned down mechanically.

Implementing two layers here would multiply the code and hide the point, so instead I supplied what the first layer would produce: each position's value carries the next character rather than its own. One line different. Everything else — the query, the keys, the matching, the softmax — is unchanged and still learned from scratch.

Perfect, with 6 times fewer parameters than the model that managed 31%. Not because it is bigger — because the operation matches the problem.

Watching it choose

Here is the part that is hard to get from anything else. The attention weights are the model's own account of where it looked, and there is one per character. These three lines query a pair in a different place each time:

The peak lands at position 6, then 9, then 12 — it moves to wherever the matching letter is. Nothing in the weights encodes those positions. The head compares the query against every key and the softmax does the rest, which is exactly the thing the fixed window could not express.

Look inside it yourself

This is the trained head — all 1,618 numbers of it, loaded into the page. Change the line and watch the weights move. It only knows the characters from its task: letters a–f, digits, and spaces.

What this is not, again

One head, at one position, in one layer. A transformer runs this at every position at once, with several heads in parallel looking for different things, stacked in dozens of layers with a feed-forward network between each, and it learns the previous-character step rather than being handed it. What does not change with any of that is the operation: compare a query to keys, softmax, take a weighted average.

Six pieces ago this series started with a word being chopped into st, raw and berry. Between there and here is every component of a language model except scale: what it reads, where those pieces came from, what the thing in the middle is, how it decides where to look, how the next piece gets chosen, and what all of it costs. None of it required trusting me — every number came from a script you can read, and every tool runs on text of your own.

Both implementations are published: attn.py and attn.js, neither using an autodiff or matrix library. The build checks each one's gradients against finite differences and checks the two agree on initialisation, forward pass and the attention weights themselves — see checkattn.py. The task generator is task.py.