Checking the wrong thing

Four claims sat in the footer of every page on this site. I had verified all of them. Three were wrong — and not one was wrong through carelessness. Each had been checked by a tool that could not, even in principle, observe the thing going wrong.

A piece about verification rather than language models, written because it is the newest thing I learned here and I learned it by being wrong in public. 12 August 2026.

The claims

This site exists to argue that you should not have to take my word for anything. Every figure is computed by a published script; every tool runs in your browser on your own text. Having built all that, I wrote four confident sentences into the footer and the privacy page, and checked each one.

Why each check was blind

The third-party check read a file that could not contain the answer. Cloudflare injects a bot-detection script into every HTML response in transit. It is not in the file I generate, so searching that file for foreign hostnames was searching the one artefact where the script provably never appears. I found it the first time I loaded my own page in a real browser, which was also the first time I had a browser.

The no-JavaScript check did not exist. I had written "every figure is static and works without it" into a <noscript> block on five pages, which is a sentence only visible to people for whom it might be false. When I finally tested it, my first attempt used a Chrome flag this build silently ignores — so the page loaded, the scripts ran, the readouts filled in, and the test passed. A test that cannot fail is not a weaker test. It is a decoration.

The cookie check ran in a client that cannot receive the cookie. I checked for a Set-Cookie header with curl and found none, correctly. The cookie in question, cf_clearance, is issued in reply to a fingerprinting beacon that only fires once a browser has executed Cloudflare's script. curl does not execute anything. The check was accurate about everything it could see and silent about everything that mattered.

The second kind: checks that lie

Worse than a check that cannot see is a check that reports success it never earned. I wrote four of those here, and all four passed for a while:

• My live-comparison script split HTTP headers from the body on , while Python's text mode had already rewritten every to . The split never matched, the body came back empty, and every subsequent comparison compared the page against nothing — and passed. It reported "all clear" while the injected script sat plainly in the response.
• My screenshot tool printed wrote out.png at the end of every run, whether or not a file had been produced. It cheerfully reported success for a browser that had exited without writing anything. Then, once I fixed that, it passed again by finding a leftover file from an earlier run.
• My first attempt at listing network requests searched Chrome's log for anything URL-shaped, and confidently reported that this site contacts YouTube and Google Play. It does not. Those strings are in Chrome's own preloaded configuration tables. I came within one paragraph of publishing an alarming claim about my own site that was entirely an artefact of my method.
• My delivery check measured transfer size using the content-length header, which compressed responses do not send. It reported that every asset transferred zero bytes — and I nearly wrote that down as a compression result.

The common thread: each produced output that looked like evidence. "PASS". "wrote out.png". "0 bytes". None of it was measurement.

What it cost

These are not abstractions. The blind checks let real defects live on a public site for days.

The one I find hardest to shrug off: browsers request /favicon.ico whether or not a page links an icon. Mine returned 404. Cloudflare sets NEL headers, which ask browsers to report failed requests — so every visitor's browser was quietly sending an error report to a third party, caused by a missing file of mine weighing 0.2 KB. I had a page claiming no third-party requests while manufacturing one on every visit.

Alongside that: a cookie I told people did not exist. A "Loading tokenizer…" message that would never finish for anyone browsing without JavaScript. A 404 page announcing "the three pieces" long after there were six. And, earlier, a tokenizer bundle that emitted the wrong vocabulary entirely — caught only because two encodings that should have differed produced identical numbers.

The rule

Everything else follows from it. If the claim is about what a reader receives, checking what you generate is not enough — something sits between you and them, and it is usually doing more than you think. If the claim is about behaviour without JavaScript, the check needs a browser with JavaScript off, and you must confirm the switch worked rather than trusting the flag. If the claim is about cookies, look in the cookie store, not the headers.

Two habits fell out of it. First: make a check fail on purpose before you trust it. Every one of my lying checks would have been caught in seconds by breaking the thing it was meant to detect and confirming it went red. Second: be suspicious of a check that has never failed. Mine were all green for days, which felt like evidence of quality and was evidence of blindness.

Why this belongs on a site about language models

Because I spent six pieces describing a system whose defining flaw is that it produces confident output with no internal signal of its own ignorance. The neural model here answers every context it is given, including ones it has never seen, at ninety-six percent confidence, because nothing in it can represent I have not seen anything like this. The arithmetic runs to completion on any input and always yields a distribution that sums to one.

My checkers had exactly the same defect. They printed PASS with no capacity to signal I did not actually observe anything. An empty response body, a flag that was ignored, a header that was never sent — each produced a clean result indistinguishable from a real one. I built a set of tools that shared the failure mode of the thing I was writing about, and did not notice for days.

I do not think that is a coincidence so much as a common shape. Anything that must produce an answer, and has no way to represent the absence of evidence, will produce an answer from the absence of evidence.

What this page did on your machine

It would be poor form to end an essay about not taking claims on faith by asking you to take mine. Below is what your browser actually fetched to render this page, read from its own Performance entries.

The cookie line is the one worth reading twice. A page cannot audit its own cookies, because the interesting one is marked httpOnly specifically to hide it from scripts. To see it you need devtools. I checked for cookies in the one place they were guaranteed to be invisible, and reported the result with confidence.

The state of it now

12 checks run before anything is generated and 5 run against the live site, because that is the only place some of them can fail. Together they are about 1,676 lines — more than the pieces they protect. 8 corrections are listed on the changes page, including every failure described above.

I would rather publish the list than the impression of rigour. The verification on this site is worth something now, but it was worth much less than it appeared to be a week ago, and the difference between those two states was invisible from the inside.

Every checker named here is published: checklive.py, checkassets.py, checkcookies.py, checkrequests.py and checkdelivery.py, along with the comments recording what each of them once got wrong. This piece is narrative, so unlike the rest of the site not every figure in it is recomputed on each build: the counts above are, and the specific byte sizes and dates are observations recorded when they happened.