Source
Every figure on this site is computed by one of these scripts rather than typed in by hand, and every interactive tool is checked against a second implementation before it ships. Here they all are.
Nothing here is compiled or obfuscated. If you want to know how a number on this site was produced, you can read the line that produced it. The JavaScript is served at its own paths — permalink.js, bpe.js, ngram.js, tokens/app.js.
| File | What it does | Lines |
|---|---|---|
build.sh | The whole build, every step | 61 |
chatcost.py | The chat billing rule, checked against the encoder | 116 |
mlp.py | The neural language model behind /learn/, by hand | 311 |
checkmlp.py | Gradient checks, and browser vs Python agreement | 141 |
bpe.py | Byte pair encoding — the reference for /vocabulary/ | 119 |
ngram.py | The n-gram model and sampling knobs behind /predict/ | 175 |
render.py | Generates every page on the site, including this one | 1900 |
precompute.py | Computes the figures on /tokens/ | 147 |
precompute_merges.py | Computes the figures on /vocabulary/ | 66 |
precompute_predict.py | Computes the figures on /predict/ | 80 |
corpora.py | The training texts used throughout | 61 |
verify.py | Checks both shipped tokenizer bundles against the reference | 103 |
makebundle.py | Builds the cl100k browser bundle upstream got wrong | 159 |
checkbpe.py | Browser BPE trainer vs the Python reference | 99 |
checkngram.py | Browser sampler vs the Python reference | 158 |
checkhtml.py | Strict HTML parse, dead links, feed and sitemap | 156 |
checkjs.py | Compiles the site's JavaScript with a real engine | 48 |
checkpermalink.py | Round-trips shareable links through a stubbed DOM | 272 |
checklive.py | Compares served bytes against what was generated | 136 |
cjsload.py | Loads the tokenizer's CommonJS build under QuickJS | 88 |
tokenlib.py | Loads the shipped browser bundle under QuickJS | 82 |