A tiny language model with its insides showing
Most likely: · Drawn:
NextWord runs a complete, if tiny, transformer in your browser, the same kind of model that powers large language models, and shows every number it computes. Knowing which numbers to trust is half of the lesson, so here is exactly where each one comes from.
Splitting the text into words; the embedding lookup; sinusoidal position encoding; LayerNorm; two-head scaled dot-product attention with a causal mask; the output projection and both residual adds; a 16-unit GELU feed-forward layer; the final scores against every word’s embedding; temperature, top-k and the random draw. Every attention row adds up to exactly 100%, and the panel checks it. All of it lives in js/model.js.
js/model.js
8 numbers per word instead of thousands; 2 attention heads instead of dozens; 1 block instead of 12 to 100; a vocabulary of 25 words instead of about 50,000; at most 8 words of context instead of thousands.
Nothing here was trained. The weights come from a fixed random seed, so on their own they would predict noise. To keep the output readable, each final score adds a word-pair prior counted from twelve short sentences. The scores panel shows the two parts separately, and the prior usually wins. Treat the predicted word as scenery; treat the arithmetic as the lesson.
Real models split text into sub-word pieces, not whole words, and the attention patterns here mean nothing in particular, because nothing taught the heads what to look for.