A tiny transformer language model, laid out as a workshop, computed live
The Prediction Workshop is an isometric workshop where a cart carries your sentence around one loop of road, and each stop on that loop is one real step of a small transformer language model: the Tokenizer Bench splits your text into tokens from a 48-word vocabulary, the Embedding Floor looks up each token's 8-number vector and places it on the floor, the Attention Loom computes real 2-head causal attention over the sentence so far, and the Prediction Counter runs a genuine softmax to produce odds for the next word. Press Step there and the cart samples a word from those odds, appends it to the sentence, and drives the whole loop again — tokenizing, embedding and attending over the longer sentence from scratch, exactly like a real autoregressive model does.
The tiles in the cart's bed are the actual tokens of the current sentence. The bars on its flank are the actual 8 numbers of the last token's vector, positive up and negative down. Both are read straight out of the model, not drawn to look busy.
Genuinely computed, live, in your browser: the tokenizer split; the embedding lookup;
sinusoidal positional encoding; 2-head scaled dot-product attention with a causal mask, so a token can
never attend to one that comes after it; the residual adds; LayerNorm; a GELU feed-forward; a tied-weight
output projection; softmax with a real temperature parameter; and weighted random sampling. The attention
weights leaving any one token sum to exactly 1.00, and the next-word probabilities sum to exactly 1.00 —
both are checked live in the panel, not just claimed here. It is all in js/model.js, about
260 lines, and it runs standalone in Node with no browser at all.
Scaled down: 8 numbers per token instead of the 768–12,288+ a real model carries; 2 attention heads instead of dozens; 1 transformer block instead of dozens stacked; a 48-word vocabulary instead of 30,000–100,000+ sub-word pieces; a 24-token context window instead of thousands. Every one of these is a size, not a shortcut — the arithmetic at each size is exact.
Assumed / invented: the vocabulary itself, and every weight in the model. Nobody trained this on any text. All of it is generated once, deterministically, from a fixed random seed when the page loads, which is why the workshop behaves identically on every visit and needs no download.
Deliberately faked: nothing about the computation. The one thing to understand instead: because the weights were never trained, the word the Prediction Counter picks is not meant to make sense, and the 2D positions on the Embedding Floor are a real but arbitrary projection, not a trained map where nearby words mean similar things. Treat every sentence the workshop writes as scenery; treat the matrix multiplication, the attention weights and the softmax as the lesson.
The first time the cart reaches a station it stops for long enough to read that station's write-up — between 9 and 26 seconds — and a progress bar under the panel text shows how much of the stop is left. Once every station has been explained, the workshop runs at a watchable pace instead of a readable one. Space holds any stop indefinitely, S steps one station at a time, and the Speed slider scales everything, reading stops included. Reset (⟲) replays the slow tour; Run keeps what you have already read.
The cart runs a few laps automatically (set by "Words to auto-generate"), then parks at the depot. Pressing Step samples one more word from the odds on screen and sends it around again — you can keep doing this for as long as you like.
Built from the isometric-explainer skill. All code and copy original.