Type over it with anything you like. Each colored piece is one token — click one for its id, or switch the output to the raw integer sequence.
A language model never sees your text. It sees those integers. A tokenizer chops your prompt into chunks and looks each one up in a fixed vocabulary. Cost, context limits, and a lot of strange model behavior start here.
How the tokens are made
Byte-pair encoding. Start from raw bytes. Merge the most frequent adjacent pair,
again and again, until the vocabulary is full. Its size is fixed up front —
cl100k_base stops at ~100k entries, o200k_base at ~200k. Both are OpenAI's and
open source; Anthropic and Google keep Claude's and Gemini's private. Flip between
the two above: same text, different tokens.
Common words survive whole. Rarer ones break into fragments. Then it freezes — the vocabulary knows nothing about meaning, only what was frequent in the corpus it was built from.
Nothing is ever unknown, though. Worst case, text falls back to single bytes. Unfamiliar is expensive, not impossible.
Splits land wherever the byte frequencies put them — sometimes mid-character. 👋 is
four bytes, and three of them can end up in one token with the fourth stranded on its
own. Neither piece decodes to anything, so the sandbox groups them.
What tokenization explains
Once you can see the tokens, these stop being mysterious.
-
Letter puzzles. Asked how many r's are in "strawberry", the model never receives the ten letters — the whole word arrives as a single token. It can't count them directly; it has to spell the word out first.
tokenize -
Arithmetic. Most numbers arrive split at arbitrary places, and nothing in the vocabulary marks the pieces as quantities.
tokenize -
The multilingual tax. The same sentence in another language can produce a completely different number of tokens.
tokenize -
Structured-output overhead. Pretty JSON, minified JSON, and CSV carry the same data at very different token counts, and you pay the difference on every request and every response.
tokenize -
Truncation surprises. Context limits count tokens, not characters, so the character budget of a "128k window" swings with language and formatting.
Counts depend on the tokenizer, prices on the model — and input, output, and cached tokens are rarely billed alike. Measure what you actually use.
Takeaway
You read letters and words. A model reads neither. Your text is cut into chunks from a fixed vocabulary and handed over as a list of ids — the letters inside a chunk are not something it ever sees.
Those ids are row indices into an embedding matrix. That's the next layer down.
Tokens are the real interface to a model — its actual vocabulary. Experiment with your own text above.