How tokenizers work (with a playground)

Models don't read text

A language model has no concept of letters or words. Its input is a list of integers, each one an index into a fixed vocabulary — typically 50k–200k entries. The component that turns "Hello, world" into [15496, 11, 995] is the tokenizer, and it is a separate, deterministic, non-neural piece of software.

Everything you pay for and every limit you hit is measured in these units, not in characters.

Why not just use words?

Three naive options, all bad:

• Characters — tiny vocabulary, but sequences get very long, and long sequences are expensive: attention cost grows quadratically.
• Words — short sequences, but the vocabulary is unbounded. Names, typos, code identifiers, and other languages all become "unknown".
• Bytes — universal, but even longer than characters.

Subword tokenization is the compromise: frequent words stay whole, rare words break into reusable pieces. Nothing is ever unknown, because the pieces bottom out at single bytes.

Byte Pair Encoding in one paragraph

BPE starts from raw bytes and repeatedly merges the most frequent adjacent pair in the training corpus into a new symbol. Do that 50,000 times and you have a vocabulary where the is one token, token is one token, and izer is one token, but a rare surname is three or four. At inference the learned merge list is replayed greedily over your text — same input always produces the same tokens.

"tokenizer"  ->  ["token", "izer"]
"unbelievably" -> ["un", "bel", "iev", "ably"]
"1234567"    ->  ["123", "456", "7"]

The leading-space trick

In GPT-style tokenizers the space belongs to the *following* word: world is one token, distinct from world. This is why the same word can have a different token count depending on where it appears, and why stray double spaces quietly cost you extra.

Rules of thumb

• English prose: roughly 4 characters per token, or ~0.75 tokens per word.
• Code: worse — punctuation, indentation, and camelCase identifiers fragment heavily.
• Non-Latin scripts: much worse — often 1–3 tokens per character, which is why the same sentence in Japanese or Tamil can cost several times more than in English.
• Numbers split into digit groups, so long IDs are surprisingly expensive.

Why this matters in practice

• Cost. Billing is per token in and per token out. Trimming boilerplate from a system prompt that runs on every request compounds fast.
• Context limits. A "128k context" is 128k tokens, not characters. Estimate before you truncate.
• Behaviour. Odd tokenization explains classic failures: counting letters in a word, reversing strings, and arithmetic on long numbers are all hard because the model never sees the individual characters.
• Streaming. Output arrives token by token, which is why partial words appear mid-stream.

About the playground above

It used to be a mock — a hand-written rule set that imitated the *shape* of BPE. It isn't any more. The playground now runs the real GPT merge ranks (o200k_base, cl100k_base, p50k_base, gpt2) inside your browser through js-tiktoken, so the token count, the token boundaries, and the token IDs match OpenAI's own tokenizer exactly.

Two things are worth saying plainly:

• Nothing leaves your browser. The vocabulary file is downloaded once and the encoding happens locally. No API call, no API key, no cost, and your text is never sent anywhere.
• Exact, but for the text you typed. A real chat request also spends tokens on chat formatting, system prompts, tool schemas and images. Those are counted by the provider on top of what you see here.

Want more? There is a full tokenizer lab with the problem, the demo and the code walkthrough, an API reference covering the endpoints and libraries each provider exposes, and a glossary for the vocabulary around tokens, embeddings and prompt engineering.

Takeaways

• Text becomes integers before the model sees anything.
• Subword BPE balances sequence length against vocabulary size.
• Token counts drive cost, context, and several well-known model weaknesses.
• Counting locally with the real merge ranks is free and exact — there is no reason to guess.