Tokenizer Glossary

The vocabulary you keep meeting around tokenizers and language models, in plain language. Pair it with the live playground and the API reference.

Text in, integers out

TokenThe unit a model actually reads.: Not a word and not a character — a chunk of bytes that the tokenizer's vocabulary has an entry for. Common words are one token, rare words split into pieces, and every price and context limit you meet is counted in these.
TokenizerDeterministic software, not a neural network.: It maps text to a list of integers and back again. Same input, same output, every time — there is no model inference involved, which is why it can run entirely in your browser.
VocabularyThe fixed list of known tokens.: Typically 50,000–200,000 entries. Each entry has an index; that index is the token ID the model receives. Different model families use different vocabularies, so counts differ between providers.
BPE (Byte Pair Encoding)How the vocabulary is built.: Start from raw bytes, repeatedly merge the most frequent adjacent pair, and record the merge order. Encoding replays those merges greedily. Because it bottoms out at bytes, nothing is ever “unknown”.
EncodingA named vocabulary plus its merge rules.: o200k_base powers GPT-4o and GPT-4.1; cl100k_base powers GPT-4, GPT-3.5 and the embedding models; p50k_base and gpt2 are the older ones. Picking the wrong encoding gives the wrong count.
Token IDThe integer sent to the model.: An index into the vocabulary. Reversible: decode the ID and you get the exact substring back, leading space included.
Special tokenStructure, not content.: Markers such as end-of-text or chat role delimiters. They occupy vocabulary slots and are counted, which is why a chat request costs a little more than the raw message text.

Cost and limits

Context windowHow much the model can hold at once.: Measured in tokens, and shared between your input and the model's output. A “128k context” is 128,000 tokens, not characters.
Input / output tokensBilled separately, at different rates.: Output is usually several times more expensive than input. Trimming a system prompt that runs on every request compounds; trimming the answer changes quality.
Prompt cachingPaying less for a repeated prefix.: Providers can cache an unchanged leading portion of a prompt and charge a reduced rate for it. It only works if the prefix is byte-identical, so keep the volatile parts at the end.
TruncationWhat happens when you overflow.: Something has to go: oldest messages, middle content, or the request fails. Counting tokens before you send is how you choose instead of finding out.

Around the model

EmbeddingA token or a text turned into a vector.: Inside the model, each token ID is looked up in an embedding table and becomes a vector of numbers that encodes meaning. Embedding APIs expose the same idea for whole texts: similar meanings land close together, which is what powers semantic search and RAG.
Vector / similarity searchFinding by meaning rather than keyword.: Store embeddings, then compare a query's vector against them (usually by cosine similarity) to retrieve the closest texts.
RAG (Retrieval-Augmented Generation)Fetch first, then answer.: Retrieve relevant passages, paste them into the prompt, and let the model answer from them. It trades tokens for accuracy — which is exactly why token counting matters.
Prompt engineeringDesigning the input so the output is reliable.: Being specific, showing examples, stating the output format, and putting constraints where the model will actually honour them. It is an editing discipline: every instruction costs tokens on every single request.
System promptThe standing instructions.: Sent ahead of the conversation to set role, tone and rules. It is re-sent on every turn, so its length is a fixed tax on the whole app.
Few-shot examplesTeaching by demonstration.: Two or three worked examples in the prompt usually beat a paragraph of description — and cost fewer tokens than people expect if you keep them short.
TemperatureHow adventurous the sampling is.: 0 is near-deterministic and best for extraction and classification; higher values add variety and are better for drafting and ideation.
StreamingTokens arriving one at a time.: The model generates left to right, so the response can be rendered as it forms. That is why partial words sometimes flash in the UI.
HallucinationFluent and wrong.: The model predicts plausible next tokens; plausibility is not truth. Grounding the prompt in retrieved facts is the main mitigation.

Missing a term? Tell me and I will add it.