Code Walkthrough

Six steps from "mock numbers" to counts that match the provider's own tokenizer — with no API, no key and no text leaving the browser.

1. 01Use the real merge ranksThe GPT vocabularies are public. js-tiktoken replays the exact same BPE merges in JavaScript, so the counts are not an approximation.

   js-tiktoken/lite is the encoder without any bundled vocabulary — you feed it the ranks yourself, which is what keeps the initial bundle small.

   TypeScriptCopy

   import { Tiktoken } from "js-tiktoken/lite";
   
   const ranks = (await import("js-tiktoken/ranks/o200k_base")).default;
   const encoder = new Tiktoken(ranks);
   
   encoder.encode("tokenizer"); // [30075, 3213]

2. 02Load each vocabulary lazilyA vocabulary is megabytes of merge pairs. A dynamic import inside a switch means the reader only downloads the encoding they picked.
3. 03Cache encoders, dedupe in-flight loadsParsing the ranks is the expensive part. Do it once per encoding, and make concurrent requests share a single promise.
4. 04Guard against out-of-order resultsTyping fires an async effect per keystroke. A run counter plus a cancelled flag makes sure only the newest result reaches state.
5. 05Decode each token for displayencode() returns integers. Decoding one id at a time gives the exact substring the model sees, including its leading space.
6. 06Say what is true in the UILoading, error and ready are three different messages, and the footnote states exactly where the computation happens.

Try it against the real thing

Paste the same string into the OpenAI tokenizer — the counts will agree.