Problem Statement: Real In-Browser Tokenizer

Overview

Build a tokenizer playground that reports exact GPT token counts without calling any API, without an API key, and without sending the user's text off the device.

A mock tokenizer is easy and useless: if the number is not the number the provider bills, nobody can plan a prompt budget with it. The real merge table is public, so the honest version is also the free version.

Requirements

Functional Requirements

1. Exact counts

• Token count, token boundaries and token IDs must match OpenAI's tokenizer byte for byte
• Support multiple encodings: o200k_base, cl100k_base, p50k_base, gpt2
• Switching encoding re-tokenizes the current text immediately

2. Privacy and cost

• No network request carrying user text — ever
• No API key, no server function, no billing surface
• The only network cost is the vocabulary file itself, fetched once and cached

3. Input and output

• Free-text input with sample strings covering prose, code, numbers and non-Latin script
• Two views: token text (with visible whitespace) and raw token IDs
• Metrics: tokens, characters, characters per token

4. Feedback states

• Loading state while the merge ranks download
• Error state if the vocabulary fails to load
• Announce updates to assistive technology

Non-Functional Requirements

• The vocabulary is large (roughly 1–4 MB depending on encoding) — it must not ship in the initial bundle
• Only the encoding the user actually selects is downloaded
• Repeated encoding switches must reuse the already-parsed encoder
• Rapid typing must not produce out-of-order results

Constraints

• Browser only; the page is statically rendered, so the encoder cannot exist during SSR
• Tokenizing must stay synchronous once the encoder is in memory, so typing feels instant
• Reduced-motion and keyboard users are first-class

Success Criteria

• Paste any string into this playground and into the provider's own tokenizer: the counts agree
• Network tab shows one vocabulary fetch and nothing else
• Initial page weight is unchanged for readers who never open the playground