Tokenizer API reference

The same library the playground on this site uses. Exact counts, no network call, no key, no cost. This is the default answer for token counting.

no endpoint — runs in your process or browser

import { Tiktoken } from "js-tiktoken/lite";
import o200k from "js-tiktoken/ranks/o200k_base";

const enc = new Tiktoken(o200k);
const ids = enc.encode("How many tokens is this?");

ids.length;            // 6
enc.decode([ids[0]]);  // "How"

• Python equivalent: tiktoken.encoding_for_model("gpt-4o").encode(text)
• o200k_base for GPT-4o / 4.1 / o-series, cl100k_base for GPT-4 / 3.5 / embeddings
• Counts the text only — chat formatting and tool schemas add a few tokens per message

Every completion returns exactly what you were billed for. This is the ground truth: use it to reconcile, not to plan.

POST https://api.openai.com/v1/chat/completions

curl https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{ "role": "user", "content": "Hello" }],
    "max_tokens": 1
  }'

# -> "usage": { "prompt_tokens": 8, "completion_tokens": 1, "total_tokens": 9 }

• Costs money — it is a real generation, even with max_tokens: 1
• Add stream_options: { include_usage: true } to get usage on streamed responses
• prompt_tokens includes chat scaffolding, so it is a few higher than a raw tiktoken count

A dedicated, free counting endpoint. Claude's tokenizer is not published, so for Claude this endpoint is the only exact option.

POST https://api.anthropic.com/v1/messages/count_tokens

curl https://api.anthropic.com/v1/messages/count_tokens \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "messages": [{ "role": "user", "content": "Hello" }]
  }'

# -> { "input_tokens": 8 }

• Free, but rate limited — cache results, do not call it per keystroke
• Counts system prompts, tools and images too
• Needs a key, so it belongs on your server, never in browser code

Same idea as Anthropic's: a free counting call that mirrors what generation will bill.

POST https://generativelanguage.googleapis.com/v1beta/models/{model}:countTokens

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:countTokens" \
  -H "x-goog-api-key: $GEMINI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "contents": [{ "parts": [{ "text": "Hello" }] }] }'

# -> { "totalTokens": 1 }

• Handles multimodal input — images and audio return their token cost too
• Generation responses also carry usageMetadata with the final counts

For open models (Llama, Mistral, Qwen…). Each model ships its own tokenizer.json, which you can load locally in Python or in the browser.

no endpoint — tokenizer.json loaded from the model repo

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
ids = tok.encode("How many tokens is this?")

len(ids)          # model-specific
tok.convert_ids_to_tokens(ids)

• JS equivalent: @huggingface/transformers, or the tokenizers WASM build
• Counts differ per model family — never reuse a GPT count for a Llama budget
• Chat models need apply_chat_template() to include role scaffolding