LLM Token Counter

About the LLM Token Counter

Language models read text as tokens — words, parts of words, punctuation and bytes — and context limits and API prices are counted in them. Paste a prompt, a document or code to get the exact number of tokens for OpenAI’s encodings (o200k_base for GPT-4o, GPT-4.1, GPT-5 and the o-series; cl100k_base for GPT-4 and GPT-3.5; o200k_harmony for gpt-oss) and for open models whose tokenizers are published: Qwen, Mistral and DeepSeek.

Every token is shown in colour, so you can see where the text is split. The page also shows how much of a context window the text fills and, from a price you enter, what it costs to send. Anthropic and Google do not publish the tokenizers of Claude and Gemini, so for those models the page gives a clearly labelled estimate based on the providers’ own rules of thumb.

Common uses

• Checking that a long prompt, document or chat history fits a model’s context window before sending it.
• Estimating what an API job costs before running it on thousands of requests.
• Comparing how many tokens the same text needs in different models — especially for Hindi and other non-English text.
• Seeing exactly how a tokenizer splits words, numbers, code and emoji while writing prompts.

Which tokenizer does my model use?

• o200k_base: GPT-5 models, GPT-4.5, GPT-4.1, GPT-4o, o1, o3 and o4-mini.
• cl100k_base: GPT-4, GPT-3.5 Turbo and the text-embedding-3 and ada-002 embedding models.
• o200k_harmony: gpt-oss-120b and gpt-oss-20b (the same tokens as o200k_base, plus the harmony chat-format tokens).
• Qwen3.5 / 3.6 / 3.8 and Qwen3 / Qwen2.5: two different vocabularies (about 248,000 and 151,000 tokens).
• Mistral Tekken: Mistral Small 4, Medium 3.5, Large 3, Ministral 3, Devstral 2 and Mistral Nemo. Mistral v3: Mistral 7B v0.3 and Mixtral 8x22B.
• DeepSeek-V3 / R1.

The OpenAI list follows OpenAI’s own tiktoken library. For the open models, each family was checked: the models named give the same tokens for ordinary text.

Claude and Gemini: estimates only

Anthropic and Google do not publish the tokenizers of Claude and Gemini, so no website can count their tokens exactly offline. The estimates here use the providers’ own rules of thumb: Anthropic says a Claude token is about 3.5 English characters, and that the newer tokenizer introduced with Claude Opus 4.7 gives roughly 30% more tokens for the same text (1 to 1.35 times as many, depending on the content); Google says a Gemini token is about 4 characters. Real counts differ, most of all for code and non-English text.

Choose Claude, newer tokenizer for Claude Opus 4.7, 4.8, 5 and 5.5, Sonnet 5 and 5.5, Fable 5 and 5.1 and Mythos 5 and 5.1, and Claude, older tokenizer for Opus 4.6, Sonnet 4.6, Haiku 4.5 and older models. For an exact number, use the providers’ token-counting API endpoints (they need an API key; Anthropic’s is free of charge).

What the count includes

The count is the tokens of your text alone. Chat APIs add a few tokens per message for roles and separators, and more for tool definitions, images and files, so the number on your bill can be a little higher. Some open models add a start token (like <s>) that is not counted here.

Special tokens such as <|endoftext|>: OpenAI encodings count them as ordinary text unless you tick Read special tokens as single tokens (the way the model’s own chat format writes them). The Qwen, Mistral and DeepSeek tokenizers always treat their own special tokens as single tokens, as their reference tokenizers do.

How exact is it?

The OpenAI encodings run OpenAI’s published tiktoken vocabularies (through the open-source gpt-tokenizer library); the Qwen, Mistral and DeepSeek counts run each model’s own tokenizer.json file. The token numbers were checked against OpenAI’s tiktoken and Hugging Face’s reference tokenizers on English, code, Hindi, Chinese, Japanese, Korean, Arabic, emoji and a 200,000-character book extract, with identical results.

Context windows and cost

A context window is the most tokens a model can handle in one request — your text plus the reply. Enter the reply length under Reserve for the reply to see what is left. Write large numbers in full or with k and M (128k = 128,000; 1M = 1,000,000). Check your provider’s current documentation for the window and prices: they change, and many providers charge different prices for input, output and cached tokens.

Where the tokenizers come from

OpenAI’s encodings come from the gpt-tokenizer package (MIT licence). The Qwen tokenizer files are © Alibaba Cloud and the Mistral files © Mistral AI, both under the Apache 2.0 licence; the DeepSeek file is © DeepSeek under the MIT licence. They are served from this site, unchanged except for removed whitespace and compression; the licence texts and a notice listing the source of each file are linked under Compare tokenizers.

Limitations

• Claude and Gemini counts are estimates (see above), not exact.
• Token boundaries are shown for the first 5,000 tokens; the count and the copied IDs always cover the whole text.
• Very long texts (several MB) take a few seconds to count, and each open-model tokenizer is a 0.4–3.3 MB download the first time.
• Images, audio, files and tool definitions are counted by the providers in their own ways; only text is counted here.

Privacy

Everything happens in your browser. What you enter or open here is not uploaded or stored by MySmartCoPilot. The tokenizer you choose is downloaded once from MySmartCoPilot (0.4–3.3 MB); your text is never sent anywhere.