Why do GPT, Claude, and Gemini give different token counts?

Different vocabularies and merge rules map the same characters to different subword sequences, so the integers diverge.

Why is my emoji more tokens on one model?

Emoji and multi codepoint sequences are covered unevenly across vocabularies. One model may keep a glyph as a single piece; another may split it.

How do I get an Exact Claude token count?

Call Anthropic’s documented count_tokens path with the same messages you will send. Browser Approx is for planning only.

Chinese text blows up my token budget on GPT but not Gemini. Why?

CJK coverage differs by vocabulary. Measure both families on your real strings with comparison plus official count APIs.

Anyone else see 15 to 20% undercount when using tiktoken for Claude?

Yes, that pattern shows up when OpenAI encodings are reused for Anthropic. Magnitude varies. Stop using tiktoken as a Claude billing oracle.

Do tool schemas change counts beyond the pasted string?

Yes. Schemas, system prompts, and chat wrappers add tokens that a bare paste misses.

Should I compare $/1M without comparing tokenizers?

No. Tokenization changes volume. Compare end to end cost on shared text with trusted counts and catalog rates.

Are browser counters Exact for Claude and Gemini on TokenCalculator?

No. Those rows are Approx. OpenAI supported encodings can be Exact in browser.

Does Gemini use SentencePiece?

Many Gemini text flows use Google’s SentencePiece style path. Confirm with current Google docs and countTokens for your model.

Home / Guides / Why GPT, Claude, and Gemini token counts differ

Why GPT, Claude, and Gemini token counts differ

Why the same text produces different token counts on GPT, Claude, and Gemini: vocabularies, BPE vs SentencePiece, Exact vs Approx, and how to budget without drift.

The short answer

Token counts differ because each model family maps text onto its own vocabulary. Subword tokenizers learn which pieces are common. OpenAI public encodings (cl100k_base, o200k_base) are not Anthropic’s Claude vocabulary and are not Gemini’s SentencePiece style path. Identical UTF-8 can become different piece lists, so the integer changes.

If your planning assumed tokens roughly equal words times 1.3 for every vendor, that assumption is the bug. See tokens vs words and tokenization.

What actually changes between providers

Vocabulary size and contents change how rare words, code glyphs, and CJK characters split. Merge rules (BPE vs other) shift boundaries between pieces. Special and control tokens from chat templates can add pieces you never typed in the user box. Multilingual coverage means the same paragraph can be cheap on one model and dense on another. Counting tool honesty matters: fake Exact labels hide heuristic error.

TokenCalculator shows Exact only when a matching OpenAI style tokenizer runs in browser. Claude and Gemini are Approx until you call official count APIs. Policy detail: exact vs approximate tokens. Side by side UI: tokenizer comparison.

Worked intuition: one sentence, three stories

English often lands in a similar ballpark across families, but not identical. JSON keys, braces, and escaped strings create many small pieces, so gaps widen. CJK characters may be single tokens or multi token sequences depending on vocabulary coverage. Teams report Chinese blowing up GPT but not Gemini when workloads hit uneven coverage. Measure; do not guess.

Emoji is another classic surprise. One glyph can be one token on one stack and several on another, especially when skin tones or ZWJ sequences appear.

How many tokens are in a word?

There is no universal answer. English averages are planning heuristics only. A word can be one token or several. Across GPT, Claude, and Gemini the average for the same corpus can shift. Always count on the target tokenizer family. Primer: tokens vs words.

Wrappers, tools, and usage metadata

Paste only the user message and you undercount production. System instructions, tool schemas, RAG chunks, and provider chat formatting add tokens. After a live call, usage metadata (or invoice lines) is the ground truth for that request. Browser Exact counts are exact for the pasted string and encoding, not a legal guarantee that every billing line matches if you omitted wrappers.

Why budgets drift 10 to 30%

Common failure chain: build a GPT estimate with Exact tiktoken, reuse that integer for Claude or Gemini, multiply by a catalog or blog price per million figure, then ship. Invoice disagrees.

Undercounts of roughly 15 to 20% when people use tiktoken for Claude are a frequent forum complaint. Direction and magnitude depend on the text. The fix is methodological: never treat OpenAI tokenizers as Anthropic oracles. Guide: do not use tiktoken for Claude.

Price per million tokens without tokenizer comparison also misleads. Prefer end to end cost on one golden prompt via the cost calculator and TokenCalculator catalog rates. Do not invent live prices.

Encoding switches inside one provider

Even inside OpenAI, o200k_base vs cl100k_base can change counts for the same string. Newer GPT-4o era models often use o200k. Older GPT-4 / GPT-3.5 era paths often used cl100k. Deep dive: o200k vs cl100k.

How to diagnose a surprising gap

Confirm you pasted the same text (including whitespace). Run tokenizer comparison. Check Exact vs Approx badges. For Approx rows, re count with Anthropic count_tokens or Google countTokens. Add system and tool text if production includes them. Re price with catalog rates, then decide if context fit still holds.

Content type deep dive: tokenizer efficiency by content. Cluster hub: tokenizer comparison.

Common mistakes

Avoid these cross provider counting mistakes.

Assuming one tokenizer rules all providers
Hiding heuristics behind precise UI copy
Comparing different prompts across models
Ignoring CJK, code, and emoji as edge cases when they are the product
Trusting word or character formulas for cross vendor budgets
Skipping official count APIs before large commits

Frequently asked questions

Why do GPT, Claude, and Gemini give different token counts?: Different vocabularies and merge rules map the same characters to different subword sequences, so the integers diverge.
How many tokens are in a word?: It depends on language, tokenizer, and the word itself. Use averages only for rough planning, then count on the target model family.
Why is my emoji more tokens on one model?: Emoji and multi codepoint sequences are covered unevenly across vocabularies. One model may keep a glyph as a single piece; another may split it.
How do I get an Exact Claude token count?: Call Anthropic’s documented count_tokens path with the same messages you will send. Browser Approx is for planning only.
Chinese text blows up my token budget on GPT but not Gemini. Why?: CJK coverage differs by vocabulary. Measure both families on your real strings with comparison plus official count APIs.
Anyone else see 15 to 20% undercount when using tiktoken for Claude?: Yes, that pattern shows up when OpenAI encodings are reused for Anthropic. Magnitude varies. Stop using tiktoken as a Claude billing oracle.
Do tool schemas change counts beyond the pasted string?: Yes. Schemas, system prompts, and chat wrappers add tokens that a bare paste misses.
Should I compare $/1M without comparing tokenizers?: No. Tokenization changes volume. Compare end to end cost on shared text with trusted counts and catalog rates.
Are browser counters Exact for Claude and Gemini on TokenCalculator?: No. Those rows are Approx. OpenAI supported encodings can be Exact in browser.
Does Gemini use SentencePiece?: Many Gemini text flows use Google’s SentencePiece style path. Confirm with current Google docs and countTokens for your model.

Try it in TokenCalculator

Paste one prompt into tokenizer comparison and watch GPT Exact rows diverge from Claude and Gemini Approx rows.

Open tokenizer comparison · Open tokenizer · Estimate API spend · All guides

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.