Home / Guides / How many tokens in a word? GPT, Claude, Gemini

How many tokens in a word? GPT, Claude, Gemini

Tokens per word for English, code, CJK, and emoji across GPT, Claude, and Gemini tokenizers, and how to measure efficiency without inventing prices.

Efficiency means tokens for the same human job

Tokenizer efficiency here means: for a fixed string that does a job (answer a question, review a diff, localize a title), how many tokens does each vocabulary emit? Fewer tokens for the same job can mean lower input cost and more context headroom, but only after you multiply by that model’s catalog rate. Cheap tokens on a pricey SKU can still lose. Always finish in the cost calculator with TokenCalculator rates. Never invent live dollar prices.

English prose vs code

English words often align with frequent subwords. Code does not. Identifiers, operators, braces, and indentation create many short pieces. Plain English is closer to popular tokens per word heuristics. Markdown docs add mild overhead from punctuation and links. Source code raises tokens per visible character. Minified JSON and logs fragment heavily.

Does code use more tokens than English? Usually yes for the same character length. That is why a 10 page PDF of prose and a 10 page PDF of code are not comparable budgets. Count both in tokenizer comparison. Background: tokens vs words, tokenization.

CJK: Chinese, Japanese, Korean

CJK characters interact strongly with vocabulary coverage. One family may keep many characters as single tokens. Another may split them into multiple byte level pieces. Product teams see Chinese blowing up token budgets on GPT but not Gemini when their corpus hits uneven coverage.

Measure real locale strings. Keep separate golden prompts per language because rankings can flip by locale. Verify Approx with official APIs. Re check after model upgrades because new encodings (for example o200k) can change CJK efficiency.

OpenAI encoding shifts: o200k vs cl100k. Cross provider why: why token counts differ.

Emoji and symbols

Emoji, especially sequences with joiners or skin tones, are classic 1 token vs 5 tokens depending on the tool cases. Some vocabularies include common emoji as single pieces. Others fall back to smaller units. If your UX is emoji heavy (chat, reactions, marketing copy), include emoji samples in the golden pack.

How many tokens in a word across GPT, Claude, Gemini?

There is no shared constant. Averages differ by family and by corpus. For English planning, people still use rough heuristics, then replace them with counts. For code and CJK, heuristics fail faster. Use Exact OpenAI paths where available, Approx labels for Claude and Gemini in browser, then Anthropic count_tokens / Google countTokens for commits.

Label policy: exact vs approximate tokens. Wrong tool warning: do not use tiktoken for Claude.

Measurement workflow

Build four short fixtures: English paragraph, TypeScript function, Japanese or Chinese title block, emoji line. Paste each into tokenizer comparison. Record Exact vs Approx and the token integers. For Approx providers you will buy, verify with official count APIs. Price each fixture times monthly volume in the cost calculator with catalog rates. Decide with end to end cost, not price/MTok alone.

Single model deep dive: tokenizer. Cluster hub: tokenizer comparison.

Should you pick the most efficient tokenizer?

Only inside a quality and rate constraint. The most efficient vocabulary on CJK might be the wrong model for your task accuracy. Efficiency is an input to cost, not a substitute for evals. Compare providers on the same fixture pack so tokenizer and price effects stay entangled on purpose.

Common mistakes

Avoid these content efficiency mistakes.

  • Using English only samples to rank multilingual products
  • Treating tokens per word as universal across code and prose
  • Ignoring emoji as noise when it is user generated content
  • Ranking solely by $/1M tokens from a blog
  • Trusting tiktoken for Claude efficiency claims
  • Forgetting wrappers and tool schemas outside the fixture string

Frequently asked questions

Does code use more tokens than English?
Often yes for similar character length, because operators and identifiers fragment into more subwords. Measure your repos rather than assuming a fixed factor.
How many tokens are in a word?
It depends on language and tokenizer. Use averages only for early planning, then count on the target family.
Why is my emoji more tokens on one model?
Vocabularies cover emoji unevenly. Multi codepoint emoji sequences amplify the gap.
Chinese text blows up my token budget on GPT but not Gemini. Why?
CJK coverage differs by vocabulary. Compare both on your real strings and verify Approx rows with official APIs.
Tokenizer comparison for code vs English: what should I paste?
A representative function or diff for code, and a paragraph of real product prose for English. Keep lengths intentional so ratios are interpretable.
Should I compare $/1M without comparing content type efficiency?
No. A model that looks cheap per million tokens can tokenize your CJK or code workload denser and lose on end to end cost.
Are browser counters Exact for Claude and Gemini efficiency tests?
On TokenCalculator they are Approx for those families. Use them for directional gaps, then verify.
Can I use tiktoken to judge Claude efficiency on CJK?
No for billing truth. tiktoken is OpenAI oriented. Use Anthropic count_tokens for Claude.
Best way to compare providers on mixed content?
One golden pack with English, code, CJK, and emoji; /tools/tokenizer-comparison; official counts for Approx rows; cost calculator with catalog rates.
Why do GPT, Claude, and Gemini give different token counts on the same code file?
Different vocabularies and merge rules. Code punctuation makes those differences visible quickly.

Try it in TokenCalculator

Paste English, a code sample, and a CJK string into tokenizer comparison and compare Exact OpenAI rows to Approx Claude and Gemini rows.

Open tokenizer comparison · Open tokenizer · Estimate API spend · All guides

Related tools

Tokenizer comparison · Token visualizer · OpenAI tokenizer · Cost calculator

Related guides

Tokenizer comparison: GPT vs Claude vs Gemini · Why counts differ · o200k vs cl100k · tiktoken and Claude · Exact vs Approx

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.