Stop using tiktoken for Claude: is that absolute?

For production money and SLAs, yes. For a napkin sketch with huge margin, some teams still peek, but they should not call it Exact.

How do I get an Exact Claude token count?

Anthropic’s documented count API for Messages style requests, using the same content you will send.

Are browser token counters Exact for Claude on TokenCalculator?

No. Claude rows are Approx. OpenAI supported encodings can be Exact.

Anyone else see 15 to 20% undercount when using tiktoken for Claude?

That complaint pattern is common. Your delta will vary. Change tools rather than hunting a universal fudge factor.

My Claude bill is higher than my GPT estimate for the same prompt. Why?

Often the Claude side used the wrong tokenizer, omitted wrappers, or ignored output tokens. Re count with Anthropic tooling, then re price with catalog rates.

Should I compare $/1M without comparing tokenizers?

No. Wrong token volumes make price/MTok rankings misleading across GPT and Claude.

Do not use OpenAI tokenizer for Anthropic: what about Gemini?

Same rule family: do not use OpenAI encodings as Gemini truth. Use Google countTokens.

What should I open first when migrating off tiktoken for Claude?

Open /tools/tokenizer-comparison for labeled side by side planning, then Anthropic count_tokens, then the cost calculator.

Home / Guides / Stop using tiktoken for Claude

Stop using tiktoken for Claude

Why OpenAI tiktoken undercounts Claude, how Anthropic count_tokens gives Exact billing counts, and how to fix multi-provider cost estimates.

The hard rule

tiktoken (and browser libraries that implement OpenAI encodings like cl100k_base and o200k_base) answers OpenAI vocabulary questions. Claude uses Anthropic’s tokenizer. Those are not the same map from characters to pieces.

Using tiktoken to forecast Claude input is a category error. It can look precise, ship in a spreadsheet, and still miss invoices by a wide margin. TokenCalculator refuses to label Claude rows Exact when only an OpenAI style tokenizer is available. That is intentional. See exact vs approximate tokens.

Can I use tiktoken for Claude?

For learning and rough relative size, people sometimes peek at an OpenAI count. For billing, context gates, and vendor bake offs: no. Authoritative Claude input counting goes through Anthropic’s Messages count_tokens (or the documented count endpoint for your API version) with the same message structure you will send.

Gemini has the same lesson with a different API: use Google countTokens, not tiktoken. Cross provider primer: why token counts differ.

Why the undercount shows up in forums

Engineers paste a prompt, run tiktoken with cl100k or o200k, multiply by a Claude price per million figure from a blog, and present a Claude is cheaper slide. Production uses Anthropic tokenization plus full message wrappers. Reports of roughly 15 to 20% undercount are common enough to treat as a process smell, not a one off. Magnitude depends on language, code density, and emoji. Do not memorize a fixed percent. Fix the method.

tiktoken cl100k / o200k is the wrong tool for Claude billing. TokenCalculator Approx Claude rows are planning only, labeled honest. Anthropic count_tokens is authoritative for that message pack. Live usage metadata or invoice is ground truth after the call.

Is it OK to approximate Claude with tiktoken offline?

Only if you treat the number as a sketch, add generous margin, and never present it as Exact. Better offline workflow for mixed stacks: Exact OpenAI counts via tiktoken compatible tools or TokenCalculator Exact rows; Approx Claude / Gemini in tokenizer comparison for early design; official count APIs before finance locks or context SLAs; cost via the cost calculator with TokenCalculator catalog rates (never invented live prices).

My Claude bill is higher than my GPT estimate

Three usual causes: Claude tokens were estimated with OpenAI encodings; the GPT estimate omitted system or tool tokens that Claude production includes; output or reasoning tokens dominated while the spreadsheet only counted input.

Fix counts first, then compare catalog rates on the same golden prompt. Prefer end to end cost over bare price/MTok screens. Encoding specifics for OpenAI: o200k vs cl100k. Hub: tokenizer comparison.

How to get an Exact Claude token count

Build the messages array exactly as production (system, user, tools, history as applicable). Call Anthropic’s count_tokens for that structure. Optionally compare the integer to TokenCalculator’s Approx row to calibrate your planning margin. Store the verified count next to the prompt hash in your internal docs.

Browser counters on TokenCalculator are not Exact for Claude. They are Approx by design. Official docs remain the source of truth for API fields.

What about local scripts and CI?

Keep OpenAI Exact checks in CI with tiktoken or gpt-tokenizer for GPT SKUs. For Claude SKUs, call Anthropic count in CI or a scheduled job with a secret scoped to counting only. Do not save an API call by swapping in tiktoken for Claude gates if those gates protect money or context SLAs.

Common mistakes

Avoid these tiktoken for Claude mistakes.

Labeling tiktoken output as Claude tokens in dashboards
Baking OpenAI encodings into Claude cost functions
Comparing GPT Exact to Claude tiktoken estimates on slides
Ignoring message wrappers after fixing the tokenizer choice
Treating Approx UI counts as invoice substitutes

Frequently asked questions

Can I use tiktoken for Claude?: Not for authoritative Claude billing or context gates. Use Anthropic count_tokens with the real message pack.
Stop using tiktoken for Claude: is that absolute?: For production money and SLAs, yes. For a napkin sketch with huge margin, some teams still peek, but they should not call it Exact.
How do I get an Exact Claude token count?: Anthropic’s documented count API for Messages style requests, using the same content you will send.
Are browser token counters Exact for Claude on TokenCalculator?: No. Claude rows are Approx. OpenAI supported encodings can be Exact.
Is it OK to approximate Claude with tiktoken offline?: Only as a rough, heavily margined sketch. Prefer Anthropic counting before commits.
Anyone else see 15 to 20% undercount when using tiktoken for Claude?: That complaint pattern is common. Your delta will vary. Change tools rather than hunting a universal fudge factor.
My Claude bill is higher than my GPT estimate for the same prompt. Why?: Often the Claude side used the wrong tokenizer, omitted wrappers, or ignored output tokens. Re count with Anthropic tooling, then re price with catalog rates.
Should I compare $/1M without comparing tokenizers?: No. Wrong token volumes make price/MTok rankings misleading across GPT and Claude.
Do not use OpenAI tokenizer for Anthropic: what about Gemini?: Same rule family: do not use OpenAI encodings as Gemini truth. Use Google countTokens.
What should I open first when migrating off tiktoken for Claude?: Open /tools/tokenizer-comparison for labeled side by side planning, then Anthropic count_tokens, then the cost calculator.

Try it in TokenCalculator

Paste a Claude bound prompt, see Approx Claude rows beside Exact OpenAI encodings, then verify with Anthropic’s count path before you spend.

Open tokenizer comparison · Open tokenizer · Estimate API spend · All guides

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.