What is o200k_base vs cl100k_base?

They are two public OpenAI style BPE encodings. Newer GPT-4o era models often use o200k; many earlier GPT-4 / GPT-3.5 paths used cl100k. Confirm per model id.

o200k vs cl100k: which encoding for GPT-4o?

Prefer o200k_base for GPT-4o family models unless OpenAI documents a different encoding for that exact SKU.

Do o200k and cl100k always produce different counts?

Often they differ, but not always by a lot. Measure your strings. Do not assume a fixed percentage.

Can I use tiktoken for Claude?

Not for authoritative Claude billing. tiktoken encodings are OpenAI oriented. Use Anthropic count_tokens for Claude commits.

Is it OK to approximate Claude with tiktoken offline?

Only as a rough sketch with large margin, and never as invoice truth. Prefer Anthropic’s count API.

Why do GPT, Claude, and Gemini still differ if I pick the right OpenAI encoding?

Because Claude and Gemini are not those encodings. Right OpenAI encoding only fixes the OpenAI side.

Are TokenCalculator OpenAI counts Exact?

For supported o200k / cl100k style paths running in browser on the pasted text, yes. Wrappers you omit still undercount production.

Should I compare $/1M without comparing encodings?

No. Encoding choice changes token volume for the same prompt. Compare end to end cost with Exact counts and catalog rates.

Best way to compare providers after fixing OpenAI encoding?

Paste one golden prompt into /tools/tokenizer-comparison, verify Approx providers officially, then use the cost calculator.

Where do I learn the labeling policy?

Read the exact vs approximate tokens guide.

Home / Guides / o200k vs cl100k: which OpenAI encoding counts your text?

o200k vs cl100k: which OpenAI encoding counts your text?

o200k_base versus cl100k_base: what changed, which GPT models use which encoding, and how to compare token counts without guessing.

What these names mean

cl100k_base and o200k_base are public OpenAI style BPE encodings used by tiktoken and compatible libraries. The prefix hints at vocabulary scale (on the order of about 100k vs 200k merges/tokens in the published encoding families). They are not Claude encodings and not Gemini encodings.

TokenCalculator can run matching OpenAI style tokenizers in the browser and label those counts Exact for supported paths. That is different from Approx rows for Anthropic and Google. See exact vs approximate tokens.

Why OpenAI shipped a newer encoding

Larger, more modern vocabularies often encode some multilingual text, punctuation patterns, and newer corpora more efficiently. Moving from cl100k era models to o200k era models can change the token integer for the same UTF-8 string. Sometimes the count drops. Sometimes a specific string stays similar. You only know by counting.

If your finance model hard coded that GPT always uses cl100k, GPT-4o era traffic will disagree. Update the encoding assumption when you update the model SKU.

Which encoding for which models (practical rule)

cl100k_base: many GPT-4 / GPT-3.5 turbo era chat models, Exact via tiktoken compatible paths. o200k_base: many GPT-4o / newer OpenAI chat models, Exact via tiktoken compatible paths.

Always confirm the encoding for the exact model id you call in OpenAI’s docs or tokenizer libraries. Model lineups change. TokenCalculator’s catalog and tokenizer help you pick a row; tokenizer comparison helps you see both encodings on one paste when available.

Forum question distilled: which encoding for GPT-4o? Prefer o200k for GPT-4o family models unless the provider documents otherwise for that SKU.

o200k_base vs cl100k_base token count on the same text

Side by side Exact counts answer the engineering question. Paste a golden prompt (include system text if production includes it). Open tokenizer comparison. Read Exact counts for both encodings when exposed. Record the delta percentage for English, code, and multilingual samples separately.

Price with the cost calculator using TokenCalculator catalog rates for the model you will actually call. Do not invent live price per million figures.

Cross provider gaps are a separate issue. Claude and Gemini will not match either OpenAI encoding. Hub: tokenizer comparison. Why: why token counts differ.

Can I use these encodings for Claude?

No, not for billing truth. tiktoken with o200k or cl100k is an OpenAI path. Using it offline to approximate Claude is a common source of 10 to 30% planning error. Read do not use tiktoken for Claude. For Claude, use Anthropic count_tokens. For Gemini, use Google countTokens.

Content type still matters inside OpenAI

Even when both counts are Exact, code, CJK, and emoji can show larger deltas between encodings than plain English. Efficiency patterns: tokenizer efficiency by content. Word heuristics: tokens vs words. Mechanics: tokenization.

Migration checklist when you change GPT SKUs

Re tokenize and re price when encodings change.

Re tokenize templates with the new encoding (do not reuse old integers)
Re check context fit; a smaller or larger count changes headroom
Re price with catalog rates for the new model id
Keep Exact vs Approx discipline if you also compare Claude or Gemini
Prefer end to end cost on one prompt over raw $/1M screens alone

Common mistakes

Avoid these OpenAI encoding mistakes.

Hard coding cl100k for every OpenAI model forever
Treating o200k counts as Claude or Gemini counts
Comparing o200k on one prompt and cl100k on another
Shipping finance models that ignore encoding changes after a model upgrade
Using blog tokens per word tables instead of Exact encoding runs

Frequently asked questions

What is o200k_base vs cl100k_base?: They are two public OpenAI style BPE encodings. Newer GPT-4o era models often use o200k; many earlier GPT-4 / GPT-3.5 paths used cl100k. Confirm per model id.
o200k vs cl100k: which encoding for GPT-4o?: Prefer o200k_base for GPT-4o family models unless OpenAI documents a different encoding for that exact SKU.
Do o200k and cl100k always produce different counts?: Often they differ, but not always by a lot. Measure your strings. Do not assume a fixed percentage.
Can I use tiktoken for Claude?: Not for authoritative Claude billing. tiktoken encodings are OpenAI oriented. Use Anthropic count_tokens for Claude commits.
Is it OK to approximate Claude with tiktoken offline?: Only as a rough sketch with large margin, and never as invoice truth. Prefer Anthropic’s count API.
Why do GPT, Claude, and Gemini still differ if I pick the right OpenAI encoding?: Because Claude and Gemini are not those encodings. Right OpenAI encoding only fixes the OpenAI side.
Are TokenCalculator OpenAI counts Exact?: For supported o200k / cl100k style paths running in browser on the pasted text, yes. Wrappers you omit still undercount production.
Should I compare $/1M without comparing encodings?: No. Encoding choice changes token volume for the same prompt. Compare end to end cost with Exact counts and catalog rates.
Best way to compare providers after fixing OpenAI encoding?: Paste one golden prompt into /tools/tokenizer-comparison, verify Approx providers officially, then use the cost calculator.
Where do I learn the labeling policy?: Read the exact vs approximate tokens guide.

Try it in TokenCalculator

Paste the same string and compare Exact o200k and cl100k style counts in tokenizer comparison.

Open tokenizer comparison · Open tokenizer · Estimate API spend · All guides

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.