Home / Guides / Exact vs fake token boundaries
Exact vs fake token boundaries
Why colored token chips are Exact only for OpenAI o200k_base and cl100k_base in TokenCalculator, and why Claude or Gemini piece maps in random UIs are often fake.
The honesty rule
Exact colored boundaries require a matching tokenizer that can encode and decode in your browser. TokenCalculator meets that bar for o200k_base and cl100k_base. For Claude and Gemini, TokenCalculator refuses to invent piece boundaries. Approx counts may appear elsewhere for planning. Fake Exact Claude or Gemini chip maps will not.
What Exact means for chips
A real OpenAI-style tokenizer ran locally. encode produced vocabulary ids. Each id decoded to the piece text on the chip. The badge says Exact and the method names the encoding. Exact scopes to the pasted UTF-8 and that encoding, not to omitted wrappers.
What fake boundaries look like
Fake visualizers often run tiktoken and label chips Claude or Gemini, split on whitespace and color the spans, or hide heuristics behind a precise UI with no Exact/Approx badge. If a site cannot run Anthropic’s tokenizer offline and still paints Claude chips, treat the boundaries as fake.
What to use instead
Visualize GPT prompts with Exact chips. Compare cross-family counts in tokenizer comparison with Exact vs Approx labels. Before large Claude or Gemini commits, re-count with Anthropic count_tokens or Google countTokens on the full messages body. Price with the cost calculator and catalog rates. Do not invent live $/1M figures. Do not use tiktoken as a Claude oracle.
Why fake maps are worse than Approx counts
An Approx count with a clear label invites margin. A fake Exact map invites false confidence and bad finance models. TokenCalculator chooses visible limits over pretty lies.
Common mistakes
Avoid these honesty traps.
- Trusting unlabeled colored chips for Claude or Gemini
- Budgeting Anthropic spend from tiktoken piece counts
- Removing Approx badges to make a UI feel more precise
- Treating Exact OpenAI chips as invoice truth when wrappers were omitted
- Inventing live prices beside any count
- Mixing o200k and cl100k screenshots without labels
Frequently asked questions
- Is the visualization Exact for Claude and Gemini?
- No on TokenCalculator. We do not draw Claude or Gemini colored boundaries. Use Anthropic count_tokens and Google countTokens.
- What is a token visualizer?
- A tool that shows token piece boundaries and counts. TokenCalculator’s Exact maps cover OpenAI o200k_base and cl100k_base at /tools/token-visualizer.
- Can I see token IDs in the browser?
- Yes for Exact OpenAI encodings. Fake maps may show decorative spans without real vocabulary ids.
- Claude piece map in browser: real or fake?
- Unless the tool runs Anthropic’s tokenizer (or an officially blessed equivalent) and says so, treat painted Claude boundaries as fake.
- Is there a tiktoken visualizer that shows chip colors?
- Yes for OpenAI encodings at /tools/token-visualizer. tiktoken chips are not Claude chips.
- What is the difference between o200k and cl100k visualization?
- Two Exact OpenAI encodings with different vocabularies. Both can be honest and still disagree.
- What does Approx mean on TokenCalculator?
- A labeled heuristic count when a provider tokenizer is not available client-side. Useful for planning with margin, not a fake Exact map.
- How much can Approx miss by?
- Often modest on plain English; larger on code, URLs, and non-English text. Confirm with provider APIs before large commits.
- Do ChatML or tool wrappers change Exact pastes?
- Yes. Exact applies to what you pasted. Add wrappers to the paste or verify with provider tooling that accepts the full request.
- How does honesty help reduce prompt cost?
- You optimize against real piece density for GPT and verified counts for other providers, then price with catalog rates instead of fictional chip math.
Try it in TokenCalculator
Open the token visualizer on an OpenAI encoding for real chips, then use provider count APIs for Claude and Gemini.
Open token visualizer · Open tokenizer · Estimate API spend · All guides
Related tools
Token visualizer · Tokenizer comparison · Token counter · Tokens ↔ words
Related guides
Token visualizer: see colored token pieces · How LLMs split text into tokens · tiktoken o200k / cl100k · Code and emoji tokens
Sources and references
Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.