Home / Guides / Why code and emoji use more tokens

Why code and emoji use more tokens

Why JSON, source code, and emoji inflate token counts faster than English prose, and how Exact OpenAI chip maps make the density visible.

The short answer

Tokenizers are trained heavily on natural language. English prose often lands in larger pieces. Code and JSON are dense with punctuation and identifiers that become many short tokens. Emoji are often multi-codepoint sequences that split into several pieces when the vocabulary lacks a single glyph entry.

English vs JSON on the same meaning

Humans often see JSON as shorter than prose. Tokenizers often see more pieces because braces, quotes, colons, and commas are frequent standalone tokens, and keys split into subwords. Pretty-printing adds newline and indent chips. Paste both forms into the token visualizer under Exact o200k or cl100k to see the explosion.

Why source code is expensive per screen inch

Operators, identifiers, indentation, rare API names, and mixed languages (SQL in strings, HTML in templates) all raise chip density. Chars per token usually drops versus prose. Screen height is a poor proxy for billable length.

Why one emoji can be many tokens

Skin tones, variation selectors, ZWJ sequences, and flag pairs mean one on-screen glyph can be several codepoints. Without a matching vocabulary entry, you pay for a stack of chips.

How visualization helps you cut cost

Paste the expensive prompt, identify dense regions, rewrite one region at a time, confirm the chip count dropped under the same Exact encoding, then price with catalog rates in the cost calculator. Never invent live $/1M figures.

Common mistakes

Avoid these density mistakes.

  • Judging cost by character length or screen height
  • Pretty-printing huge JSON into every prompt for readability
  • Sprinkling emoji in system prompts without measuring
  • Using English word heuristics on code-heavy workloads
  • Trusting fake Claude or Gemini chip UIs for code bots

Frequently asked questions

Why does code use more tokens than it looks?
Punctuation, identifiers, and indentation become many short subword pieces. Screen length is a poor proxy for token length.
Why does one emoji use multiple tokens?
Multi-codepoint sequences often lack a single vocabulary entry, so the tokenizer emits several pieces.
Why do spaces and punctuation become their own tokens?
They are frequent units in training data. Tokenizers keep them as pieces instead of discarding them.
Why did my JSON prompt explode in tokens?
Keys, quotes, braces, and commas are expensive per character. Duplicated schemas multiply the tax. Visualize to confirm.
How does visualization help reduce prompt cost?
It localizes dense regions so you can rewrite them, then re-price with catalog rates.
Do o200k and cl100k disagree on code?
They can. Punctuation-dense text amplifies encoding differences. Check both in the visualizer.
Should I minify JSON in prompts?
Often yes for cost if the model still follows the schema. Measure before and after with Exact chips.
Is emoji cost the same on Claude and Gemini?
Not necessarily. Vocabularies differ. Compare counts carefully; do not trust fake Claude/Gemini chip UIs.
How do I teach my team this visually?
Live-demo prose vs JSON vs emoji in /tools/token-visualizer during onboarding.
Where do I price the cheaper prompt?
Use the cost calculator with TokenCalculator catalog rates for the model you call. Do not invent live prices.

Try it in TokenCalculator

Paste a short English sentence, then the same idea as JSON, then an emoji line. Watch Exact OpenAI chips multiply.

Open token visualizer · Open tokenizer · Estimate API spend · All guides

Related tools

Token visualizer · Tokenizer comparison · Token counter · Tokens ↔ words

Related guides

Token visualizer: see colored token pieces · How LLMs split text into tokens · tiktoken o200k / cl100k · Exact vs fake boundaries

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.