Home / Guides / How LLMs split text into tokens
How LLMs split text into tokens
Plain-language walkthrough of subword tokenization, BPE-style merges, whitespace quirks, and why GPT, Claude, and Gemini disagree on the same sentence.
The short answer
LLMs read tokens, not human words. A token might be a whole word, a fragment, a space plus a word, punctuation, or part of an emoji sequence. The split is deterministic for a given tokenizer and UTF-8 string. Change the vocabulary and the piece list changes.
See the split in the token visualizer. Count with the tokenizer. Compare families in tokenizer comparison.
Why models use subwords
Whole-word vocabularies fail on rare names and typos. Character-only sequences become very long. Subword methods (often BPE and relatives) keep a manageable vocabulary while representing almost any string by merging frequent pairs from training data.
Whitespace, punctuation, and invisible pieces
Leading spaces often attach to the next word. Newlines and tabs can be their own pieces. Punctuation frequently stands alone, which is why JSON explodes. Visualizers show spaces as · so you stop ignoring them.
GPT Exact vs Claude and Gemini honesty
TokenCalculator runs Exact encode/decode chip maps for o200k_base and cl100k_base. Claude and Gemini are not given fake colored boundaries. Use Anthropic count_tokens and Google countTokens for production truth. Do not treat tiktoken as a Claude oracle.
Content type changes the split
English prose often uses larger pieces. Code, JSON, URLs, emoji, and CJK are denser. Always measure on the target encoding or provider count API before you budget.
Common mistakes
Avoid these tokenization mistakes.
- Assuming tokens equal words
- Assuming one tokenizer rules every provider
- Trusting fake Claude or Gemini boundary UIs
- Ignoring whitespace and punctuation pieces
- Forgetting chat wrappers and tool schemas
- Hard-coding cl100k after moving to o200k models
Frequently asked questions
- How do LLMs split text into tokens?
- They map UTF-8 onto a fixed vocabulary using subword algorithms. Common fragments become single tokens; rare strings split into smaller pieces.
- What is a token visualizer?
- A UI that shows piece boundaries as colored chips plus a count. TokenCalculator’s tool is at /tools/token-visualizer.
- Why do spaces and punctuation become their own tokens?
- The learned vocabulary treats them as frequent units. Spaces often attach to words; braces and quotes often stand alone.
- Why does one emoji use multiple tokens?
- Many emoji are multi-codepoint sequences. Without a single vocabulary entry for the whole glyph, several pieces appear.
- Can I see token IDs in the browser?
- Yes on Exact OpenAI paths in the visualizer.
- Is the visualization Exact for Claude and Gemini?
- No on TokenCalculator. We do not invent those boundaries. Use provider count APIs.
- What is the difference between o200k and cl100k visualization?
- Different OpenAI encodings with different vocabularies. Same text, different chips and ids.
- How does visualization help reduce prompt cost?
- It shows dense substrings so you can rewrite them, then re-price with catalog rates in the cost calculator.
- Do chat templates add tokens beyond my paste?
- Yes. System roles, tool schemas, and formatting markers can add pieces you never typed in the demo box.
- Does four characters per token explain the split?
- No. That is an English planning heuristic only. Real splits come from the vocabulary.
Try it in TokenCalculator
Paste one sentence into the token visualizer and watch Exact OpenAI chips for o200k_base or cl100k_base.
Open token visualizer · Open tokenizer · Estimate API spend · All guides
Related tools
Token visualizer · Tokenizer comparison · Token counter · Tokens ↔ words
Related guides
Token visualizer: see colored token pieces · tiktoken o200k / cl100k · Code and emoji tokens · Exact vs fake boundaries
Sources and references
Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.