Is a token the same as a word?

No. English often averages about 0.75 words per token, but that is only a rough rule of thumb. Code, URLs, and non-English text can diverge sharply from that average.

Why do APIs charge per token?

Models process subword units during training and inference. Billing per token tracks that work better than billing per word, which would ignore how different text types compress into pieces.

How do I count tokens?

Paste your prompt into TokenCALC’s tokenizer. Use Exact labels for supported OpenAI encodings and Approx labels elsewhere, then confirm critical counts with provider tools or count APIs before large commits.

Do all providers use the same tokens?

No. Each major provider uses its own tokenizer vocabulary, so counts can differ on identical text. Always count on the model you will actually call.

How many tokens does a typical chat message use?

A short one or two sentence message is often on the order of tens of tokens. Long pasted documents can reach thousands. There is no universal number without tokenizing the actual string.

Home / Guides / What is a token in AI?

What is a token in AI?

A plain-language definition of AI tokens, why APIs bill in tokens instead of words, how counting works, and how to estimate cost before you call a model.

A token is a chunk of text the model reads

Large language models do not bill by word or character. They split text into tokens: short pieces that may be whole words, parts of words, spaces, or punctuation. The model reads and generates those pieces, and providers charge for how many pieces move through the API.

For common English text, OpenAI’s public rule of thumb is about four characters per token, or about three quarters of a word per token. That is a planning heuristic. Code, URLs, markup, and many non-English scripts often use more tokens for the same human-readable length.

Why APIs bill in tokens

Tokens are the unit models were trained to process. Subword tokenization lets a fixed vocabulary handle rare words, typos, and multiple languages without exploding the dictionary size. Billing follows that same unit so price tracks actual model work more closely than a naive word count.

Input tokens cover what you send (system instructions, tools, history, user text, retrieved context). Output tokens cover what the model generates. Output is usually priced higher. Knowing both counts is the only reliable way to estimate spend before you ship.

Tokens vs words vs characters

Words are for humans. Characters are a rough length signal. Tokens are what APIs price. A Microsoft Word word count is not an invoice. Convert words or characters to tokens only for early ballparks, then count real prompts with a tokenizer.

See the tokens vs words guide for conversion formulas, worked examples, and failure modes. Use the Tokens ↔ words tool when you only have a word estimate, and the main tokenizer when you have the actual text.

Why counts differ by provider

OpenAI, Anthropic, Google, and others use different vocabularies. The same paragraph can become a different number of tokens on Claude than on GPT. Never assume an OpenAI tokenizer is Exact for Claude or Gemini.

TokenCALC labels OpenAI browser encodings Exact when we can run a matching tokenizer locally. Other providers are labeled Approx when no official browser tokenizer is available. That honesty matters more than a false sense of precision.

How tokens affect context windows

Every model has a maximum context window measured in tokens. System prompts, tools, chat history, user input, and planned output all share that budget. If you only measure the visible user message, you will undercount and risk overflow.

TokenCALC shows a context meter so you can see when a prompt is getting tight. Read the context windows guide before you assume a large document fits because “it is only 20 pages.”

From tokens to cost

Cost is roughly (input_tokens × input_rate + output_tokens × output_rate) / 1,000,000, then scaled by request volume. Cache hits, batch discounts, and long-context tiers can change the effective rate when the catalog publishes them.

Paste one representative prompt into the cost calculator, set output size and monthly users × messages, and compare models on the same workload. Do not invent live prices in a spreadsheet when curated model pages already carry lastVerified rates.

How to count tokens in TokenCALC

Open the tokenizer, pick a model, and paste your prompt. For OpenAI models on supported encodings, the count is Exact in your browser. For other providers, treat the number as Approx planning guidance and confirm critical budgets with the provider’s count APIs or invoices.

Your text stays on your device. TokenCALC does not upload prompts to our servers for counting.

Common mistakes

These mistakes show up constantly when teams first adopt LLMs.

Treating word count as the API bill
Ignoring output tokens when forecasting cost
Using an OpenAI tokenizer to budget Claude or Gemini
Forgetting system, tool, and history tokens in the context window
Shipping on heuristics without checking Exact or provider counts

Best practices

Count real prompts whenever they exist. Keep one golden example for model comparisons. Label every estimate Exact or Approx. Recheck monthly forecasts against invoices after traffic is real.

When you only have a document length, start with tokens vs words, then move to the tokenizer and cost calculator before you commit spend.

Real-world scenarios

A customer support bot might send a 400 token system prompt, 200 tokens of retrieved FAQ chunks, and a 50 token user question on every ticket. If you only count the user message, you miss most of the input bill. A coding assistant might tokenize very differently from a marketing email because brackets, indentation, and identifiers split into more pieces than plain prose.

Document summarization apps often stuff entire PDFs into context. A 30 page report might look like 15,000 words in Word, but tokenization plus system instructions can push you toward context limits faster than a word count suggests. Agents that loop through tool calls accumulate history tokens across turns, so a single user question can balloon into thousands of tokens over a session.

Multilingual products face another twist. The same sentence in English, Spanish, or Japanese can produce different token counts on the same model. Budget per locale when traffic is split across languages rather than assuming one global average.

Step-by-step in TokenCALC

Open the tokenizer from the main navigation or the link in this guide. Select the model you plan to call in production, not a random default. Paste the full text you intend to send: system instructions, user message, and any static context blocks you reuse.

Read the token count and note whether the badge says Exact or Approx. Check the context meter against the model window. If you are above roughly 85 percent of capacity, plan to shorten, summarize, or switch to a larger window model before you optimize price.

Switch to the cost calculator with the same model selected. Set expected output length and monthly volume using users times messages per day. Compare at least two models on identical input so tokenizer and price differences show up together.

Related concepts

Tokenization explains how text becomes those countable pieces. The tokens vs words guide covers conversion heuristics when you only have a document length. Context windows explains how input and output share one budget. Exact vs Approx explains why some counts in TokenCALC are stronger than others.

Provider bridge guides for OpenAI, Claude, and Gemini walk through family-specific counting paths after your in-browser estimate. The LLM API pricing guide ties token counts to monthly spend, cache, and batch levers.

Expert notes

Product teams sometimes publish “max words” limits in UI copy because users understand words. Engineering should still enforce token limits internally. A word cap that works in English demos can fail in production when users paste JSON, stack traces, or mixed language content.

Fine tuning and distillation do not change how inference billing works. You still pay per token at call time. Training costs are separate. When comparing hosted models to self hosted open weights, tokenizer differences and hardware efficiency matter, but API list prices remain token based for most commercial endpoints.

Logging token usage per request in production helps you reconcile calculator forecasts with invoices. Even rough per feature logs beat guessing from a single demo prompt.

Frequently asked questions

What is a token in AI?: A token is a chunk of text a language model reads or generates. It may be a word, part of a word, space, or punctuation. Providers count these pieces on every API call and bill accordingly.
Is a token the same as a word?: No. English often averages about 0.75 words per token, but that is only a rough rule of thumb. Code, URLs, and non-English text can diverge sharply from that average.
Why do APIs charge per token?: Models process subword units during training and inference. Billing per token tracks that work better than billing per word, which would ignore how different text types compress into pieces.
How do I count tokens?: Paste your prompt into TokenCALC’s tokenizer. Use Exact labels for supported OpenAI encodings and Approx labels elsewhere, then confirm critical counts with provider tools or count APIs before large commits.
Do all providers use the same tokens?: No. Each major provider uses its own tokenizer vocabulary, so counts can differ on identical text. Always count on the model you will actually call.
How many tokens does a typical chat message use?: A short one or two sentence message is often on the order of tens of tokens. Long pasted documents can reach thousands. There is no universal number without tokenizing the actual string.

Try it in TokenCALC

Paste any prompt into the tokenizer to see how many tokens it uses for your model.

Open tokenizer · Cost calculator · Exact vs Approx · All guides