Home / Guides / Embedding vs generation cost in RAG

Embedding vs generation cost in RAG

Why generation usually dwarfs embedding spend in retrieval-augmented generation, when index cost still matters, and how to compare the two lines with real rates.

Two different meters

Embedding and generation both cost tokens, but they are not the same bill. Embedding buys vectors for chunks (and maybe the query) at index build, refresh, and optional per-query time. Generation buys LLM input and output for the answer on every user question and re-bills retrieved context repeatedly.

Embedding vs generation cost in RAG is mostly a frequency story plus a rate story. Embed rates are often much lower per million tokens than LLM input rates. A chunk embedded once can be retrieved and re-billed as generation input on every query that selects it. The RAG cost calculator keeps embed $/1M editable and uses catalog rates for generation. Do not invent either number.

Why generation is usually more expensive than embedding

Retrieved chunks are input tokens on every call. System + question + chunk×top-k is often thousands of input tokens per query. Completions add a second generation line, often at higher $/1M than input. Frontier generation SKUs amplify the gap.

That is why ops channels sound like “my RAG bill is all LLM input from retrieved chunks.” The embed job finished offline; the LLM meter never sleeps.

When embedding cost is worth obsessing over

Model embedding when the corpus is very large relative to monthly queries, you fully re-embed on a short cadence, you embed heavy overlap, query embedding runs at extreme QPS, or you are choosing among embed providers and need index budget clarity.

Is embedding cost even worth modeling? Yes, amortize it. Then spend engineering time where the share chart points. For busy assistants, that is usually generation.

How much does it cost to embed a knowledge base?

Index cost ≈ (corpus tokens / 1,000,000) × embed $/1M × full re-embed count. Use tokens after cleaning, not PDF page counts. Paste embed $/1M from your provider into the calculator. TokenCalculator does not treat embedding prices as catalog LLM rates.

Amortized monthly index = index cost × refreshes per month. Blended per query = monthly index / queries when you need a unit cost story for finance.

Comparing the two lines in practice

Enter corpus tokens, embed $/1M, and re-embed cadence. Enter chunk tokens, top-k, system, question, output, and queries/month. Select the generation model from the catalog. Read monthly generation vs monthly index vs query embed. Optionally set cache hit rate for stable system prefixes.

If generation share is high, shrink the pack, try a cheaper gen model, and cap outputs. If index share is high, fix refresh policy and corpus scope before you argue about a small embed rate difference. Holding retrieval fixed, swapping only the generation SKU often moves the bill more than swapping embed models. A cheap generation line that overflows the window is not cheap: validate packs in the context window calculator.

Common mistakes

Avoid these embed-vs-gen mistakes.

  • Comparing embed $/1M to LLM $/1M without looking at monthly frequency
  • Ignoring retrieved context when claiming generation is only the answer tokens
  • Re-embedding the full corpus to fix a retrieval quality issue that top-k could address
  • Treating vector DB fees as embedding API cost (different line)
  • Inventing rates instead of catalog + provider quotes

Frequently asked questions

Why is generation more expensive than embedding in RAG?
Lower embed unit rates and mostly one-time (or rare) index jobs, versus LLM rates applied to large retrieved packs on every query. Output tokens add more generation spend.
What is embedding vs generation cost in RAG?
The comparison between index/query embedding spend and LLM input/output spend for the same pipeline and volume. Use /tools/rag-cost to see shares.
How much does it cost to embed a knowledge base?
Corpus tokens / 1M × your embed $/1M × how often you fully re-embed. Enter the provider rate in the tool.
Is embedding cost even worth modeling?
Yes. Amortize it. For high-traffic RAG, expect generation to dominate; still keep index honest for finance and refresh planning.
What are the cost components of a RAG system?
Index embeddings, optional query embeddings, generation input and output, plus infra.
My RAG bill is all LLM input from retrieved chunks. Does that mean embeddings failed?
No. It usually means retrieval packs are large or frequent relative to index work. Shrink chunk × k or generation rates.
Should I use a frontier model for RAG generation?
Only if quality requires it. Try stronger retrieval with a cheaper generation SKU first.
Does prompt caching help RAG?
It can reduce the cost of stable prefixes (often the system prompt). It does not erase per-query evidence tokens.
Are embedding rates taken from the TokenCalculator catalog?
No. You enter embed $/1M. Generation uses catalog rates for the selected model.
How do I reduce whichever line dominates?
Generation: pack shape, model tier, cache, output caps. Embedding: incremental updates, less overlap, fewer full refreshes.

Try it in TokenCalculator

Enter corpus tokens and embed $/1M beside your generation pack and monthly queries; read the share chart.

Open RAG cost calculator · Open cost calculator · Estimate API spend · All guides

Related tools

RAG cost calculator · Embedding cost · Context window · Generation cost

Related guides

RAG cost calculator: estimate pipeline spend · RAG pipeline breakdown · Reduce RAG costs · Top-k and chunk cost · Context for RAG

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.