Home / Guides / RAG pipeline cost breakdown

RAG pipeline cost breakdown

A stage-by-stage map of retrieval-augmented generation cost: embedding the corpus, retrieval, and generating answers, so you can see which line dominates.

Why breakdown beats a single “RAG price”

Pipelines do not have one price. They have stages with different rates, different frequencies, and different failure modes.

A useful RAG pipeline cost breakdown separates the index path (chunk, embed, store, refresh), the retrieve path (query embed, search, optional rerank), and the generate path (system + question + retrieved chunks + completion). TokenCalculator’s RAG cost calculator models the token-billed stages with catalog generation rates and an editable embed $/1M. Vector DB fees and third-party rerankers stay on your infra spreadsheet.

Stage 1: Index and embed the corpus

You pay to turn documents into embedding vectors (and again when you re-embed). Cost ≈ corpus tokens × embed price per million × refresh frequency. Measure tokens actually embedded after cleaning, how often you fully re-embed vs incremental updates, and whether overlap windows double-count tokens.

Index share rises when traffic is low, the corpus is huge, or you re-embed the whole store on every tiny edit. For many chat products with steady query volume, index is a small monthly slice. Model it once, amortize it, then focus on generation unless refreshes are aggressive.

Stage 2: Retrieve (and optional rerank)

If you embed the user question, that call is usually a few dozen to a few hundred tokens and often tiny next to generation at scale. Still include it for honesty at very high QPS. Vector DB storage and QPS are real money outside the catalog. A reranker can improve precision so you can lower top-k and cut generation input enough to pay for itself.

Stage 3: Generate with the retrieval pack

Generation input ≈ system tokens + question tokens + (chunk tokens × top-k). Retrieved context is usually the largest slice. Completions bill at output rates; cap max_tokens to the answer format you need. Frontier generation on every query maximizes the quality ceiling and the bill. Stable system prompts may qualify for cached input pricing; that does not shrink per-query evidence unless those chunks are also in the cached prefix.

From per-query to monthly

Index monthly = (corpus tokens / 1M × embed $/1M) × re-embeds per month. Query embed monthly = queries × (query embed tokens / 1M × embed $/1M). Generation monthly = queries × generation cost for the full pack. Token API monthly ≈ those three lines; add vector DB and rerank outside the tool.

Per-query “what does RAG cost?” for product decisions is usually generation plus query embed, with index amortized when you need a blended unit cost. Check fit when packs grow, and use catalog rates rather than invented unit prices.

Common mistakes

Avoid these breakdown mistakes.

  • Collapsing all spend into “LLM API” without stage labels
  • Ignoring re-embed cadence when the corpus churns
  • Counting chunk count instead of retrieved tokens
  • Leaving rerank and vector DB out of stakeholder reports
  • Optimizing embed model while generation pack stays oversized
  • Skipping output caps in the generation stage

Frequently asked questions

What are the cost components of a RAG system?
Index embeddings, optional query embeddings, generation input and output, plus infra (vector DB, rerank, hosting). The RAG cost calculator covers the token API stages.
What is a RAG pipeline cost breakdown?
A stage-by-stage view of those components so you can see which line dominates per query and per month, not a single blended marketing number.
How much does it cost to embed a knowledge base?
Corpus tokens / 1M × your embed $/1M, multiplied by how often you re-embed. Use your provider’s rate in the tool’s embed field.
Is embedding cost even worth modeling?
Yes, once, with amortization. For busy query paths, generation usually dwarfs it. For huge corpora with rare queries or constant full re-embeds, index can matter a lot.
Why is generation more expensive than embedding in RAG?
Different rates and different frequency: embeddings are cheap per token and mostly batch; generation re-bills large retrieved packs every query.
How many tokens of context does a typical RAG query use?
Whatever you configured: system + question + retrieved pack. There is no industry-standard default. Measure your pack.
Does higher top-k always improve answers?
No. Past a quality sweet spot you pay more input tokens and can add noise. Test quality vs cost.
Should I use a frontier model for RAG generation?
Only if evals justify it. Many products use a mid-tier or mini generation model with strong retrieval. Compare catalog models in the calculator.
Mini model + good retrieval vs Sonnet on every query?
Run both in /tools/rag-cost with the same pack shape, then decide with quality metrics, not vibes.
How do I reduce costs once I see the breakdown?
Attack the dominant line: usually shrink retrieval or generation model rate, cache stable system text, and stop wasteful full re-embeds.

Try it in TokenCalculator

Split corpus embed, query embed, and generation lines, then scale by monthly queries.

Open RAG cost calculator · Open cost calculator · Estimate API spend · All guides

Related tools

RAG cost calculator · Embedding cost · Context window · Generation cost

Related guides

RAG cost calculator: estimate pipeline spend · Reduce RAG costs · Top-k and chunk cost · Embedding vs generation · Context for RAG

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.