Home / Guides / How to reduce RAG costs

How to reduce RAG costs

Practical levers to cut retrieval-augmented generation spend: retrieval shape, generation model, caching, output caps, and smarter re-embeds, checked against quality.

Start with a baseline, not a guess

Capture average chunk tokens, top-k, system tokens, question tokens, and output size. Enter monthly query volume. Pick the live generation model in the RAG cost calculator. Paste your real embed $/1M for index math. Note which line dominates (almost always generation for busy apps).

Do not invent unit prices. Use catalog generation rates and your embed provider’s published rate.

Shrink retrieved context (top-k and chunk size)

Retrieved tokens ≈ chunk tokens × top-k. That product is generation input on every query. Lower top-k and measure answer quality. Prefer smaller, focused chunks when that improves precision. Deduplicate overlapping chunks. Add a reranker if it lets you cut top-k without hurting evals.

Higher top-k does not always improve answers. Past a sweet spot you pay more and can distract the model. Also keep packs inside the window with context window budgeting for RAG.

Use a cheaper generation model when quality allows

Generation rates dominate most RAG bills. Freeze the retrieval pack in the calculator, swap generation models from the TokenCalculator catalog, compare per-query and monthly totals, and promote only candidates that pass the same quality suite.

Good retrieval often unlocks a smaller generation model. Blind frontier defaults are an expensive habit.

Cache, cap output, and stop wasteful re-embeds

If the provider supports prompt caching and your system block is identical across calls, cached prefix tokens can bill cheaper than fresh input. Caching does not remove per-query retrieved chunks unless those chunks are part of a stable cached prefix.

Cap output length to the answer format you need. Prefer incremental embedding updates over full corpus re-embeds on every typo fix. Fix context fit before you optimize price: an overflowing pack wastes money and quality.

A sensible order of operations

Baseline in /tools/rag-cost. Confirm context fit. Reduce retrieved tokens with quality gates. Try a cheaper generation SKU with the same pack. Enable caching for stable prefixes where supported. Tighten output caps. Clean up re-embed policy. Re-check monthly total and generation share. Change one variable at a time so you know what moved the bill.

Common mistakes

Avoid these cost-cutting mistakes.

  • Cutting embed quality first while top-k stays huge
  • Lowering top-k with no evals (silent quality regressions)
  • Assuming frontier generation is mandatory for RAG
  • Ignoring output caps
  • Counting infra savings while LLM input still dominates
  • Re-embedding daily “just in case”

Frequently asked questions

How do I reduce RAG costs?
Measure the pack, shrink retrieved tokens, pick the cheapest generation model that passes evals, cache stable system text, cap outputs, and avoid needless full re-embeds. Start at /tools/rag-cost.
Why is generation more expensive than embedding in RAG?
Retrieved context is re-billed as LLM input every query; embeddings are cheaper per token and mostly batch.
Does higher top-k always improve answers?
No. Test quality vs cost. Extra chunks can add noise and always add input tokens.
Should I use a frontier model for RAG generation?
Only when evals require it. Many stacks use a smaller generation model with strong retrieval.
Does prompt caching help RAG?
It can, for stable prefixes like the system prompt. Model hit rate with /tools/cache-savings.
My RAG bill is all LLM input from retrieved chunks. What should I change first?
Lower top-k or chunk size with evals, or switch generation model rates. That line is generation input, not embedding.
top-k=20 killed our margins. What now?
Baseline retrieved tokens, run quality at several lower k values, keep the lowest k that holds evals, and consider a reranker.
Anyone caching the system prompt in RAG?
Yes when the system block is identical call to call and the model supports cache pricing. Verify with catalog fields and the cache tool.
Mini model + good retrieval vs Sonnet on every query?
Compare both in the RAG cost calculator with the same pack, then ship the cheaper one that clears your quality bar.
What are the cost components of a RAG system?
Index, optional query embed, generation input/output, plus infra. See the RAG pipeline cost breakdown guide.

Try it in TokenCalculator

Baseline your pack in the RAG cost calculator, change one lever, and re-read monthly total and generation share.

Open RAG cost calculator · Open cost calculator · Estimate API spend · All guides

Related tools

RAG cost calculator · Embedding cost · Context window · Generation cost

Related guides

RAG cost calculator: estimate pipeline spend · RAG pipeline breakdown · Top-k and chunk cost · Embedding vs generation · Context for RAG

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.