Home / Guides / RAG top-k retrieved context cost

RAG top-k retrieved context cost

How top-k and chunk size set retrieved context tokens, drive generation input cost every query, and when raising k stops helping answers.

The simple formula ops forgets

For each generation call, retrieved tokens ≈ average chunk tokens × top-k (after dedup and caps). Those tokens sit in the prompt as evidence and are billed as generation input, not as embeddings. That is RAG top-k retrieved context cost in one line.

If chunks average 750 tokens and top-k is 16, you are near 12,000 input tokens of evidence alone, before system and question. At high monthly volume, that line dominates. Model it in the RAG cost calculator.

Chunk size is a cost and quality knob

Larger chunks keep more local context but burn more tokens per hit and make one irrelevant hit expensive. Smaller chunks allow finer retrieval and lower tokens per hit, with more risk of fragmented evidence if k is too low.

There is no universal best size. Pick a size for your corpus, then cap total evidence tokens, not only k. See context window budgeting for RAG.

Does higher top-k always improve answers?

No. Gains usually flatten. Extra chunks can add contradictory text, push important passages into the middle of a long pack, blow the context budget or long-context pricing tiers, and multiply input cost linearly with k for fixed chunk size.

Treat k as an experiment factor. Run the same eval set at several k values. Keep the lowest k that holds quality. If you need higher recall, try hybrid search or a reranker, then lower k again. Stuffing more of the corpus is not the same as better RAG.

Cost sensitivity: change k, hold everything else

In the RAG cost calculator, fix generation model, system, question, output, and volume. Set your average chunk tokens. Sweep top-k upward and watch retrieved tokens and monthly generation cost. You will usually see generation dominate and move almost in lockstep with chunk × k.

A reranker costs its own fee but can let you send top-5 instead of top-20 to the LLM. Fit and cost are separate checks: validate large packs in the context window calculator and watch long-context rate tiers in the catalog.

Common mistakes

Avoid these top-k and chunk mistakes.

  • Raising top-k to “be safe” with no evals
  • Budgeting chunk count instead of evidence tokens
  • Ignoring dedup so overlapping windows inflate k effectively
  • Comparing models on different k values
  • Forgetting history tokens in multi-turn RAG when sizing evidence

Frequently asked questions

What is RAG top-k retrieved context cost?
The generation input cost of the evidence pack, driven mainly by chunk tokens × top-k (plus how often you query). Modeled in /tools/rag-cost.
Does higher top-k always improve answers?
No. Quality often plateaus while cost keeps rising. Evaluate and cap.
How many tokens of context does a typical RAG query use?
System + question + retrieved pack (chunk × k), and history if present. Measure your stack; there is no single industry number.
Why is my RAG bill all LLM input from retrieved chunks?
Because those chunks are generation input every time. Shrink chunk × k or generation rates.
top-k=20 killed our margins. What should we try?
Sweep lower k with evals, shorten chunks, dedupe, add rerank, or switch to a cheaper generation SKU that still passes quality.
Is stuffing more chunks better than tight top-k?
Usually no for both cost and clarity. Cap evidence tokens and keep top-ranked chunks.
How do I reduce RAG costs related to top-k?
Lower k or chunk size with quality gates, rerank into a smaller k, cache system text, and cap outputs.
What are the cost components of a RAG system?
Index embeddings, optional query embed, generation input/output, infra. See the RAG pipeline cost breakdown guide.
Should evidence be capped in tokens or in chunk count?
Prefer a token cap. Chunk count lies when sizes vary.
How does this relate to context window budgeting?
Evidence is one slice of a shared window with system, history, and reserved output. See the context window budgeting for RAG guide.

Try it in TokenCalculator

Set chunk tokens and top-k, watch retrieved tokens and monthly generation spend move together.

Open RAG cost calculator · Open cost calculator · Estimate API spend · All guides

Related tools

RAG cost calculator · Embedding cost · Context window · Generation cost

Related guides

RAG cost calculator: estimate pipeline spend · RAG pipeline breakdown · Reduce RAG costs · Embedding vs generation · Context for RAG

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.