Context window budgeting for RAG

RAG fails when the window is treated as one blob

Retrieval augmented generation sounds simple: find relevant chunks, paste them into the prompt, ask the model to answer. The failure mode is also simple: the prompt grows until calls reject, truncate, or quietly drop the wrong pieces.

A context window is a shared budget. System instructions, retrieved evidence, chat history, and the model’s answer all compete. If you only cap how many chunks you retrieve, you still overflow when history grows or when you raise max_tokens for longer answers.

Budgeting means assigning hard token caps to each slice, measuring them with a real tokenizer, and enforcing those caps in code. TokenCalculator’s context window calculator and tokenizer help you set those numbers from production-shaped text, not from guesses.

For the basics of shared input/output windows, start with the context windows guide. For what breaks when you exceed the sum, see context window overflow.

RAG context window budgeting: system, history, and evidence slices

Most production RAG prompts use four slices. System holds role, policies, output format, and tool schemas: keep it small and stable, and version it. Evidence holds retrieved chunks, citations, and metadata: cap total tokens, not only chunk count. History holds prior user and assistant turns you keep: use a sliding window or summary, never unbounded. Output is reserved completion tokens: always subtract from the window first.

Some teams add a fifth slice for scratch (scratchpad, intermediate tool results). Treat it like evidence: it is still input and still counts.

Output reservation is not optional. If the window is W and you allow up to O completion tokens, the maximum input you may send is W minus O (minus any provider overhead your stack adds). Skipping that subtraction is a common cause of “it worked in staging with short answers.” Details: reserve output tokens.

A practical budgeting formula

Use this planning formula before you write retrieval code. Choose model window W from the TokenCalculator catalog for the SKU you will call. Choose max output O your product needs for the worst case answer format. Choose system budget S for the longest system prompt you ship (include schemas). Choose history budget H for the longest history you keep after summarization. Evidence budget E equals W minus O minus S minus H (leave a small safety margin).

Then enforce E in the retriever: stop adding chunks when tokenized evidence would exceed E. Prefer dropping low-ranked chunks over silently overflowing.

Check the full pack in the context window calculator. If you are tight, shrink H or S before you inflate W by jumping to a larger model. Larger windows can help, but they are not free of quality and pricing tradeoffs. See effective vs advertised context window and long-context pricing.

Designing the evidence slice

Cap tokens, not only chunk count. “Top 8 chunks” is not a budget if chunk sizes vary. One dense table can burn more tokens than three short paragraphs. Tokenize candidates, accumulate until E is reached, then stop.

Prefer top-k relevance over stuffing the window. Stuffing more of the corpus (for example dumping 200K tokens of RAG chunks) rarely improves answers once you pass a quality sweet spot. Lost-in-the-middle effects and distraction from noisy chunks can hurt more than they help. Measure answer quality at several evidence budgets instead of always maximizing fill.

Separate citeable text from metadata. IDs, scores, and long URLs consume tokens. Keep citation metadata compact. Put full documents in your store, not in every prompt.

Deduplicate overlapping chunks. Overlapping windows from the same section waste budget. Dedup by document section or embedding similarity before you fill E.

Designing the history slice

Chatty RAG products fail when every prior turn stays forever. Keep the last N turns only, with N derived from token budget H, not message count alone. Summarize older turns into a short memory block that still fits under H. Store full transcripts offline; inject only what the next answer needs.

Measure history the same way you measure evidence: tokenize the exact string you will send. Approximate word ratios are for planning only. See exact vs approximate tokens and tokenization.

Designing the system slice

System prompts drift. Someone adds another policy paragraph, then another JSON example, then a tool schema. Suddenly S has eaten the evidence budget.

Version system prompts. Diff them. Tokenize after every change. Prefer short, testable instructions over encyclopedic policy dumps. Format instructions that belong in few-shot examples can often move into a small, fixed template counted under S.

Cost is a second pass, not the first

After the pack fits, estimate spend with the cost calculator. Large evidence slices raise input tokens every call. Some models also cross long-context rate tiers. Fit first, then cost.

How token counts become bills: prompt cost. Catalog exploration for large prompts: long-context pricing. Do not invent unit prices; use verified catalog rates in TokenCalculator.

Example allocation on a 128K window

Numbers below are planning examples, not recommendations for every product. On a 128K window you might reserve 4,000 for structured answers with citations, 2,000 for policies plus a compact schema, 6,000 for recent turns or a rolling summary, 2,000 as a safety margin for provider overhead and growth, and about 114,000 for evidence filled by ranked chunks until the cap.

A support bot might invert that shape: smaller evidence, larger history. A single-shot document QA flow might zero out history and raise evidence. The method stays the same: assign caps, measure, enforce.

Common mistakes

Avoid these RAG budgeting mistakes.

• Budgeting chunk count instead of evidence tokens
• Leaving history uncapped in multi-turn RAG
• Raising max output without shrinking evidence
• Letting system prompts grow without remeasuring S
• Filling the entire window because you paid for it
• Using word heuristics as a production gate
• Switching to a huge window model without checking quality and long-context tiers

Frequently asked questions

What is context window budgeting?

Assigning explicit token caps to system, evidence, history, and output so the sum stays under the model window with margin.

Should I put my whole PDF in the prompt or use RAG?

Use RAG (or section selection) when only part of the document is relevant. Full paste burns the evidence budget, raises cost, and can hide the important passage in the middle of a long pack.

Why not put the whole knowledge base in the prompt?

Because the window is finite, cost scales with input tokens, and answer quality often peaks before the window is full. Retrieve what matters.

Is stuffing 200K of RAG chunks better than top-k retrieval?

Usually no. Cap evidence tokens, keep top-ranked chunks, and evaluate quality. Blind fill increases distraction and lost-in-the-middle risk without a guaranteed quality gain.

How do I pick the evidence budget?

Subtract reserved output, system, history, and a safety margin from the model window. Enforce the remainder in the retriever with token counts.

How many tokens should I reserve for the reply?

Size O from the worst-case answer format for that route (JSON schema, citations, long brief). Subtract O before you fill evidence. See the reserve output tokens guide.

Should history include tool traces?

If you send them to the model, yes. Tool traces often dominate history. Summarize or omit them when the next step does not need the raw payload.

How much headroom should I leave for tool results?

Treat tool results like evidence: they are input tokens. Keep a dedicated buffer inside H or E so a large payload cannot blow the pack. Cap and summarize raw tool dumps before they enter the prompt.

Should agents summarize history or hard-drop old turns?

Both work. Hard-drop (sliding window) is simpler and predictable. Summarization preserves more long-range facts if you keep the summary under H. Unbounded history is the failure mode either way.

Exact or approximate tokens for budgeting?

Use exact (or the best available tokenizer path) when setting hard caps. Approximate ratios are only for early sizing. See the exact vs approximate tokens guide.

How do I validate a budget change?

Rebuild a worst-case prompt pack (max system, max history, max evidence, max output), run it through the context window calculator, then load-test the API path for rejection and truncation behavior.

Where do I check document fit interactively?

Use the Will my document fit? Context window calculator guide and the live tool at /tools/context-window.

Try it in TokenCalculator

Measure each slice with real tokens, set caps, and verify the sum still fits before you tune retrieval quality.

Open context window calculator · Open tokenizer · Estimate API spend · All guides

Related tools

Context window calculator · RAG cost · Cost calculator · Long-context pricing

Related guides

Context windows and overflow · Will my document fit? Context window calculator · What happens when you exceed the context window · Why you must reserve output tokens · Effective vs advertised window

Sources and references

Official documentation used for definitions, counting methods, or rate cards. Always confirm critical budgets on the provider page.

• OpenAI API pricing
• Anthropic Claude pricing
• Google Gemini API pricing