Context Window vs Cost Tradeoff
Overview
Bigger context windows tempt you to paste everything. Cost and latency scale with tokens — stuffing 200k tokens because you can often burns budget without improving answers. Retrieval, summarization, and selective @-mentions usually win.
When to Use
- Prompts growing past a few thousand tokens
- Monthly token bill climbing faster than traffic
- Model supports huge context and team wants "just include all docs"
- Skip when the entire corpus truly fits cheaply and static (rare)
Quick Reference
| Situation | Prefer |
|---|
| Large doc corpus | RAG / chunk retrieval |
| Long chat history | Summarize older turns |
| Repeat static instructions | Cache system prompt |
| One-off deep dive on 3 files | @-mention or attach only those |
| Need full contract text once | Large window for that request only |
Implementation
- Measure current tokens per request — input breakdown by section.
- Identify redundant context — duplicated docs, full thread, verbose tool dumps.
- Replace with retrieval or summary — keep only what changes the answer.
- A/B cost and quality — smaller context + RAG vs mega-prompt.
- Set soft caps — max input tokens per route with trim strategy.
- Monitor p95 tokens in observability — alerts on drift.
On This Portfolio
Assistant injects retrieved chunks with titles and URLs, not full pages. Copilot graph builds state per node instead of one mega-prompt. Cursor skills and @-files beat pasting the whole repo.
Common Mistakes
- Pasting entire knowledge bases because the model "supports it"
- Sending full tool JSON responses into the next turn without compression
- Optimizing context size without measuring retrieval quality drop
Related Skills
- RAG pipeline design
- Context engineering
- Caching strategy for LLM calls