Do I need RAG or a bigger context window?
The honest 2026 answer: long context handles more than it used to, and it still cannot be your library.
Answered in short
5 things that decide this
- 01Use a long context window when the material fits, rarely changes, and every reader is allowed to see all of it.
- 02Use retrieval when the corpus is bigger than the window, changes often, or when different users must see different documents.
- 03Claude Opus 5, Claude Sonnet 5 and several other models now carry a 1M-token context window at standard pricing, so the old cost argument against long context has weakened.
- 04Anthropic documents that accuracy and recall degrade as a context window fills, a effect its docs name context rot.
- 05Access control is the deciding factor most teams forget, because a context window has no permissions and a retrieval layer does.
Long context got good enough to change the question
Two years ago this was easy to answer. Windows were small, long inputs cost a lot, and retrieval was the only practical way to work over real document sets.
That has shifted. Anthropic now documents a 1M-token window across Claude Opus 5, Claude Sonnet 5 and several sibling models. Its docs state that 1M is the default, billed at standard pricing. No beta header, no premium tier.
So the cost objection has softened, and plenty of jobs that needed a retrieval pipeline in 2024 now fit in a single request. Anyone still saying long context is always too expensive is working from old information.
- A single request can also carry up to 600 images or PDF pages on those models, which covers a lot of real document review.
Three things a bigger window cannot do
Accuracy is not flat across a full window. Anthropic's own documentation says that as token count grows, accuracy and recall degrade, and it names the effect context rot. More context is not automatically better, and the vendor says so.
Permissions are the harder wall. A context window is one bucket with no access rules inside it. If your sales team may read one folder and your finance team another, retrieval gives you a place to enforce that. Stuffing everything into the prompt gives you a leak waiting to happen.
Freshness is the third. Documents that change daily have to be re-sent in full on every request, whereas a retrieval index updates the part that moved.
- Our position: if the material fits comfortably and everyone may see all of it, start with long context. It is simpler, and simpler systems break less.
- Does it fit?Comfortably inside the window, not at the edge.
- Who may read it?Different answers per user means retrieval.
- How often does it change?Daily churn favours an index.
- Is recall slipping?Long inputs lose detail as they fill.
- Pick the simpler oneLong context first when it genuinely fits.
Two yes answers on permissions and churn settle it. Retrieval wins.
Systems where the corpus decided the answer
Related questions
01Can I use both?
Yes, and the combination is often right. Retrieve a focused subset, then give the model room to reason across all of it in one long request. That gets permissions and freshness from the retrieval layer while avoiding the tiny-snippet problem that makes early RAG systems answer badly.
02Does a 1M-token window cost more per token?
Not on Claude models with a 1M window, where Anthropic documents 1M as the default billed at standard pricing. You still pay for every token you send, so a full window on every request is expensive by volume even at standard rates. Prompt caching changes what you pay for repeated prefixes, not whether they count.
03How do I know if context rot is hurting my answers?
Build a fixed set of questions with known correct answers, then run it at different context sizes. Quality that falls as you add material is the signal. Without that test you are guessing, and the degradation is gradual enough to miss by eye.
