RAG vs long context vs fine-tuning
These are not three ways to do one job. They fix different faults, and the fastest way to choose is to name what is going wrong today.
The short answer
Try long context first when every document fits in one prompt, move to retrieval when the corpus is too large or users must see different things, and fine-tune only when the complaint is about format or tone rather than facts.
Start with the symptom. Wrong facts point at retrieval or context. Wrong shape, wrong tone or a stubborn category points at fine-tuning. Nothing points at fine-tuning to teach the model new information, which is the most common and most expensive misdiagnosis.
Order matters too. Long context takes an afternoon, retrieval takes weeks, and fine-tuning needs labelled data you probably have not built yet.
Three options, side by side
Compared on the decisions that change the outcome, checked against vendor documentation in August 2026.
| Dimension | Long context | Retrieval (RAG) | Fine-tuning |
|---|---|---|---|
| Fixes | Missing facts, small corpus | Missing facts, any size corpus | Wrong format or tone |
| Effort to try | An afternoon | Weeks of engineering | Labelled data, then training |
| Updating content | Change the file you send | Edit the document, live at once | Retrain the model |
| Per-user permissions | All or nothing per request | Filter inside the query | Impossible once trained |
| Citations | Possible if you ask for quotes | Natural, you have the passage | Not available |
| Cost driver | Input tokens on every call | Storage plus search per query | Training runs, then cheaper calls |
| Main failure | Accuracy falls as the prompt grows | Fetches the wrong passage | Still invents, now in your style |
| Try it when | The whole corpus is small and shared | It is large, changing or restricted | Facts are right, output is not |
- The complaintWhat users actually report
- Wrong factsContext, then retrieval
- Wrong shapePrompt first, then tuning
- Too much dataRetrieval, with filters
- MeasureScored examples, or you are guessing
Teams that skip the first box spend months building the wrong one of the middle three.
Long context
Where it wins
- Almost no engineering. Paste the documents in with the question and see whether the answer is good enough.
- No pipeline to maintain, which means nothing to break at two in the morning.
- The model sees whole documents, so questions needing the full contract work better than with chunked retrieval.
- It is the cheapest way to find out whether the idea is worth more investment.
Where it hurts
- Accuracy and recall degrade as the prompt grows. Anthropic names this effect context rot in its own documentation.
- You pay input tokens on every single request, so a busy feature gets expensive quickly.
- There is no per-user filtering. Everyone in that request sees everything you sent.
- It stops working once the corpus outgrows a prompt, and that day arrives without warning.
Retrieval and fine-tuning
Where each wins
- Retrieval scales to any corpus size, and an edited document shows up in the next answer with no retraining.
- Retrieval filters per user, so one index can serve many customers safely.
- Fine-tuning makes output shape reliable, which matters when another system must parse it.
- Fine-tuning can teach a house style or a classification scheme that resists prompt instructions.
Where each hurts
- Retrieval is a system to build and run: ingestion, chunking, indexing and evaluation all need owners.
- Retrieval quality caps answer quality. A passage never fetched may as well not exist.
- Fine-tuning does not reliably teach facts, and a tuned model still invents in your house style.
- Fine-tuning needs labelled examples, and producing enough good ones is the real cost people underestimate.
How to choose
- Choose long context if everything the model needs fits in one prompt and every user may read all of it.
- Choose retrieval if the corpus is large, changes often, or different users are allowed to see different parts.
- Choose retrieval if answers must carry a citation, since auditors and regulated users ask for the source.
- Choose fine-tuning if the facts are correct but the output shape, tone or category keeps drifting.
- Choose a combination once you are past the first version. Tuned formatting over retrieved facts is a normal production setup.
- Choose none of them if the documents do not exist. No technique retrieves knowledge that only lives in someone's head, and writing it down is the actual project.
Retrieval systems we have shipped
Questions teams ask when the answers are wrong
01Which should we try first?
Long context, in almost every case where the documents fit. It takes an afternoon and tells you whether the model can do the job at all when it has the right information. If quality is good but the corpus is too big to send every time, you have proved retrieval is worth building.
02Is a million-token window enough to skip retrieval?
Only for a small, stable, shared set of documents. Anthropic documents that accuracy and recall degrade as token count grows, an effect it calls context rot, so a huge window is not a free upgrade. Per-user permissions are the other blocker: a prompt cannot show different users different subsets of what you sent.
03Can we combine all three?
Yes, and mature systems often do. A lightly tuned model returns a consistent format, retrieval supplies the facts, and long context handles the occasional request that needs a whole document. Keep the jobs separate so that when an answer is wrong you can tell which part failed.
04How do we prove which option is better for us?
Write twenty to fifty real questions with known good answers before building anything. Run each option against that set and compare the scores. This is unglamorous and it is the only thing that turns the decision into evidence instead of opinion, especially when someone senior has a preference.
05Does fine-tuning ever teach facts?
It can shift what a model tends to say, which is not the same as making it reliable. The information ends up in the weights, so you cannot cite a source, update a fact without retraining, or hide anything from a particular user. For knowledge that changes or needs an audit trail, retrieval is the sound choice.
