RAG vs fine-tuning
The two-way question is out of date. Long context is the third option, and it is the one most small projects should try first.
The short answer
Choose retrieval when answers must reflect documents that change, choose fine-tuning when you need a consistent format or tone rather than new facts, and try long context first when the whole corpus fits in a single prompt.
Most teams asking this question want current facts, so retrieval is the answer more often than not. Fine-tuning is chosen too early and too often, usually to fix a problem a better prompt would have solved.
They also combine. A lightly tuned model that always returns the same JSON shape, answering from retrieved passages, is a normal production setup.
The three options side by side
Compared on the things that change the decision, rather than on feature counts.
| Dimension | Retrieval (RAG) | Fine-tuning | Long context |
|---|---|---|---|
| What it changes | The facts available at answer time | How the model behaves and formats | Nothing. You paste the sources in |
| Updating content | Edit the document, live at once | Retrain on a new dataset | Change the file you send |
| Setup work | Ingestion, chunking, index, evals | Labelled examples and a training run | Almost none |
| Cost driver | Storage and retrieval per query | Training runs, then cheaper prompts | Input tokens on every call |
| Citations | Natural. You already have the passage | Not available. Facts are in the weights | Possible, if you ask for quotes |
| Access control | Filter per user inside the query | Impossible. Training data is baked in | All or nothing per request |
| Main failure | Retrieves the wrong or stale passage | Learns your format, still invents facts | Accuracy drops as the prompt grows |
| Fits best | Corpus that changes, or per-user access | Fixed output shape or classification | Small, stable, shared document set |
- QuestionArrives from a user
- RetrievalFetches passages. Facts enter here
- PromptLong context pastes sources here
- ModelFine-tuning changed these weights
- AnswerCited only if a passage was fetched
Three different points in one pipeline. That is why they combine rather than compete.
Retrieval (RAG)
Where it wins
- An edited document is reflected in the very next answer, with no retraining.
- Every answer can carry the passage it came from, which is what auditors ask for.
- Access rules can run per user, so one index safely serves many customers.
- Works with any model, so you can switch providers without redoing the work.
Where it hurts
- It is a system, not a setting: ingestion, chunking, indexing and evals all have to be built and run.
- Answer quality is capped by retrieval. A passage that is never fetched cannot be used.
- Questions needing a whole document, such as summarising a contract, suit chunked retrieval badly.
- Filtering on approximate indexes is subtle and gets built wrong. See the caution below.
Fine-tuning
Where it wins
- Reliable output shape. A tuned model returns your JSON schema far more consistently.
- Teaches tone, house style, or a classification scheme that resists prompt instructions.
- Shorter prompts once trained, because the instruction no longer travels with each request.
- A smaller tuned model can match a bigger one on one narrow task.
Where it hurts
- It does not teach facts reliably. A tuned model still invents, now in your house style.
- Every content change means another training run on a fresh dataset.
- No per-user permissions. Anything in the training data reaches every user.
- You need labelled examples, and producing enough good ones is the real cost.
How to choose
- Choose retrieval if your documents change, if answers need a citation, or if different users may see different things.
- Choose fine-tuning if the complaint is about shape, tone or a stubborn classification, and the facts are already correct.
- Choose long context if the whole corpus fits in one prompt and every user may read all of it.
- Choose both when you need a fixed output format over facts that keep moving.
- Choose neither if the real problem is that nobody has written the documents down. No method retrieves what does not exist.
Which one fits your situation?
Four questions about your data, not your preference.
How often does the content change?
How big is the document set?
Can every user read everything?
What is actually wrong today?
Every outcome
- Retrieval (RAG)
- Your facts move, or different users are allowed to see different things. Retrieval handles both, and gives you a citation for free.
- Fine-tuning
- Your facts are stable and the complaint is about behaviour: inconsistent format, wrong tone, a classification that will not stick.
- Long context
- A small, stable set of documents that everyone may read. Send them with the question and skip the infrastructure entirely.
01Can you use RAG and fine-tuning together?
Yes, and it is a common production setup. Fine-tune for the output shape or tone, then let retrieval supply the facts at query time. Keeping the two jobs separate is what makes the system debuggable when an answer is wrong.
02Has long context made RAG obsolete?
No, though it has taken the small end of the market. Anthropic's own documentation warns that as token count grows, accuracy and recall degrade, a phenomenon it calls context rot. Long context also gives you no per-user access control, so a multi-tenant product still needs retrieval.
03How do we migrate from fine-tuning to RAG?
Keep the tuned model and put retrieval in front of it. The tuned behaviour still applies, while facts now come from fetched passages rather than from the weights. Build a scored question set before you switch, or you will have no way to prove the change helped.
04Which is cheaper?
Cost depends on how often content changes and how many questions you serve, so the honest comparison is per workload rather than in general. Storage and per-query search drive the retrieval bill. Fine-tuning pays up front for training runs and labelled data, then less per call. Long context pays input tokens on every single request.
