Why does your RAG return wrong answers?
Nine times in ten the model never received the passage that held the answer.
Answered in short
5 things that decide this
- 01Most wrong RAG answers are retrieval failures, which means the correct passage never reached the model at all.
- 02You can separate the two causes in one test: paste the correct passage into the prompt by hand, and if the answer comes out right, retrieval is at fault.
- 03Chunking splits the answer away from the question it answers, which is the most common cause we find in production systems.
- 04Filters quietly shrink the candidate pool, so a search that matches few records can return almost nothing while looking like it worked.
- 05Swapping in a stronger model does not fix a retrieval problem, and it makes the wrong answers more convincing.
Find out whether retrieval or the model is at fault
Run this before changing anything. Take a question your system answers badly. Find the passage that holds the correct answer yourself. Paste it into the prompt directly and ask again.
A right answer means retrieval failed. The model was always capable, and it was working from material that did not contain the answer. Every hour spent on prompt wording would have been wasted.
A wrong answer even with the passage in hand points at the model or the instructions. That case is real, and it is much rarer than teams assume when they start debugging.
- Do this test on ten bad answers before touching the architecture. The split tells you where the work is.
Four causes, in the order we usually find them
Chunking is the usual culprit. Documents get cut into fixed-size pieces, and the cut lands between the question and its answer. A policy document that explains a rule in one paragraph and its exception in the next will answer half of every query.
Filters are the quiet one. Restrict a search by department, date or customer and the pool of candidates shrinks fast. A filter matching one record in ten can leave you with almost nothing to rank, and the feature looks broken when the data is fine.
Stale indexes cause the errors that damage trust most. The document was updated, the index was not, and your system now cites last year's policy with total confidence.
Missing content is the honest one. Sometimes the answer is not in the corpus. A system that says so is working correctly, and teams often mistake that for a bug.
- Your engineers will know the fixes for each of these. The value is in knowing which one you have.
- Question askedPhrased nothing like the document.
- Filters appliedCandidate pool shrinks quietly.
- Chunks retrievedCut between question and answer.
- RankingRight passage sits below the cutoff.
- Model answersConfidently, from what it was given.
Only the last step involves the model. Four earlier steps decide what it ever sees.
Related questions
01How do we know if chunking is the problem?
Look at the chunks your system retrieved for a failed question, and read them as a person would. If the answer is split across two of them, or the retrieved piece lacks the heading that gives it meaning, chunking is your issue. This takes minutes and teams skip it constantly.
02Should we re-embed everything when we change the model?
Yes, because embeddings from different models are not comparable. Mixing them in one index produces rankings that look plausible and mean nothing. Budget the re-index as part of any embedding change rather than discovering it afterwards.
03Can we just show users the source passage?
Showing sources is worth doing and it does not fix accuracy. It shifts checking onto your users, which helps when they are experts and hurts when they are not. Treat citations as a safety net under a system that retrieves correctly, never as a substitute for one.
