Bad RAG Answers Start in the Ingest Pipeline
By the time someone edits the prompt, the failure already happened. It happened when a document was chunked, indexed or left stale weeks earlier.
The short version
5 things that decide this
- 01Most RAG failures are decided at ingestion, weeks before anyone types a question.
- 02Chunking, stale documents, missing metadata and contradicting sources are the four usual causes.
- 03A team that debugs the prompt for a retrieval failure will edit words that were never the problem.
- 04Source quality and document structure decide the output more than model choice does.
- 05An ESG research platform we built, Greenlight, treats ingestion as the product: 50 or more topics, each backed by 10 to 15 independent sources.
The prompt gets blamed for a retrieval failure
A RAG system gives a wrong answer, and the first move is almost always the same. Someone opens the prompt and starts rewording it. Add an instruction, tighten the phrasing, try a stricter system message. Ship it, test it, and the answer is still wrong.
That instinct makes sense. The prompt is the part everyone can see and edit in minutes. The ingest pipeline is upstream, unglamorous, and usually built once and forgotten. But a model can only answer from what retrieval hands it. If the right passage never reached the model, no amount of prompt engineering fixes that, because the fix has to happen earlier.
This is the companion problem to a question we get asked constantly: why does a RAG system return the wrong answer. The short answer is that the failure is usually retrieval wearing a model costume. This post is about where retrieval failures actually start, which is earlier than most teams look.
Four failures, and none of them live in the prompt
Chunking is the most common cause of a bad answer. Documents get sliced into fixed-size pieces so they fit a search index. The cut often lands between a rule and its exception. A policy explains the standard case in one paragraph and the exception in the next. Split them, and the system answers the easy half of every question. It misses the half that mattered.
Stale documents cause the errors that cost the most trust. A policy changes, and the source file gets updated. The index does not. The system keeps citing last year's rule with full confidence. Nobody notices until a decision made on that answer turns out to be wrong.
Missing metadata quietly breaks filtering and ranking. A passage with no date, no source type and no author is indistinguishable from any other passage at retrieval time. Ask the system to prefer recent, authoritative sources and it cannot, because nothing in the index tells it which passage is which.
Contradicting sources are the hardest of the four, because retrieval did its job correctly. Two documents disagree, both get retrieved, and the model has to pick one without being told which source to trust more. A pipeline with no source-quality signal leaves that choice to chance.
- 01Chunking that respects document structure, and not only a fixed size, keeps a rule and its exception in the same passage.
- 02A refresh schedule tied to how often a source actually changes, not a calendar default, stops stale citations.
- 03Metadata on source type, date and author lets the system rank a regulator's filing above a company's own claim.
What an ingest pipeline looks like when it is the product
We built Greenlight, an AI platform that scores how sustainable a company actually is, using GPT-4 and Perplexity Sonar-Pro with retrieval grounding. Every scan covers 50 or more ESG topics, and each topic is backed by 10 to 15 independent sources with live citations. That number is not a marketing figure. It is the ingest budget the product needed to be trustworthy.
ESG research is a contradicting-sources problem by nature. A company's own sustainability report says one thing. A regulator's filing says another. Greenlight's scoring weights AI judgment at 66 percent and data averages at 33 percent. It favors independent sources over a company's own claims, because those claims are the most likely to flatter rather than inform.
None of that is a prompt decision. It decides which sources enter the index, how they get tagged, and how conflicts between them get resolved. That happens before a question is ever asked, and it is the work that makes a wrong answer rare instead of routine.
Questions this raises
01Why does my RAG system return wrong answers even after I improve the prompt?
Because the prompt was rarely the cause. If retrieval never surfaced the passage with the correct answer, no wording change gives the model access to it. Test this directly: find the right passage yourself, paste it into the prompt, and ask again. A right answer there means retrieval failed, not the model.
02What is the best chunking strategy for RAG accuracy?
Chunk by document structure rather than a fixed character count, so a rule and its exception stay in the same passage. A chunk boundary that splits a clause from its condition is how a policy document produces confidently wrong answers on half its questions.
03How do I improve RAG accuracy without changing the model?
Audit ingestion first. Check chunk boundaries against real documents, confirm the index refreshes when a source changes, and add metadata for source type and date. Then decide how conflicting sources get ranked. Most accuracy gains live in that pipeline, not in a newer model.
Related
- Why your RAG returns wrong answers →The direct diagnostic: how to tell retrieval and model failures apart.
- Chunking →The term this post argues about, defined.
- RAG development →How we build the retrieval pipeline, and not only the prompt.
- Greenlight: AI ESG research platform →The ingest pipeline described in this post, in production.
