Your Knowledge Base Is Not Ready for RAG
Most RAG failures get blamed on the model. Open the source documents and the real problem is usually sitting there: duplicates, outdated policies, and answers that only live in someone's head.
In short
5 things that decide this
- 01A RAG system answers from whatever it retrieves, so a corpus full of duplicates and dead pages produces confident, wrong answers even with a strong model.
- 02The most common failure is retrieving two documents that answer the same question differently, because one of them is three policy revisions out of date.
- 03Tribal knowledge, the fix that lives in a Slack thread or one engineer's memory, cannot be retrieved if it was never written down.
- 04Run a source audit before a build starts: count duplicate answers, flag undated documents, and list what only exists as spoken knowledge.
- 05Cleanup is ongoing work, not a one-time pass. A wiki that grows unmanaged again is a RAG system that degrades again.
The model gets blamed for a documents problem
A retrieval system is only as accurate as what it retrieves. Ask it a question, and it pulls a handful of documents, then writes an answer grounded in whatever came back. Suppose the wiki holds two conflicting versions of a refund policy. The system does not know which one is current. It just picks the passage that scored closest to the question and answers from that.
Teams usually notice this after launch, when a customer gets an answer that used to be true. The instinct is to blame the model or tune the retrieval settings. That rarely fixes it, because the model did its job correctly. It found a real document and reported what it said. The document was the problem.
Most internal knowledge bases were never built to be read by a retrieval system. They were built by people, over years, with no owner responsible for removing what stopped being true. Nobody deletes an old onboarding doc. They just write a new one next to it and hope people find the right one.
Four ways a wiki poisons its own answers
Duplicate content is the biggest one. Two pages answer the same question in different words, sometimes with a different answer, because support wrote one and product wrote the other. Nobody reconciled them. Retrieval has no way to know which is authoritative, so it returns both, and the model has to guess which passage to trust.
Stale policy is close behind. A pricing page, a compliance rule, or a feature limit that changed six months ago but the document did not. Without a date on the page, retrieval treats a 2023 answer and a 2026 answer as equally valid, because relevance scoring has nothing to do with recency.
Tribal knowledge is the quiet failure. The real fix for a common support issue often lives in a Slack thread or one engineer's head, never in the wiki. A retrieval system cannot answer from a document that does not exist. Each gap becomes a moment where the agent guesses, or admits it does not know.
Structure failures matter too, even when the content is correct. A 40-page PDF with no headings forces retrieval to chop the text into arbitrary chunks. A chunk boundary landing mid-sentence can separate a rule from the exception right below it. The content was right. It just never got seen whole.
- 01Duplicate or conflicting documents on the same question.
- 02Undated policy pages retrieval cannot tell apart from current ones.
- 03Fixes and answers that exist only in a person's memory or a chat thread.
- 04Long, unstructured documents that get chopped into meaningless chunks.
Run this audit before you scope a build
Pull every document your support and product teams actually reference, then ask your ten most common customer questions against the whole set by hand. Count how many questions return more than one source with a different answer. That number is your duplicate problem, in plain terms rather than an abstract risk.
Check for a last-reviewed date on your highest-traffic pages: pricing, policy, compliance, and setup guides. A page with no date and no owner is a page nobody will notice going stale. Flag anything undated as a review task before it becomes a retrieval source.
Interview your support and success teams for the answers they give that are not written anywhere. This is often the most useful hour of the whole audit. It surfaces the exact gaps a retrieval system will hit first: the workaround, the edge case, the exception that only gets explained out loud.
Assign an owner to keep the corpus current after launch. A knowledge base that grows unmanaged again produces a RAG system that degrades again, on the same schedule. Greenlight, an AI research platform we built, backs every finding with 10 to 15 independent sources and live citations. Grounding a system in unverified sources is what makes it wrong with confidence.
Questions this raises
01How do you prepare documents for a RAG system?
Start by removing duplicate or conflicting documents on the same topic, since retrieval cannot tell which one is authoritative. Date every policy page so stale answers can be filtered out. Write down tribal knowledge that only exists in chat threads or a person's memory. Structure long documents with real headings, so chunking does not cut a rule away from its exception.
02What is knowledge base cleanup for AI, and why does it come before the build?
It is the process of auditing a document set for duplicates, outdated pages, and missing content before feeding it into a retrieval system. Doing it first is cheaper. A messy corpus produces wrong answers. Those get blamed on the model, when the fix was in the source documents all along.
03How much of a RAG project is data preparation versus model work?
In most engagements, document quality is the larger source of wrong answers, not model choice or retrieval tuning. A clean, current, deduplicated corpus retrieved with a modest setup usually beats a messy corpus retrieved with a more sophisticated one.
