Agentic RAG Is RAG That Knows When to Look Again
A standard RAG pipeline retrieves once and answers with whatever came back. An agentic one checks the results first, and searches again if they fall short.
The short version
5 things that decide this
- 01Classic RAG runs one retrieval pass per question and answers from whatever it returns, even when the results are thin or off-topic.
- 02The agentic version adds a judgment step. It checks whether the retrieved passages actually answer the question before generation runs.
- 03When the check fails, there are three moves: re-query with different terms, reformulate the question, or refuse to answer.
- 04Greenlight, an AI ESG research platform we built, needed this because one search pass across environmental, social and governance topics was never going to be enough.
- 05The mechanism is a loop with an exit condition, not a bigger model or a longer prompt.
One retrieval pass, or a loop that checks itself
Standard retrieval-augmented generation runs a fixed sequence. A question comes in. A search runs against an index. The top-ranked passages get attached to a prompt, and a model writes an answer from them. That sequence runs exactly once. If the search missed the right passage, the model still answers. Nothing in the pipeline checks the retrieval before generation starts.
Agentic retrieval inserts a decision between those two steps. A checking step looks at what came back and asks whether it actually supports an answer. Are the passages on topic. Do they cover the question, or only part of it. Do two sources disagree. That judgment decides what happens next. Generation only runs once the evidence holds up.
This is a mechanism, not a marketing label. Most vendor pages using the term describe a chatbot with extra tools, and stop there. The real difference is narrower, and testable. Does the system evaluate its own retrieval and act on that evaluation? Or does it generate from whatever the first search returned, regardless?
Three moves when the first pass is not enough
A weak retrieval has three honest responses. A pipeline has to be built to pick the right one, not always take the same exit.
- 01Re-query: search again with different terms or a different index, when the passages came back off-topic.
- 02Reformulate: rewrite the question, when the original phrasing was too vague or bundled two questions into one.
- 03Refuse: give no answer, or state the gap, when retrieval still cannot back up a claim.
- 04Loop once more only if the budget allows it, then stop and take the refuse path.
One search pass was never going to cover an ESG scan
We built Greenlight, an AI platform that scores how sustainable a company is, on GPT-4 and Perplexity Sonar-Pro with retrieval grounding. Each scan covers 50 or more ESG topics. Every topic needs 10 to 15 independent sources before it counts as answered. A single retrieval pass per topic could not reach that bar. The first search on a topic like supply-chain emissions or board governance rarely surfaces enough independent, credible sources in one try.
That pipeline runs parallel deep-dives per topic instead of one shared search. Each dive keeps gathering sources until the topic has enough independent coverage to score. Company self-reports and regulator filings often disagree. That is exactly the case a single-pass system cannot resolve. It retrieves both, hands them to the model, and lets the disagreement pass through unexamined. Greenlight's scoring weights AI judgment at 66 percent and data averages at 33 percent. It favors independent sources over a company's own claims. That only works if the pipeline knows which sources it is still missing before it scores anything.
None of that is a bigger model doing more work per call. It is a pipeline that checks whether it has enough before generation runs, topic by topic, across every scan.
Questions this raises
01What is the difference between RAG and agentic RAG?
Standard RAG retrieves once and generates an answer from whatever came back. Agentic RAG adds a checking step first. It judges whether the retrieval supports an answer, and re-queries, reformulates or refuses if it does not. The difference is the check, not the model size.
02Does agentic RAG always mean using more tool calls?
It means the system can choose to make another retrieval call, not that it always does. If your first-pass result is strong, the loop should exit after one search. Extra calls only fire when the check fails, so cost scales with how often retrieval actually falls short.
03When should a RAG system refuse to answer instead of retrying?
Once repeated re-querying and reformulation still return sources too thin or too contradictory to support a claim. A system that always answers eventually answers confidently from bad evidence. Refusal, or an answer that states the gap, is the correct outcome when the evidence genuinely is not there.
Related
- RAG →The base mechanism agentic RAG adds a decision loop to.
- Agentic AI →The broader pattern this post's decision loop is one instance of.
- RAG development →How we build retrieval pipelines, including the check-and-retry step.
- Greenlight: AI ESG research platform →The multi-source pipeline described in this post, in production.
