What is legal RAG, and does a firm need it?
Retrieval-augmented generation means the model answers from documents you hand it and shows you where the answer came from, instead of answering from memory. That's the whole idea, and it's why you'll trust it more than a general chatbot for matter work.
The short answer
5 things that decide this
- 01Legal RAG (retrieval-augmented generation) retrieves passages from your own matter files, then has the model write an answer grounded in those passages, with a citation back to the source document.
- 02It's different from asking a chatbot a question from memory. A general model answers from its training data. RAG answers from what you actually indexed, and it can say 'I can't find that' instead of inventing a source.
- 03A good legal RAG tool is scoped by who may see the matter, keeps an audit trail of what it read and returned, and reads your existing systems without moving or duplicating the files.
- 04Even purpose-built legal RAG tools still get facts wrong at a measurable rate. Citation verification has to run as a separate check, not as something you trust the model to get right on its own.
- 05A firm with matters scattered across email, a DMS, practice management and shared drives, and with associates who keep re-learning the same precedent, is the firm this is built for. A firm whose PMS search already answers most questions doesn't need to add it yet.
The matter file lives in six places
Ask an associate where a matter's key documents actually live and you'll get a list, not an answer. Some of it's in Clio, MyCase, PracticePanther, Smokeball or whatever your firm runs for practice management. Some is filed in NetDocuments or iManage, under a naming scheme only the paralegal who set it up still understands. A chunk lives in email threads nobody moved into the DMS, and a signed engagement letter or two lives only in DocuSign. Another chunk sits on a shared drive, and the rest is in local folders on someone's laptop, or in that person's memory. Memory leaves the building when they do.
None of that is a discipline problem the firm administrator can fix with a memo. A matter accumulates documents from a dozen sources over months or years. Every system keeps its own search box that only searches itself. So the honest answer to 'have we handled something like this before' is usually a guess. Or it's an email to whoever's been at the firm longest, often the managing partner.
That's the gap legal RAG closes. It doesn't replace your DMS or your PMS. It reads across them, indexes what's already there, and answers a question by pointing at the document that actually contains the answer.
What a good legal RAG tool actually does
You can tell a well-built system from a demo by five things you should check yourself, and most of them are boring on purpose.
- 01It cites the source with a link. Every answer points at the specific document and passage it came from, so you can open it and check in one click, not go hunting.
- 02It's scoped to who may see the matter, and it respects a conflict check the same way your DMS does. A screened lawyer can't get privileged content back through a search box that ignores the wall the firm already built.
- 03It keeps an audit trail: what you asked, what it retrieved, what it returned, and when. That record is what you'd need to answer a client's security questionnaire or explain a privilege call later.
- 04It says 'I can't find that' rather than guessing. A retrieval system with nothing relevant to return should say so. A model padding the gap with a plausible-sounding answer is the failure mode the whole design exists to prevent.
- 05It's read-only. It indexes what you already have. It doesn't move your files, doesn't become a second system of record, and doesn't ask your firm to re-file anything to make the search work.
- QuestionAsked inside a matter, scoped to who's asking
- RetrievalSearches indexed documents, not the model's memory
- Grounded draftWritten from the retrieved passages only
- Citation checkVerified against the source, separately from the model
- Answer with linkLawyer opens the source and decides what it means
Retrieval and citation checking are separate steps from generation. That separation is what keeps an answer honest.
Even the purpose-built tools still get it wrong
Here's the part vendors don't put on their homepage, and you should ask about it before you sign anything. Stanford's RegLab and Institute for Human-Centered AI benchmarked leading legal research tools in May 2024. They used a set of over 200 real legal queries, and every tool they tested hallucinated at a measurable rate. Lexis+ AI and Ask Practical Law AI produced incorrect information more than 17% of the time. Westlaw AI-Assisted Research hallucinated more than 34% of the time. A general-purpose model with no retrieval at all, GPT-4, was wrong on 58% to 82% of legal queries.
These weren't unfinished side projects. They were shipped products from the largest legal research vendors, built specifically to reduce hallucination, and the rate was still real. Researchers drew a useful line between two failure types. One is flatly wrong. Misgrounded is the other: the statement sounds right, but the citation it points to doesn't actually support it. Misgrounding is the more dangerous kind, because it survives a quick glance.
So don't take a vendor's word that the citation is real, ours included. Citation verification has to be a separate, deterministic check against an authority database, not something baked into the model's confidence. And you, not the software, decide what a correctly cited passage actually means for the matter. A system that gets the citation right hasn't finished the job. It's handed you a starting point you still have to read.
The privilege question, answered on paper before anything gets built
A retrieval system that touches every matter in the firm could leak across an ethical wall, if nobody designs against it. So you settle the access model before a document gets indexed, not after. You name every recipient of a matter's text: the model endpoint and any logging or eval layer sitting in front of it. You also check whether that endpoint's written terms rule out training on your inputs. ABA Formal Opinion 512 is explicit here. A boilerplate consent clause in an engagement letter doesn't satisfy informed consent. Your firm has to actually explain the risk, and record that it did.
That written-terms question isn't hypothetical. A federal court in the Southern District of New York ruled from the bench on 10 February 2026, in United States v. Heppner. It held that a defendant's conversations with a consumer AI tool weren't privileged. Its own privacy policy allowed the provider to use and disclose the inputs. Bad intent didn't matter; the terms did the work on their own. Enterprise tools with no-training terms in writing are a different question the court didn't decide, but the direction is clear. Your vendor's contract is now something you may have to produce.
So the working pattern looks like this: named vendors, no-training terms in writing, matter-scoped access rather than a firm-wide login. Add private deployment where the practice area or the client demands it. It's a design decision, made on paper with your firm, before a single document gets pulled into an index.
Which firms need this, and which ones already have enough
You probably need this if your firm runs 10 or more attorneys across a handful of practice areas. Staff turnover is another sign: when associates and paralegals leave often enough, institutional knowledge keeps walking out the door with them. A third sign is simpler still. 'Has anyone handled something like this' currently gets answered by asking around, not by searching. Picture a firm juggling matters across several practice groups, where a precedent from one group would help another if anyone could find it. That's exactly the shape this was built for.
You might not need it yet if you're a small, stable practice in one or two areas. Maybe the same two or three people have handled every matter and still remember them. A well-organized DMS with decent full-text search covers a lot of that ground on its own. Adding a retrieval layer on top of a firm that already finds things fast solves a problem you don't have. Headcount isn't the honest test. It's whether 'where did we handle this before' currently takes a search or takes a conversation.
Related questions
01Does legal RAG replace our DMS or practice management system?+
No. It's read-only and sits on top of NetDocuments, iManage, Clio, MyCase, PracticePanther or whatever you already run. It indexes what's there and answers with a citation back to it. Nothing moves, and nothing gets re-filed to make it work.
02Can it search across matters, or is it walled off per matter?+
It has to respect the same ethical walls your DMS enforces today. A lawyer screened off a matter shouldn't be able to retrieve its content through a search box just because the underlying index technically has access. That scoping is a design requirement, not an afterthought bolted on later.
03How do we know the citations are actually real?+
You verify them against a citation database as a separate step, deterministically, rather than trusting the model's confidence. Purpose-built tools still hallucinated at a measurable rate in the Stanford benchmark, so a claimed citation gets checked before you rely on it.
04Is this the same thing as an AI drafting tool?+
Related, but not the same job. Legal RAG answers a question by pointing at a document. Drafting assembles a first draft of a new demand letter or motion from matter facts and your templates, which is work product a lawyer still has to review. Firms often want both, but they're different pipelines with different review points.
05What does 'audit trail' actually mean here?+
A record of what was asked, what got retrieved, and what the system returned, with a timestamp on each. You'd need that record to answer a client's security questionnaire. You'd also need it to reconstruct a privilege decision months later, if anyone asks how an answer was produced.
Related
- Legal hub →AI, automation and custom software for law firms, built around the system you already run.
- Document automation for law firms →Where matter-document retrieval fits alongside drafting and clause libraries.
- How to automate legal document drafting with AI →The drafting pipeline this page's retrieval layer feeds into.
- The matter file lives in six places →The problem this page answers, told as a story from a real intake.
