How do you add RAG to an existing SaaS product?
The short answer
5 things that decide this
- 01Adding retrieval to a multi-tenant SaaS product is a permissions problem before it is an AI problem. Without a tenant filter inside the vector query, one customer's assistant can answer from another customer's documents.
- 02Tag every chunk with its tenant id at ingestion time, because a chunk that reaches the index untagged cannot be filtered later and has to be rebuilt.
- 03Start with one document type and one question your support team already answers by hand, so you can score the results against known-good answers.
- 04Retrofits rarely require rebuilding the product. For Military Cruise Deals we added a conversational search module to their live WordPress site and handed results back to the search engine they already ran.
- 05Ship the refusal path before the answer path. An assistant that says nothing found is safe to launch; one that always produces a paragraph is not.
Why tenant isolation is the whole job
Your app already has permissions. The vector index does not inherit them. Embeddings live in a separate store. Unless you carry the tenant id across at ingestion, every customer's text sits in one shared pile of numbers.
There is a second trap under the first. Approximate indexes filter after they scan, not before. pgvector states it plainly: "Vector search narrows to a shortlist first, then applies your filter. So a filter that matches one row in ten can leave you with almost nothing, and the feature looks broken when the data is fine."
So a correct tenant filter returns too few results, the feature looks broken, and the quickest fix is to drop the filter and screen results in application code. That fix is the leak. The passages have already been read into the model's context, and only the interface is hiding them. The database has fixes for this and your engineers will know them. The decision that matters is where the filter runs: inside the search, or after it.
- Pick one jobA question support answers daily
- Tag at ingestTenant id on every chunk
- Filter in queryNever after the search
- Score itKnown-good answers first
- Add refusalNothing found is shippable
- Then widenSecond source, second question
Ingestion tagging comes before anything user-facing. Untagged chunks cannot be filtered later, only re-indexed.
What to build first, and what to leave alone
Pick the narrowest useful job. One document type, one question, one place in the interface where the answer appears. A support macro your team already sends by hand is ideal. You have the correct answers sitting in a queue, so you can score against them from day one.
Leave your database schema alone. Embeddings can live beside your existing rows with pgvector, which keeps one backup and one access model instead of two. Leave your auth alone too. The tenant id you already trust is the value that belongs on each chunk.
- 01Write the eval set before the feature. Twenty real questions with known answers beat a demo that impresses in a meeting.
- 02Keep the assistant read-only at first. Retrieval that answers is a smaller blast radius than an agent that acts.
- 03Log the retrieved passage ids with every answer, or you cannot debug a wrong reply afterwards.
01Do we need a separate vector database?
Usually not. If your product already runs Postgres, pgvector stores embeddings next to the rows they describe, so your existing backups and access rules cover them. A dedicated engine is worth it once scale or filtering behaviour demands it, which is a decision you can make later with real numbers.
02How do we keep the index current as customers edit documents?
Re-embed on write, using the same event that already updates the record. Treat the index as derived data you can rebuild from source at any time. Nightly rebuilds are fine early on. They stop being fine the day a customer edits a document and expects the assistant to know within the hour.
03What if our documents are mostly scanned PDFs?
Run OCR at ingestion and check the output before indexing it, because a bad scan produces confident nonsense in the index. Budget real time for this. Document preparation is routinely the largest single piece of work in a retrieval project, and it is the part that gets left out of estimates.
04Should we build this in-house or bring someone in?
Build in-house if someone on the team has shipped retrieval before and has time to own it after launch. The failure mode is not the first version, which is easy. It is the sixth month, when documents have changed and nobody is scoring the answers any more.
Related
- rag vs fine tuning →Including long context, the third option.
- where the embeddings live →The filtering behaviour described above.
- an assistant that cites its source →How we build and run these systems.
- people who have built retrieval that works →Engineers to embed in your team.
- matching patients to oncology trials →De-identification and an audit trail in practice.
