What should I ask an AI vendor before I sign?
Every vendor knows the right answers by now. Ask to see the thing instead.
Answered in short
5 things that decide this
- 01Ask every question in a form that demands an artefact, because a vendor who has shipped can show you a file and a vendor who has not will describe one.
- 02The single most useful request is this: show me the graded test cases from a system you run today, and show me who wrote them.
- 03Ask a vendor to replay one real production run end to end, from the input through to the output, on a screen while you watch.
- 04Ask who is on call after launch, by name and role, and get the answer written into the contract rather than the sales deck.
- 05A vendor who quotes a firm price before reading your data is pricing a demo, and the number will move once they open the data.
Good questions stopped working when everyone learned the answers
Search for vendor questions and you get the same twenty. Do you run evals. Who owns the code. How do you handle hallucination. Those are fair questions, and every vendor selling AI in 2026 has an answer ready.
None of it separates anybody now. A team that has never kept a system alive past month one says "yes, we run evals" as fluently as a team maintaining nine. You cannot hear the difference.
So change the shape of the question. Do not ask whether something exists. Ask to be shown it, from a system already running, with the client's name redacted. Artefacts cannot be improvised in a meeting.
The same question, asked so it cannot be talked around
Left column is what buyers usually ask. Right column is the version that produces evidence.
| What you want to know | The question that gets a yes | The question that gets a file |
|---|---|---|
| Can they prove accuracy | Do you run evals on your systems? | Show me the graded test cases from a system you run now. Who wrote the correct answers, you or the client's experts? |
| Can they debug it | How do you handle errors in production? | Pick a real run from last week. Replay it on screen: the input, what it retrieved, what it called, what it returned. |
| Who is responsible later | Do you offer support after launch? | Who gets paged at 2am, by name and role, and which clause of the contract says so? |
| What happens when the model changes | What if the model is deprecated? | Walk me through the last model migration you did. What broke, and how did you find out? |
| What the price covers | What will this cost? | Which parts of this quote are measured and which are assumed? Show me where the data work sits. |
| Where the model is not allowed to act | Is there a human in the loop? | Name one action in a system you built that the model is blocked from taking alone. Why that one? |
Three answers that should end the conversation
"We can show you a demo instead." A demo is a rehearsed path through a system. You asked for a production run, which is the unrehearsed one. Swapping one for the other is the tell, and it is worth naming out loud in the meeting.
"Accuracy is around ninety-five percent." Ask what the denominator was. A number with no test set behind it is a feeling. A team that measures will tell you how many cases, who graded them, and which ones the system still gets wrong.
"Support is handled by our team." Which person, on which day, under what agreed service level? An unnamed team is nobody. This one matters more than it sounds, because it is the clause that decides what your system looks like in month six.
- Ask each of these while sharing a screen. A vendor who has the artefact will open it. A vendor who does not will offer to send it later, and later is where these requests go to die.
- Show the evalsGraded cases from a live system. Who wrote them?
- Replay a runOne real production trace, unrehearsed.
- Name the ownerA person and a clause, not a team.
- Open the dataWhat did they measure before quoting?
- Check the exitCode, prompts, models, pipelines. Yours in writing.
The first two questions do most of the filtering. Ask them before anyone opens a slide deck, because a vendor answering from a deck is answering about a different project.
Systems we can answer these questions about
The next things buyers ask
01Is it reasonable to ask a vendor to show a client's system?
Yes, with the client's data redacted, and a good vendor will have a way to do it. We can walk through the shape of a run on ZhoopZhoop's voice agents without exposing a single customer call. If a vendor says confidentiality prevents showing you anything at all, ask them to build you a run against your own sample data instead.
02How many vendors should I put through this?
Two or three is enough, because the artefact requests take real time on both sides. This process is deliberately expensive to fake, which is the point. Running it against six vendors mostly buys you six sales meetings.
03What if we are too small for a vendor to take seriously?
A small first project is a normal way to start, and it is a fair thing to say out loud in the first call. Ask the vendor to scope the narrowest useful version rather than the version they would like to sell. If they cannot describe a small one, they are selling you a programme rather than a system.
04Do these questions change for an enterprise procurement process?
The artefact requests stay identical. On top of them you add what your risk team needs: data residency, subprocessors, incident reporting, and who signs off a model change. Certifications belong in that second list, not the first. A vendor with a certificate and no eval set has proved something about its own paperwork, not about your system.
Related
- What does an AI agent project actually cost →The four drivers behind any quote you are handed.
- How do you know if an AI agent is production ready →The bar the artefacts above are measuring against.
- Why do AI projects fail →What goes wrong when nobody asked.
- Twelve questions to ask an AI agency →The full list, with the warning-sign answer for each.
