Your AI demo works. Now make it survive real users.
The gap between a prototype that impresses a board and a system you can support is mostly invisible until you cross it.
Anyone can build the demo. We build what runs after. A prototype only has to work once, in front of people who want it to work. Production has to work on the worst input a stranger can supply, at 3am, when nobody is watching.
What actually blocks the launch
It is rarely the model. The prototype proved the model can do the task. What is missing is everything around it that turns a capability into a system somebody can be accountable for.
Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027. The RAND Corporation puts AI project failure above 80%, roughly twice the rate of IT projects that involve no AI. Those numbers are not about model quality.
- 01No evaluation suite, so nobody can tell whether a change made it worse.
- 02No guardrails, so the model is one unusual input away from an answer you cannot defend.
- 03No observability, so the first sign of a problem is a customer email.
- 04No rollback, so a bad prompt change is an outage rather than an inconvenience.
The odds you are fighting
80%+
AI projects that fail (RAND)
40%+
agentic projects cancelled by 2027 (Gartner)
22
production systems we have shipped
8
of them handling payments
The hardening checklist
An evaluation suite that runs on every change, with cases drawn from your real failures rather than invented ones.
Guardrails on input and output, including what the system must refuse to answer.
Retrieval that is measured, not assumed. Most bad answers are retrieval failures wearing a model costume.
Observability: every call, its cost, its latency and the passages it read.
A human checkpoint wherever an error would be expensive to reverse.
A rollback path and a model-deprecation plan, because the model you launch on will be retired.
What you have at the end
- A number for how often the system is right, and a way to watch it move.
- The ability to change a prompt on a Tuesday without holding your breath.
- An answer when a regulator or a customer asks why the system said what it said.
- BaselineMeasure what it does today.
- EvalsCases from real failures.
- GuardrailsInput limits, output checks.
- ObserveCost, latency, retrieved context.
- ShipWith a rollback that works.
The order matters. Without the baseline you cannot prove any later change helped.
Is yours production ready?
Three questions. If you cannot answer them, you have a prototype, whatever it is currently serving.
How often is it right, on cases you did not choose?
What happens on an input nobody anticipated?
If a prompt changed this morning, would you know by lunchtime?
Every outcome
- Close. Harden it.
- You have the instincts and probably the evals. What is usually missing is observability and a rollback that has been tested rather than assumed.
- Still a prototype
- Nothing wrong with that, as long as nobody is treating it as a system. The first move is a baseline: measure what it does today, on cases you did not pick.
Where we usually start
The evaluation gap
We build the test set first, from the cases that already embarrassed you. Everything after that is measurable.
Retrieval quality
When answers are wrong, retrieval is usually the cause. We measure what the model was given before blaming what it wrote.
Human in the loop
Deciding which actions a model may take alone, and which need a person. Usually the difference between shippable and not.
Cost and latency
Caching, routing to smaller models, and cutting the context nobody reads. Production economics differ from demo economics.
Most teams cannot answer one question
Ask how often the system is right, measured on cases nobody hand-picked. A prototype cannot answer that. Production has to.
We build the test set first, drawn from the failures that already embarrassed you. Everything after becomes measurable, including whether our own changes helped.
- Cases drawn from real failures, not invented ones.
- A score you can watch move week to week.
- A prompt change becomes an experiment rather than a gamble.
The stack behind this work
Models
Retrieval
Runtime
Observability
Systems we took into production
“They will treat your vision like their own and build it that way.”
Ron Klabunde · Founder, SmartREI ↗
Demo-ready against production-ready
| Criterion | A prototype | A production system |
|---|---|---|
| Correctness | It worked in the demo | A measured score on cases nobody chose |
| Bad input | Nobody tried one | Refused, logged, and handled |
| A change | Hope | Evals run before it ships |
| Failure | A customer tells you | An alert, with the context it read |
| Model retired | An emergency | Re-run the suite, swap, ship |
On stage, and on the floor
Common questions
01How do I know if my AI system is production ready?
Ask whether you can answer three questions. How often is it right, measured on cases you did not choose? What does it do with an input nobody anticipated? And if you changed a prompt this morning, how would you know by lunchtime whether it got worse? A system that cannot answer all three is a prototype, whatever it is currently serving.
02Can you work with a prototype another team built?
That is most of this work. We start by measuring what it currently does rather than rewriting it, because the prototype usually encodes real knowledge about the problem. What gets replaced is the scaffolding around it.
03What if the prototype turns out to be the wrong approach?
We tell you in the first fortnight, before you have spent the budget. Occasionally the honest answer is that the task does not need a language model, and a smaller deterministic system would be more accurate and cheaper to run.
04How long does hardening take?
It depends on how much of the checklist already exists and how expensive a wrong answer is. A system where errors are annoying moves faster than one where an error is a regulatory event. We scope it after a diagnostic rather than guessing on a call.
05Do you hand it over or stay?
Either, agreed before we start. We can stay under a defined service level, or hand over with the evaluation suite, the runbook and your team trained to run it. What we will not do is leave a system nobody is watching.
