What breaks in production AI
Seven failure modes that show up after launch, not before. Each one with the signal that arrives first and what closing it takes.
What the report covers
4 things that decide this
- 01AI systems rarely fail at launch. They fail in week eleven, when an input shape changes and nothing is watching for it.
- 02The most expensive failure is the quiet one: a system that keeps answering, still sounds confident, and has been wrong since a model version changed under it.
- 03Every failure mode here has an early signal that costs nothing to watch. Most teams do not watch any of them.
- 04Fixing these after launch costs more than building them in, because you are now debugging a system people already depend on.
The failures we get called about all look the same
Nobody calls us when the demo goes badly. They call when a system that worked for two months starts producing answers a customer noticed before anyone internal did.
MIT's 2025 State of AI in Business report puts a number on the gap. Of the enterprise-grade AI tools organisations looked at, 60 percent got investigated and 20 percent reached a pilot. Five percent reached production. That last drop is where this list lives.
You will recognise most of the seven modes below. They come from systems we run. AI voice agents answer a repair shop's phone at ZhoopZhoop. Document extraction sits inside premium audits at PremiumAudit.io. A nurse signs off every trial match at TrialTriage. Each mode gives you a signal first.
- Input shiftsA form, a vendor, a document layout changes
- Retrieval degradesThe right document stops being returned
- Model swapsThe provider ships a new version underneath
- Tool call failsThe system it depends on times out
- Nobody sees itOutput still looks confident and complete
- A customer finds itThe first real signal, weeks late
Each stage is detectable. The chain only completes when nothing is watching the ones before it.
Seven failure modes, the signal, and the fix
This is the substance of the report. The formatted version adds the checks we run for each one.
| Failure mode | What you see | Earliest signal | What closes it |
|---|---|---|---|
| Silent model swap | Answers change quality with no deploy on your side | Output length or format shifts on a fixed test set | Pin the model version and run the same test set daily |
| Retrieval drift | Confident answers built on the wrong document | The correct source stops appearing in the top results | Score retrieval separately from the answer, on known questions |
| Input shape change | One vendor's documents start extracting badly | Extraction confidence drops for one source only | Track accuracy per source, never as one blended number |
| Tool timeout | The agent invents a plausible answer instead of failing | Rising latency on one dependency before the errors start | Fail loudly on a timeout, and never let the model guess |
| Duplicate action | The same booking, charge or ticket created twice | Retries logged with no matching idempotency key | An idempotency key on every action that changes something |
| Prompt regression | A fix for one case quietly breaks four others | No test failed, because there were no tests | An eval suite that runs before the change ships |
| Unowned system | Everyone assumes someone else is watching | Nobody can name who gets paged at 2am | A named owner and an agreed service level, in writing |
A wrong answer that looks right is worse than an outage
An outage tells you it happened. A quietly degraded model keeps serving, keeps sounding fluent, and keeps being believed. Nobody files a ticket. That is why a better model never fixes this.
In an insurance premium audit, a figure read from the wrong column still produces a complete-looking report. TrialTriage handles this by keeping a nurse as the final decision-maker on every ranked result. It tracks 23 audit action types, so any output traces back to what produced it.
The pattern holds anywhere. Decide up front what your system may settle alone. Log what it did. Check its output against a fixed set of known-correct cases on a schedule. None of that is exotic engineering, and all of it gets cut when a project is measured by its demo date.
- 01Score the retrieval step and the answer step separately, or you cannot tell which one broke.
- 02Keep a frozen set of real cases with known-correct answers, and run it after every change.
- 03Log the inputs, the tool calls and the output for anything a person will act on.
- 04Put a human in the loop where being wrong has a cost that cannot be undone.
Systems running unattended, with a person kept in the loop
Related
- how to tell an agent is production-ready →The checks that decide it, before launch.
- running evals before launch →How the frozen test set is built.
- what happens after the pilot →The gap between a working demo and a running system.
- fixing an AI system somebody else built →Where most of these failures were found.
