Why do clinical AI models fail in production?
Not because the maths was wrong, but because the hospital, the patients and the paperwork all moved after training day.
Answered in short
6 things that decide this
- 01Dataset shift is the main cause: your model underperforms because the data it was built on no longer matches what it now sees.
- 02Independent testing of a widely used sepsis model, across 38,455 hospital stays, found an AUC of 0.63 and poor calibration.
- 03That same model missed 1,709 sepsis patients, or 67%, while alerting on 18% of all patients in hospital. Its authors call it a large burden of alert fatigue.
- 04Shift isn't always slow, either. One hospital switched off its sepsis model in April 2020, when pandemic changes in who was admitted produced false alerts.
- 05A newer release of that sepsis model scored much higher on discrimination, yet both versions still tended to alert after clinicians had already spotted sepsis.
- 06You can measure shift over time. One study found a maximum AUROC drop of 0.090 on sepsis prediction between year groups, and that the usual correction methods didn't fix it.
Three things move, and none of them is the model
Finlayson and colleagues wrote about this in the New England Journal of Medicine in 2021. They define dataset shift as a model underperforming because of a mismatch. The data it was built on no longer matches the data it now runs on. They group the causes into three.
Tech changes: a new lab machine, a different patient record system, a coding update. Now a field your model leaned on means something a little different, and nobody touched the software.
Your population changes too, with a new referral pattern, a different case mix, a ward opening or closing. Behaviour changes as well: clinicians who learn what sets off an alert start writing notes differently, which quietly rewrites the model's inputs.
From the published literature
What external validation actually found
0.63
AUC of a deployed sepsis model, Wong et al., JAMA Internal Medicine, 2021
67%
of sepsis patients that model did not identify
18%
of all admitted patients it generated alerts for
0.090
maximum AUROC drop from temporal shift, Guo et al., Scientific Reports, 2022
What the sepsis study really showed
Wong and colleagues studied 27,697 patients at Michigan Medicine, across 38,455 hospital stays between December 2018 and October 2019. Sepsis hit 2,552 of them, about 7%, and that's your baseline for everything that follows.
It scored an AUC of 0.63, with poor calibration as well as poor discrimination. It missed 1,709 sepsis patients while alerting on 6,971 stays. Its authors flag two things: sensitivity was low next to ordinary clinical practice, and the alert burden was heavy.
This follow-up matters more if you're building today. A 2025 ER study found a newer release scored far better at telling sepsis apart from other cases. Yet at a threshold set for 60% sensitivity, both versions beat the clinician to the diagnosis in only about a third of cases. A better score didn't make the alert earlier.
- Discrimination isn't usefulness. A model that alerts accurately, after the nurse already knew, has changed nothing about the patient's care.
- Trained on the pastOne site, one period, one case mix.
- DeployedScores well on a held-out split.
- The setting movesNew analyser, new referral pattern.
- Behaviour adaptsStaff document to the alert.
- Accuracy slidesNo error, no log line, no alarm.
- Trust goesClinicians start ignoring it.
The fifth station is the whole problem. Nothing here throws an exception, so a model can be wrong for two quarters before anybody assembles the evidence to say so.
Systems where a wrong answer was not acceptable
Related questions
01Does a higher AUC mean the model is clinically useful?+
No, and the sepsis follow-up shows it cleanly. The newer model scored much better on discrimination and still alerted after clinicians had spotted sepsis themselves. Timing against the human, and calibration, matter at least as much as the headline score.
02How often should a clinical model be revalidated?+
Often enough that measurement catches a shift before a complaint does. How fast your setting changes sets the schedule, and the pandemic example shows the gap can be weeks rather than years. In practice, watch your inputs all the time, and re-score against a graded set at intervals.
03Can we buy a model that avoids this?+
Every model carries the same exposure, because what moves is your environment rather than the vendor's code. A bought model is often harder, since you can't see what it learned from. Ask for the external validation, the population behind it, and how the vendor spots drift at your site.
04Are the published validation studies themselves reliable?+
Treat them with care. A 2024 review in BMJ Medicine covered 47 tests of one model. It found 91% carried a high risk of bias, and that calibration was rarely checked. Run your own validation on your own data rather than trusting a summary figure.
Related
- When does my software become a regulated medical device →The classification question these systems raise.
- EHR integration →Clinical documentation and the monitoring that catches drift.
- Clinical research →TrialTriage's problem shape: a wrong answer was never acceptable.
- AI, automation and custom software for healthcare →The hub this answer sits under.
