Why do clinical AI models fail in production?
Not because the maths was wrong. Because the hospital, the patients and the paperwork all moved after training day.
Answered in short
6 things that decide this
- 01Dataset shift is the main cause. A model underperforms because the data it was built on no longer matches what it now sees.
- 02External validation of a widely used sepsis model across 38,455 hospitalisations found an area under the curve of 0.63, plus poor calibration.
- 03That same model missed 1,709 sepsis patients, or 67%, while alerting on 18% of all patients in hospital. The authors call it a large burden of alert fatigue.
- 04Shift is not always slow. One hospital switched off its sepsis model in April 2020, when pandemic changes in who was admitted produced false alerts.
- 05A newer release of that sepsis model scored much higher on discrimination. Both versions still tended to alert after clinicians had already spotted sepsis.
- 06Shift over time is measurable. One study found a maximum AUROC drop of 0.090 on sepsis prediction between year groups, and that the usual correction methods did not fix it.
Three things move, and none of them is the model
Finlayson and colleagues wrote about this in the New England Journal of Medicine in 2021. They define dataset shift as a model underperforming because of a mismatch. The data it was built on no longer matches the data it now runs on. They group the causes into three.
Technology changes. A new lab analyser. A different electronic record. A coding update. Now a field the model leaned on means something a little different, and nobody touched the software.
The population changes too. A new referral pattern. A different case mix. A ward opening or closing. Behaviour changes as well. Clinicians who learn what sets off an alert start writing notes differently, which quietly rewrites the model's inputs.
From the published literature
What external validation actually found
0.63
AUC of a deployed sepsis model, Wong et al., JAMA Internal Medicine, 2021
67%
of sepsis patients that model did not identify
18%
of all hospitalised patients it generated alerts for
0.090
maximum AUROC drop from temporal shift, Guo et al., Scientific Reports, 2022
What the sepsis study really showed
Wong and colleagues studied 27,697 patients across 38,455 hospitalisations at Michigan Medicine between December 2018 and October 2019. Sepsis occurred in 2,552 of them, about 7%.
The model scored an AUC of 0.63, with poor calibration as well as poor discrimination. It missed 1,709 sepsis patients while alerting on 6,971 stays. The authors flag two things. Sensitivity was low next to ordinary clinical practice, and the alert burden was heavy.
The follow-up matters more for anyone building today. A 2025 emergency-department validation found a newer release scored far better on discrimination. Yet at a threshold set for 60% sensitivity, both versions beat the clinician to the diagnosis in only about a third of cases. A better score did not make the alert earlier.
- Discrimination is not usefulness. A model that alerts accurately, after the nurse already knew, has changed nothing about the patient's care.
- Trained on the pastOne site, one period, one case mix.
- DeployedScores well on a held-out split.
- The setting movesNew analyser, new referral pattern.
- Behaviour adaptsStaff document to the alert.
- Accuracy slidesNo error, no log line, no alarm.
- Trust goesClinicians start ignoring it.
The fifth station is the whole problem. Nothing here throws an exception, so a model can be wrong for two quarters before anybody assembles the evidence to say so.
Systems where a wrong answer was not acceptable
Related questions
01Does a higher AUC mean the model is clinically useful?
No, and the sepsis follow-up shows it cleanly. The newer model scored much better on discrimination and still alerted after clinicians had spotted sepsis themselves. Timing against the human, and calibration, matter at least as much as the headline score.
02How often should a clinical model be revalidated?
Often enough that measurement catches a shift before a complaint does. How fast your setting changes sets the schedule. The pandemic example shows the gap can be weeks rather than years. In practice: watch the inputs all the time, and re-score against a graded set at intervals.
03Can we buy a model that avoids this?
Every model has the same exposure, because what moves is your environment rather than the vendor's code. A bought model is often harder, since you cannot see what it learned from. Ask for the external validation, the population behind it, and how the vendor spots drift at your site.
04Are the published validation studies themselves reliable?
Treat them carefully. A 2024 systematic review in BMJ Medicine covered 47 validations of one model. It reported that 91% were at high risk of bias, and that calibration was rarely checked. So run your own validation on your own data rather than trusting a summary figure.
Related
- Dataset shift →The term, defined.
- When does my software become a regulated medical device →The classification question these systems raise.
- MLOps consulting →The monitoring that catches this before a clinician does.
- AI evals →The graded set behind any claim of accuracy.
- Healthcare software development →Where we build under these constraints.
