What is dataset shift?
Nothing errors. No alert fires. The model still returns confident answers, and they are worse than last quarter's, because its data stopped looking like the data it learned from.
Dataset shift
Data driftDistribution shiftConcept drift
Dataset shift is what happens when a machine learning model performs worse because the data it meets in production no longer matches the data it was trained on. The model's code and weights are unchanged. The world it was measured against moved.
Three things move, and they move for different reasons. The inputs change, so a new patient mix or a new customer segment arrives. The relationship changes, so the same input now means something else. Or the labels change, because someone redefined what you were counting.
The last one catches teams out most. A code set migration, a new lab assay, a renamed field upstream. None of these look like AI problems. Each can move a model's inputs overnight.
A pandemic switched off a hospital's sepsis model
In April 2020 the University of Michigan Hospital had to turn off the Epic sepsis model. The reason, in the authors' words, was "spurious alerting owing to changes in patients' demographic characteristics associated with the coronavirus disease 2019 pandemic". The model was fine. Its patients were different.
That account comes from Finlayson and colleagues in the New England Journal of Medicine, and the same paper lists quieter versions of the problem. A model built on ICD-9 diagnosis codes may misread a hospital that has moved to ICD-10. Routine electronic health record updates can change what a field means without touching the model. One tool for predicting hip fractures turned out to be reading the scanner model and the technician, not the bone.
Outside hospitals the mechanism is identical. A pricing model trained before a competitor entered the market, a fraud model trained before attackers read your rules, a support classifier trained before your product renamed half its features.
- TrainMeasured once, on old data.
- DeployAccuracy looks good.
- DriftInputs move. No error.
- DecayConfident and wrong.
- NoticeA human spots it, months on.
There is no exception to catch. A drifted model returns a well-formed answer every time, which is why monitoring has to watch the inputs and the outcomes, not the uptime.
Related questions
01What is the difference between dataset shift and overfitting?
Overfitting is a flaw at training time: the model memorised its training set and never generalised. Dataset shift happens to a model that did generalise, and then the world moved away from it. One is caught by testing on held-out data before launch. The other can only be caught after launch, by monitoring.
02How often should a model be retrained?
Retrain on evidence of drift rather than on a calendar. A fixed quarterly schedule retrains models that were fine and leaves broken ones live for up to three months. Track input distributions and outcome quality, and let those trigger the work. Some models need attention monthly, others hold for a year.
03Does dataset shift affect LLM applications?
Yes, and it arrives through two doors instead of one. Your data drifts, as it always did. The model provider also updates the model underneath you, so the same prompt can return different output than it did last month. That is why an eval suite you can rerun matters more with a hosted model than with one you trained.
Keep reading
- AI evals →The test suite that tells you a model got worse.
- How do you know if an AI agent is production-ready? →What has to be true before a model runs unattended.
- Healthcare software development →Where clinical models fail, and what we build instead.
- TrialTriage: AI clinical trial matching →AI ranks the trials, a nurse signs off on every one.
