What is label leakage?
A model that scores near perfect in testing and falls apart in week one was almost certainly shown something it will never have again.
Label leakage
Target leakageData leakage
Label leakage is a flaw in machine learning where a model trains on information that would not be available at the moment it has to make a real prediction. The model appears highly accurate in testing, because part of the answer was hidden inside its inputs.
Timing is the whole issue. Training data is assembled after the fact, so every row already knows how the story ended. Columns filled in later look like ordinary features and quietly carry the outcome.
A churn model trained with a cancellation-reason field is the standard example. That field is only populated once a customer has cancelled, so the model learns to read the answer rather than predict it.
It hides in ordinary places. An identifier assigned in a later step. A timestamp that only exists after a claim is settled. A status column somebody updates at the end of the process.
Good scores are what make it dangerous
Most bugs announce themselves. Leakage does the opposite: it makes the project look finished. Results beat the target, everyone is pleased, and the flaw survives review because nobody questions a number that exceeded expectations.
The bill arrives after launch. In production the leaked column is empty or arrives too late, so the model falls back on features it barely learned to use. Accuracy drops toward guessing, and the team spends weeks hunting the wrong cause.
Trust is the real casualty. A team burned this way struggles to get budget for the second attempt, even when the second model is honest.
- 01For every input, ask when it gets filled in. If the answer is after the thing you predict, cut it.
- 02Split data by time, not at random, so training never contains the future.
- 03A sudden jump in accuracy after adding one column is the loudest warning you get.
- AssembleHistory, already resolved.
- TrainA late column comes along.
- ScoreResults beat the target.
- ShipNobody argues with 99.
- DropThe column is empty live.
The fourth station is the failure. Reviewing a bad result is routine; reviewing a great one is a discipline.
Where a score has to survive contact with real data
Common questions
01How is label leakage different from overfitting?
Overfitting means a model memorised the training set and fails on anything new. Leakage means the training set itself was wrong. You see overfitting as a gap between training and test scores. Leakage keeps both scores high, which is why the standard checks miss it.
02How do you catch it before launch?
Run the model against the data as it existed at decision time, not as it looks today. Rebuild a handful of real cases using only what was known that morning and compare the answers. Anything that collapses under that test was leaking.
03Can leakage happen without machine learning?
Yes, and it catches people out in reporting too. A dashboard that scores past decisions using fields completed afterwards flatters every decision equally. Same flaw, no model involved.
04Who usually spots it first?
Whoever knows the business process spots it first, not the data science team. The person who fills in the form knows which box gets ticked at the end. That knowledge turns a suspicious column into an obvious one. Walk the real workflow with them before training.
