After launch
The spec changed. Here is what it does.
A tolerance tightened. The pixels are identical and every label you own is now wrong.
From the floor
Engineering moved a threshold, or a regulator did. Surface pitting that was acceptable last quarter is a reportable defect this quarter. The footage looks the same, the model looks healthy, and it is now confidently applying last quarter's rulebook.
Which failure is it
Concept drift
The input distribution has not moved at all. What moved is the mapping from image to correct answer, which is the definition of concept drift and the reason it cannot be caught by watching the inputs.
Four modes, and they need different responses: covariate, concept, prior, and a pipeline event wearing drift’s coat.
What moves first
Accuracy is the last thing to tell you.
There is no input signal. Confidence stays high because nothing looks unfamiliar. The first real evidence is human reviewers overriding the model in a consistent direction, which is why the override rate is worth logging even when accuracy looks fine.
Measuring accuracy directly needs fresh labels, so teams lean on proxies between audits. By the time a quarterly audit shows a drop, the system has been wrong for a quarter. The cheap signals are confidence per class, detection rate per camera against its own history, and how often a reviewer overrides the model.
What to do
Re-label against the new specification and retrain. There is no shortcut: the ground truth itself changed, so the old labels are liabilities rather than assets.
What it costs to ignore
The most expensive failure on this list. A confidently obsolete model passes defects that are now reportable, and it does it consistently, at scale, with an audit trail saying everything was fine.
Where this shows up
What it gets first