After launch
The defect rate changed. Here is what it does.
The images and the definitions hold, but how often the thing occurs has moved, and every threshold tuned around the old rate is now wrong.
From the floor
The model is finding things correctly and the reports have stopped making sense. Alert volume either floods the queue or falls silent. Per-detection the system looks fine, which is what makes the argument hard to have.
Which failure is it
Prior shift
Neither the inputs nor the mapping moved. The base rate did. It is the mode most often mistaken for a model problem when it is a threshold problem.
Four modes, and they need different responses: covariate, concept, prior, and a pipeline event wearing drift’s coat.
What moves first
Accuracy is the last thing to tell you.
Per-detection accuracy holds while downstream counts swing. Alert volume against its own history is the cheapest place to see it, and it is a number most teams already have.
Measuring accuracy directly needs fresh labels, so teams lean on proxies between audits. By the time a quarterly audit shows a drop, the system has been wrong for a quarter. The cheap signals are confidence per class, detection rate per camera against its own history, and how often a reviewer overrides the model.
What to do
Re-tune thresholds against the current base rate before touching the model. If the rate moved because the world moved, that is an operations finding worth surfacing on its own.
What it costs to ignore
A queue nobody trusts. A flood gets ignored and a silence gets read as everything being fine, and both end with people no longer acting on alerts.
Where this shows up
What it gets first