In this lesson
The model did not change
Start here, because it corrects the mental picture. The weights are identical to the day you shipped. Nothing rusted. What changed is the world the model is looking at, and the gap between that world and the one in its training set.
This is why “the model degraded” is a misleading phrase. The model is a fixed function. Drift is the distance between your data and its assumptions growing, and that distance can grow for four quite different reasons.
One: the input moved
Covariate shift. The things you are detecting look different than they used to. Seasons change the foliage behind the power line. A camera gets bumped four degrees in a wash-down. A site upgrades to a new equipment generation with a different housing.
Signature: the confidence distribution moves across a whole class - falling is the common case, though shifted inputs can also stay confidently wrong - detections get flakier frame to frame, and the change often correlates with a date, a site, or a camera rather than with a defect type.
Two: the meaning moved
Concept drift. The pixels are the same but the correct answer changed. A tolerance was tightened, so surface pitting that was acceptable last quarter is now a reportable defect. A safety rule changed what counts as a violation.
Signature: the model is confident and wrong. Accuracy against fresh ground truth falls while confidence stays high, because nothing about the image looks unfamiliar to it. This one is invisible to any monitor that only watches the input.
Concept drift is the dangerous one. Covariate shift at least disturbs signals a watcher can catch - the confidence distribution moves, detections get flaky. Concept drift disturbs nothing visible from the inputs. The model is simply, confidently obsolete.
Three: the mix moved
Prior shift. Both the images and the definitions are stable, but the frequency changed. You built during a quiet season and now it is harvest. Defect rate quadrupled, or collapsed.
Signature: per-detection accuracy holds up but downstream numbers stop making sense. Alert volume swings, and thresholds tuned against the old base rate start producing either a flood or a silence.
Four: the pipeline moved
Not drift at all, and worth ruling out first because it is the most common and the cheapest to fix. A firmware update changed the camera's white balance. Someone altered a compression setting. A resize step is now doing something slightly different than it did in training.
Signature: an abrupt cliff rather than a slope, often to the exact hour, and usually confined to whatever part of the fleet got the change. Drift from the world itself is usually gradual; a step change means something discrete changed - hardware, configuration, an asset population - and every one of those has a change log.
This is not a niche caveat. When Google audited fifteen years of outages on one of its largest ML pipelines, sixty of the ninety-six had causes that were not machine learning at all - pipelines, orchestration, bad joins. Check the plumbing before you blame the world.
Why you cannot wait for accuracy to tell you
Accuracy is a lagging indicator, and computing it requires fresh labels, which is the expensive thing you were trying to avoid. By the time your quarterly audit shows a drop, the system has been wrong for a quarter.
The leading signals are cheaper: confidence distributions per class, detection rate per camera against its own history, the share of frames where nothing is found at all, and the rate at which human reviewers override the model. Those move first, and they move differently for each of the four failure modes above, which is what makes them diagnostic rather than merely alarming.
One craft rule for the per-camera baseline: compare like windows with like - the same hour, the same season. A baseline that ignores the calendar alarms every dawn.