After launch
Rain, fog and dust. Here is what it does.
Conditions the model rarely saw in training arrive for a week and detection falls off a cliff, then recovers.
From the floor
A storm front, a dust season, fog that sits until mid-morning. The camera is working, the feed is being processed, and detections thin out for days before returning without anybody touching anything.
Which failure is it
Covariate shift
Transient covariate shift. Unlike seasonal change it arrives and leaves, which is precisely why it gets dismissed as noise instead of being collected and used.
Four modes, and they need different responses: covariate, concept, prior, and a pipeline event wearing drift’s coat.
What moves first
Accuracy is the last thing to tell you.
The share of frames where nothing is found at all rises sharply, then returns. Confidence drops across every class rather than one, and the pattern repeats with the weather rather than the calendar.
Measuring accuracy directly needs fresh labels, so teams lean on proxies between audits. By the time a quarterly audit shows a drop, the system has been wrong for a quarter. The cheap signals are confidence per class, detection rate per camera against its own history, and how often a reviewer overrides the model.
What to do
Keep the bad-weather footage. Those days are rare in a training set and disproportionately valuable in it, and the correction comes fast because the frames are already recorded.
What it costs to ignore
A system that is least reliable in exactly the conditions where the assets are most at risk, and which nobody trusts to have been watching during the storm.
Where this shows up
What it gets first