Monitoring data drift in computer vision, and the one signal that catches both kinds

The camera on the light mast at the north end of the container yard was trained in June, in daylight, on a fortnight of frames where every box was lit from above and the stacks threw short shadows. In June it found containers, flagged the damaged ones, and counted the stacks against the yard plan. By October the yard runs two night shifts under sodium lamps, the containers are the same, and the model has quietly become a daytime instrument.

The model did not change. The weights are the ones that shipped. What moved is the distance between the frames it learned and the frames it is now shown, and that distance can grow for more than one reason. The drift lesson separates four; this post takes the two a yard manager will meet first, and the one signal that catches both.

The night shift in the container yard is data drift

Under the sodium lamps every container is the same orange, the corner castings vanish into shadow, and the reflective tape on a reefer unit blooms into a white bar the training set never contained. The objects are unchanged. The correct answer for every frame is unchanged. The pixels are different, and the model has weak evidence for them.

That is data drift, covariate shift in the textbook: the input distribution moved while the task stayed where it was. It has a shape you can see if you look. Detections per hour on the mast camera fall after dusk and recover at dawn, the frames the model returns as doubtful cluster in the night hours, and none of it correlates with anything the yard did on purpose.

The season does the same thing more slowly. Leaves come off the trees behind the fence line, the sun sits lower and throws the mast's own shadow across the stack, and by November the model is being asked about a yard it has never seen. The drift catalog covers this as the season turned, and its advice holds here: the first winter is where the corrections matter most, and the baseline has to know the calendar, or the monitor alarms every dawn.

A policy that now counts dents is concept drift, and the frames cannot show it

In the same yard, in the same month, the terminal's damage policy changes. Until now a dent under a hand's width on a container side was cosmetic and the box went out. From the first of October it is reportable, because a shipping line's insurer changed its terms.

The camera sees exactly what it saw last week. The model, trained on labels drawn under the old policy, passes the same dents it always passed, with the same confidence, and it is now wrong on every one of them. That is concept drift: the pixels held still and the mapping from pixel to correct answer moved. The catalog calls it the spec changed, and it is the expensive one, because a confidently obsolete model passes reportable damage consistently, at scale, with a log saying everything was fine.

There is no input signal to watch for. Nothing looks unfamiliar to the model because nothing is.

Watching the frames catches the first kind and misses the second

Most drift monitoring is built to watch the inputs. Compare a window of recent frames against a reference set, by embedding distance or by a histogram of brightness and colour, and alarm when the gap widens. Log detections per camera per hour and compare against that camera's own history. Count the frames where nothing was found at all.

Those are good instruments for the night-shift problem. The gap between June frames and October frames is large and measurable, and the mast camera's detection curve tells the story on its own. Every one of them is blind to the dent policy, though. The October frames are the October frames whether dents are reportable or not, and no comparison of pixels against pixels will notice that a rule changed in a document.

An accuracy figure would catch it, eventually. Computing one needs fresh labels, which is the expensive thing the monitoring was meant to avoid, and by the time the quarterly audit shows the drop the yard has been shipping reportable dents for a quarter.

The operator correction rate is the signal that catches both

The person who catches it is the checker at the gate. Under the new policy the checker walks a container the model passed, finds a dent, and overrides the verdict. Then does it again on the next one. The override rate on the damage class rises in a consistent direction from the first of the month, while the frames look the same as ever.

The same rate rises for the night-shift problem, from the other side: the model returns more doubtful frames after dusk, more of them come back corrected, and the corrections cluster on one camera and one time of day. One number, two very different causes, and the shape of the rise tells you which. A rise tied to a policy date and spread across every camera is a spec change. A rise tied to the clock, or to one mast, is the input moving.

This is why LexData treats the correction rate, and nothing fancier, as the drift signal. Frames the model doubts come back to a person, the corrections land, and when they cross the project's threshold a new version trains. It is the one instrument that works whether the world moved or the rulebook did, and it costs nothing extra, because the corrections were going to be made anyway.

My own view, which not every ML engineer shares, is that a drift monitor with no human override in it is decoration. It will catch the lighting change that anyone standing in the yard could have told you about, and it will sleep through the policy change that costs the terminal a customer.

Rule out a pipeline change before you call it drift

Before anyone retrains, check the change log. A recorder patch that went out on a Tuesday night can drop a camera's bitrate or move its white balance, and the model will show a cliff at the exact hour, confined to the cameras that got the patch. That is a pipeline event rather than drift, and the catalog files it under a firmware or encoding change for a reason: it is the most common cause of a sudden drop and the cheapest to reverse.

The clue is the shape. Drift from the world is a slope. A step means something discrete changed, and every discrete change in a yard has a ticket somewhere.

The corner castings on a shipping container are the same size on every container in the world, which is why the stacker drivers never have to look at them. The model does have to look, and under the lamps it cannot.

Corrections retrain the model without starting over

Once the cause is known, the fix is the same motion for both kinds, with one difference in what gets kept.

For the night shift, the corrected frames from October join the June set. The old frames stay, because a model retrained only on nights would swap which shift it fails on, and the new version is checked per camera and per hour against the conditions that caused the retrain before it replaces the old one. On a fixed mast camera this is the cheapest correction in the catalog, since the frames already exist.

For the dent policy the old labels on the damage class are liabilities now. They were correct under a rule that no longer applies, and keeping them teaches the new version the old rule. That class is re-cut against the new policy; the rest of the set, the container and stack classes the policy did not touch, stays.

LexData takes the yard model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the mast cameras the yard already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The night frames and the re-cut dents are what the next version learns from, and the checker at the gate is the monitor.