How to catch model drift before your customers do

Every production vision model is a snapshot of the world on the day its training set was frozen. The world does not stay frozen. Lighting shifts with the seasons, equipment corrodes, cameras get bumped a few degrees, new objects walk into frame. The model keeps emitting confident predictions the whole time, and confidence is exactly the problem: a drifting model does not tell you it is drifting.

The model was fine on launch day. The world simply moved.

Accuracy in production is invisible by default. Unless you are re-labeling a sample of live footage and scoring the model against it, the only accuracy signal you have is downstream damage: a missed defect, a false alarm storm, a denied claim. Teams discover drift weeks late because the tooling that measured their model died the day it left the training notebook.

First, confidence distribution shift. Log every prediction's confidence and watch the distribution week over week. A healthy model has a stable shape; drift shows up as the shape moving - often mass sliding toward the middle. Second, disagreement sampling. Run a second, different model on a small slice of traffic and route disagreements to a human. Disagreement rate is a cheap, powerful drift alarm. Third, verified spot checks. A human verifying twenty frames a day beats a dashboard nobody opens.

Detection without correction just moves the pain forward. The frames that caught the drift are the exact frames the next training run needs. Wire them straight into your dataset with verification, retrain on a schedule the drift rate sets, and the incident becomes an improvement. That loop, label, monitor, retrain, is the whole reason LexData exists, but the pattern holds no matter what you build it with.