Skip to content
LexDataLexData
PlatformIndustriesCustomers
DocsThe Field GuideBlogWhy models drift
AboutCareersSecurityContact
Log inStart now

// After launch

A firmware or encoding change. Here is what it does.

Someone updated a camera or altered a compression setting, and the input the model receives is no longer the input you tested.

From the floor

An IT update went out across the fleet overnight. White balance moved, or bitrate dropped, or a resize step now runs before inference that did not before. Nobody associated a routine patch with a vision model, so nobody mentioned it.

Which failure is it

Not drift: a pipeline event

Rule this out first

The world did not change and neither did the definitions. An engineering change altered the data path. Worth ruling out first on any sudden drop because it is the most common cause and the cheapest to reverse.

Four modes, and they need different responses: covariate, concept, prior, and a pipeline event wearing drift’s coat.

What moves first

Accuracy is the last thing to tell you.

An abrupt cliff, often to the exact hour, confined to whatever part of the fleet received the change. Drift from the world itself is usually a slope; a step means something discrete changed - and this catalog's step-shaped conditions (a swapped sensor, an equipment rollout) all leave a change log to check before anyone retrains.

Measuring accuracy directly needs fresh labels, so teams lean on proxies between audits. By the time a quarterly audit shows a drop, the system has been wrong for a quarter. The cheap signals are confidence per class, detection rate per camera against its own history, and how often a reviewer overrides the model.

What to do

Find the change and revert or account for it. Retraining against a pipeline regression is expensive, slow, and bakes a mistake into the weights.

What it costs to ignore

Weeks of retraining effort spent on something a configuration diff would have found in an afternoon.

// sources

  • Papasian & Underwood, How ML Breaks: A Decade of Outages for One Large ML Pipeline - Google, USENIX OpML 2020 ↗

Where this shows up

Manufacturing →Retail →Energy & utilities →

What it gets first

The jobs where this one is the usual cause.

Multi-Sensor MonitoringFusing thermal and visual data for comprehensive line analysis.ManufacturingEmployee Activity MonitoringMeasure staff engagement and service levels in key zones.RetailSensor Fusion LabelingSynchronized annotation for camera, LiDAR, and radar data streams.Robotics

The others

A camera movedThe season turnedThe spec changedA new site came onlineAll ten →

We watch for this on your feeds, and retrain when it lands.

Start with your footageBook a demo →
LexData
LexData

Product

  • Platform
  • Industries
  • Use cases

Resources

  • Docs
  • The Field Guide
  • Blog
  • Why models drift

Industries

  • Energy & utilities
  • Oil & gas
  • Agriculture
  • Manufacturing
  • Insurance
  • Retail
  • Robotics

Company

  • About
  • Customers
  • Careers
  • Contact

Trust

  • Security
  • Privacy
  • Terms

Stay updated

What we learn running vision models in production.

See everything.
Miss nothing.

Stay updated

What we learn running vision models in production.

Terms of use & Privacy policy

© 2026 LexData Labs · All rights reserved