After launch
A new site came online. Here is what it does.
The model was trained at one site and deployed at another, and the second site does not look like the first.
From the floor
It worked at the pilot site, so it was rolled out. The new site has different equipment vintages, different mounting heights, different lighting and a different background. Performance is visibly worse and it is tempting to conclude the model is bad, when it is doing exactly what a model trained on one site does.
Which failure is it
Covariate shift
Same task, same definitions, different input distribution. Site-to-site is covariate shift arriving all at once instead of gradually, which makes it easier to diagnose and just as expensive to ignore.
Four modes, and they need different responses: covariate, concept, prior, and a pipeline event wearing drift’s coat.
What moves first
Accuracy is the last thing to tell you.
The new site's detection profile differs from the pilot's from day one rather than degrading into it. Per-site comparison catches this immediately; a fleet-wide average hides it, because the good sites carry the bad one.
Measuring accuracy directly needs fresh labels, so teams lean on proxies between audits. By the time a quarterly audit shows a drop, the system has been wrong for a quarter. The cheap signals are confidence per class, detection rate per camera against its own history, and how often a reviewer overrides the model.
What to do
Label a window from the new site and fold it in. A model that has seen three sites generalises to the fourth far better than one that has seen a single site very thoroughly.
What it costs to ignore
A rollout that stalls because the second site loses confidence in a system the first site trusts, and a reputation problem that outlasts the technical fix.
Where this shows up
What it gets first