Why computer vision pilots fail on the factory floor

The pilot went in on the inspection station at the end of the stamping line in January. A camera over the conveyor, a model trained on a month of frames, a screen that turned red when it saw a dent. In the lab it had been fine. By week five the line lead had put a piece of tape over the screen, and by week eight the station was back to the inspector, alone, and the pilot was a line in a slide deck about lessons learned.

Nothing in that story is about the model.

Manufacturing pilots die on the floor for reasons that have nothing to do with the architecture and everything to do with what was built around it. In the meeting where the pilot is closed, the model's accuracy figure is never the thing on the table.

Mask R-CNN was never the problem

The detectors and segmenters available today are mature. Whether the pilot used Mask R-CNN, a single-shot detector or a transformer, it could learn a dent on a stamped panel from a modest set of frames and run on a box beside the recorder. Picking between them is a real decision and a small one.

The larger decisions are organisational. A camera on the station changes who makes the quality call and how a disagreement gets settled. The inspector who used their own judgment now has a second opinion on a screen. The supervisor has to decide what happens when the two disagree. The quality manager has a new stream of events to fold into reporting. A pilot plan that is only a software rollout underestimates all of this, and that is where week five comes from.

The model has to be validated against the inspector beside it

The model at the stamping station was measured against a held-out set of frames in the lab in December. Nobody measured it against the inspector standing next to it, and the inspector is the reference the floor actually trusts.

That comparison is the validation that matters. Run the model and the inspector on the same panels for a shift, count where they agree and where they part, and look at the disagreements one by one. Some will be the model's mistakes. Some will be the inspector's, because a person at hour eight of a shift does not judge a borderline scuff the way they did at hour two.

The aim is to know, panel by panel, what the disagreements look like, and to decide in advance what the line does when they happen. A false reject that stops the belt costs as much as a miss, and both need a number before the pilot starts.

The operator needs a way to say no, and that no is a label

What was missing at the stamping station was any way for the operator to answer the screen. It turned red and the operator could do nothing about it except ignore it. No way to say that the dent was a reflection, and no way for that judgment to change anything. So the red screen kept being wrong in the same way, the operator kept being right, and the tape went on.

An override is the most valuable thing an operator can give a pilot. It says the model was wrong on this frame, in this way, and that is exactly what the next version needs to learn. On the pilots we run, the override path is live before the camera is.

LexData takes the model through its whole life on that principle. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the station camera, on a runner beside the recorder or on your servers. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The override is a label, and it has to be one before the pilot goes live rather than bolted on in week six.

The tape, for what it is worth, was blue painter's tape, and it stayed on the screen for a month after the pilot ended because nobody was sure whose job it was to take it off.

The first line should be small enough to win on

The stamping station in January was the wrong first pilot for a different reason. It was the most visible line in the plant and the one with the widest range of panels, finishes and lighting. The distance between what the model could do on day one and what the line needed was large, and every day closed a little of it with a small discouraging surprise until the plant decided to stop.

My view is that most first pilots land on the wrong line for political reasons: the line the plant manager walks visitors past. A better first pilot is one that is measurable, that everyone on the floor understands, and that is achievable with the frames you have. One station, one defect class, one camera, on a line where a false reject is annoying rather than expensive. A quiet win the floor can see is what earns the second pilot.

The plants that keep correcting are the ones that keep the model

The plants where a camera becomes part of how quality is done are the ones that kept going: pilot, corrections, retrain, next version, for months, and rarely the ones that started with the best model. The manufacturing work behind our numbers holds 99%+ accuracy maintained in production, and it holds it because the corrections kept landing as lighting changed, fixtures moved and new panels arrived.

Persistence is cheap when the infrastructure exists and expensive when it does not. If every retrain is a project, the plant does one a year and the model drifts between them. If corrections flow into the next version on their own, the plant stops thinking about retraining at all. Retraining without starting over is the discipline: keep the old frames and add the corrections, compare per class, and promote only when the new version wins on the thing the line cares about.

The second month has a rhythm of corrections and versions

With the override path in place, the second month of a pilot settles into a pattern. The operator corrects a handful of frames a shift. The override rate is the number the quality manager watches, because when it rises on one class the line changed, whether a new panel variant or a lamp replaced. When corrections cross the project's threshold a new version trains on them, rolls out with no downtime, and the override rate falls back.

That is the pilot that survives, and by March it has learned the line from the people on it. The surface defect use case is where most stamping pilots start, and the same shape holds for a weld cell or a packing line. Validate against the person beside the camera, give that person a way to say no, and start on a line small enough to win.