Human-in-the-loop computer vision, the mistakes the model is sure about and the inspector who catches them

The inspector on the coating line, on the 6 am shift, sees the frame on the screen above the outfeed a second after the camera does: a panel, a box, the word "blister" beside it. She looks at the panel in her hand, which has an oil sheen where the box is and no blister, and presses the pedal under the bench. The whole judgment took three seconds and she did not stop working to make it.

That press is the most valuable thing the line produces for the model all day. It is also the thing most production systems throw away.

The inspector can tell in seconds, and that verdict is the signal

Nobody on the line needs to know what a loss function is. The person who has been grading coated panels on line 4 for nine years needs to answer one question: was the model right about this one. Yes or no, with the frame attached, in the time it takes to look up.

That is a lighter thing than labeling. Drawing boxes on a frame is a labeler's afternoon. Saying a box is wrong is a glance, and it can be done by the person who is already looking at the part, at the moment they are looking at it. The verdict lands beside the detection it was about, with the line, the shift and the time, and that is a record the model's next version is built from.

On this line the flag is a foot pedal, because both her hands are in gloves.

The confidently wrong frames are the ones no threshold would catch

The usual plan is to review the frames the model was least sure of. It is a good plan and it misses half the problem. A model can be entirely sure and entirely wrong. That happens when it has learned something that correlates with the defect rather than the defect itself: the oil sheen that was on most of the blistered panels in the March training frames, and is on this panel with no blister at all.

Those frames never come back on their own. The model has no doubt about them, so no threshold surfaces them, and the systems downstream trust them for exactly that reason. The inspector is the only thing in the building that catches them, and she catches them because she is looking at the panel and the model is looking at the sheen.

My own view is that a hundred pedal presses a day from one inspector are worth more than a thousand low-confidence frames pulled by a rule. The rule finds the frames the model was already unsure of. The inspector finds the ones it does not.

The correction rate is the drift signal

Put those verdicts on a chart per week and the chart is the model's health. A flat rate means the model is holding on the panels the line is making. A rate that climbs means the world under the camera has changed: a new primer, a lamp replaced, a supplier's coil that marks differently. The climb shows up before any accuracy figure moves, because the inspector on line 4 is grading every part and the test set is grading none of them.

The lesson on what model drift actually is draws the same line: the signal is people overriding the model in a consistent direction, and it is worth logging even when everything looks fine. The surface defect detection use case is the setting where it matters most, because the defects change with the material and the material changes with purchasing.

When the corrections cross the threshold, a new version trains

The pedal presses do not sit in a spreadsheet. Each is a corrected label on a frame, and when the corrections in a project cross its threshold a new version is trained on them, compared against the old one on the same frames, and rolled out. The old version is kept with what it was trained on, so the question "what changed between April and May" has an answer.

LexData takes the coating model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the outfeed camera the line already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. Across the manufacturing lines we run that is 99%+ accuracy maintained in production, and the inspector with the pedal is who maintains it.

The oil sheen stopped reading as a blister two versions after she started flagging it. Nobody retrained anything by hand.

Feedback that never reaches the training set is the common failure

The way this fails is quiet. An operator emails a screenshot. An engineer keeps a folder of bad frames on a laptop. A site two hundred miles away flags the same failure with a different word for it, and the two sets never meet. Six months later the model is retrained on whatever the ML team could find, and the pedal presses from the coating line are not in it.

Three things stop that. The verdict lands next to the detection automatically, so there is nothing to email. Every site uses the same definition of wrong, written once, so a blister at plant A is a blister at plant B. And the path from a flagged frame to the next training run is short enough that the inspector sees her corrections come back as a model that no longer makes that mistake, which is the only thing that keeps anyone pressing the pedal.

A loop that closes in a quarter is a loop nobody believes in. The inspector on the coating line believes in hers because the sheen stopped being a blister within a month.