What active learning is, and why it labels the frames that change the model

The archive from the camera over line 2 holds about fifty thousand frames from the last quarter. In almost every one of them a steel panel is moving down the conveyor past the inspection station with nothing wrong with it. The quality engineer has found nine frames with a split in the flange, which is the defect the model is meant to catch, and she found them by reading the scrap log and scrubbing to the timestamps.

Labeling all fifty thousand frames would cost a month and teach the model mostly that panels are fine. Labeling nine frames would teach it almost nothing. The question active learning answers is which frames in between are worth a person's time.

Fifty thousand frames of good panels teach the model one thing

A random sample of the archive is a random sample of good panels. Label five hundred of them and the model learns the panel, the conveyor and the lighting at line 2, and it learns that a split is rare, which is true and unhelpful. Its first version will find the obvious splits and hesitate on the rest, and the hesitation is where the useful frames are.

The frames that change a model are the ones it gets wrong or nearly wrong. A frame the model already handles well contributes a label that confirms what it knew. A frame it is unsure of contributes a label that moves the boundary. Most of the archive is the first kind, and a random sample is mostly wasted labeling.

Active learning scores the frames and sends the doubtful ones to review

The mechanism is simple to state. Train a first version on a small labeled set. Run it across the unlabeled archive. Score every frame by how unsure the model was, using the margin between its best guess and its second guess, or the disagreement between two versions trained on different splits. Sort. Send the top of the list to a person.

On line 2 the top of the list is not random. It is the frames where the flange edge is in shadow, the frames from the Monday the line ran the thicker gauge, and the frames with a scratch that looks a little like a split. It is also the frames with a split the engineer never found in the scrap log, because the panel was reworked rather than scrapped. Those are the frames a reviewer should see first, and the scoring found them in the fifty thousand without anyone scrubbing footage.

Two guards keep the list honest. A share of the frames sent to review is chosen at random, so the model's blind spots, the frames it is confidently wrong about, still have a route to a person. And the reviewer marks a frame as unusable when it is blurred or the panel is out of frame, so the model stops asking about frames nobody can label.

The reviewed frames are labels the first set did not have

A reviewer opens the queue, and each frame comes with the model's proposed box. She corrects the box, adds the split the model missed, or confirms the frame is a good panel. The labeling doc describes the verification pass that makes these labels usable for training. It is the same pass as for any label: a second person checks each one before anything trains on it, which is how labels come back at up to 99.9% accuracy.

The corrected frames are the highest-value labels the project will ever get. A label drawn on a random frame teaches the model something it may already know. A correction on a frame the model got wrong teaches it exactly what it did not.

My own view is that a team with a labeling budget of a thousand frames should spend a hundred on a first random set and the other nine hundred through this loop. Most teams spend all thousand up front and then wonder why the model never improves.

Retraining folds the corrections in without starting over

Version 2 trains on the first set plus the reviewed frames. Both, together. The retraining lesson covers why the old frames stay: a model trained on the corrections alone fixes the shadowed flange and forgets the ordinary panel, and the ordinary panel is most of what the line makes.

Version 2 keeps a record of what it trained on, so when the engineer asks why it now catches the thin-gauge splits and version 1 did not, the answer is a list of frames rather than a guess.

Then version 2 runs across the archive again, and its doubtful frames are different from the first version's. The shadowed flange is no longer on the list. The reworked panels still are, and so is a set of frames from the day the camera housing was cleaned. The list is shorter, and the frames on it are stranger, which is what progress looks like from inside the queue.

Object detection on the live line makes the cycle continuous

The archive is finite. The camera over line 2 is not. Once the model is watching the line, the same scoring runs on the live frames, and the doubtful ones come back to the engineer as they happen rather than at the end of the quarter.

LexData takes the object detection model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the camera over line 2, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The accuracy you launched with is the accuracy you keep.

The signal that the cycle needs to run faster is the correction rate. When the reviewer's corrections on the live frames rise, the line has changed in a way the model has not seen, a new coil supplier, a new lamp, a shifted camera, and the frames coming back are already the ones the next version needs.

Active learning on its own has three failure modes

The first is a queue nobody opens. The scoring finds the frames, and if the reviewer is not part of a shift routine the frames sit there and the model never changes. The queue is a job with a name on it.

The second is scoring that only ever asks about one thing. If the doubtful frames are all shadowed flanges, the reviewer labels shadowed flanges for a month and the model gets very good at one corner of the problem. The random share and a cap on how many frames of one kind go to review in a batch keep the set varied.

The third is treating the archive as the whole world. The fifty thousand frames are last quarter. Next quarter's frames have the new supplier's coil in them, and no amount of scoring inside the archive finds a condition that is not there yet. The quality engineer at line 2 keeps her scrap log open beside the queue, and when a defect appears in one and not the other, that is the frame the model needs next.