Training a model for a defect that appears three times a month

The test engineer on the board line found the first three cracked boards in February, all from one panel, all with a crack a hair wide running from a mounting hole toward the nearest track. She keeps them in an anti-static bag in her desk drawer, labelled "the three". By June the bag holds eleven. Eleven boards, from a line that has built a hundred thousand in the same months, is the entire positive class for the defect the customer cares most about.

A model cannot be trained on eleven of anything in the ordinary way. It can be trained on eleven if every decision about the frames is made with the eleven in mind.

Three a month is the whole positive class

A training set sampled from the line's footage since February at random holds a hundred thousand good boards and eleven cracked ones, and a model trained on it learns, correctly given what it was shown, that cracks do not happen. It reaches a near-perfect accuracy by calling everything good. The surface defect detection use case says the same about scratches on metal: the good parts vastly outnumber the bad, so the set is unbalanced by construction, and the imbalance has to be handled on purpose.

The rare class also sets the calendar. The how much footage you actually need lesson puts it plainly: the timeline is governed by the least common class you promised to detect, and eleven is the number the board line has.

Tiles keep the crack's pixels through the network

The camera over the board line is a high resolution unit, chosen in March so a crack a hair wide is a few pixels across in the full frame. A detector does not see the full frame. It resizes the image to its input size, and by the time the board has been shrunk to fit, the crack that was four pixels wide is less than one and has vanished before the first layer sees it.

Tiling is the fix. The full frame is cut into overlapping tiles, each one small enough that the resize barely touches it, and the detector runs on every tile. The crack keeps its four pixels. At inference the same tiling runs on every live frame, with the overlap chosen so a crack on a tile boundary appears whole on at least one tile. It costs time per frame, and on a board line sampling about every two seconds that time is available.

The test engineer's eleven boards were photographed on the line camera at full resolution before anything else was done, because a crack photographed on a phone at the bench is a different object from the same crack under the line's light.

Rare frames are kept and weighted, near misses beside them

None of the eleven from the February bag is discarded, however awkward the frame. The one where the crack runs under a component shadow, the one on a board that was also slightly rotated in the fixture, are the ones the model most needs, because the cracks on the line will not arrive under ideal conditions either. Each rare frame counts for more in training than a good board does, so the eleven are not drowned by the hundred thousand.

What goes in beside them matters as much. The line has hundreds of boards with a scratch across the solder mask that is not a crack, a stray fibre that looks like one, and a hairline of flux residue near a mounting hole. Those near misses are labeled as what they are, and they are the frames that teach the model the difference between a crack and everything that resembles one. I would argue the near-miss frames are worth more per frame than the cracks themselves, because there are enough of them to learn a boundary from.

Every box is checked by the test engineer before training, with the crack boxes drawn tight. A box that includes a border of good board on each side of a hair-wide crack is mostly good board, and the model learns mostly good board.

A recall target per defect class replaces the accuracy figure

The number the customer wants is how many of the cracked boards the model catches, and that is recall on the crack class, measured on the held-out cracked boards alone. An accuracy figure across all boards is dominated by the hundred thousand good ones and says nothing about the eleven. The reporting should refuse to print it. Manufacturing lines that hold 99%+ accuracy in production do so class by class, and on the board line the only class that matters for the customer is the one with eleven examples.

The threshold follows from the same asymmetry. A crack that ships is a field failure; a good board sent to review is a minute of the test engineer's time. So the threshold is set low on the crack side, the review queue is expected to hold mostly good boards, and the false send rate is reported beside the recall rather than hidden.

The doubted frames are how a rare class grows

The model trained on eleven will be unsure of the twelfth, and the review queue exists for the twelfth. A board with something near a mounting hole that the model cannot place comes back to the test engineer with the frame and the box, she decides, and her decision is a label. Half the time it is a crack and the bag in her drawer grows; half the time it is flux, and the model has learned one more thing flux looks like.

LexData takes the board model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the line camera, in the cloud, on your servers or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. For a class with eleven examples, the review queue is the only place a twelfth can come from, and every version trained on the corrections is a version with a slightly larger positive class.

A changed crack rate is a threshold finding first

In September a new panel supplier comes in and the cracks stop, for six weeks, and then come back at twice the old rate on a different corner of the board. The model finds them. What changes is the volume in the review queue, first falling to nothing, then flooding, and the temptation is to conclude the model has drifted.

The drift catalog calls this the defect rate changed, and it is the condition most often mistaken for a model problem when it is a threshold problem. Per-board the model is as good as it was. The base rate moved, and the thresholds tuned around three a month are wrong for twelve a month. The fix is to re-tune against the current rate before anyone retrains, and to tell the supplier about the corner, because the camera has found something about the panels and not only about the model.