How to detect small defects with computer vision when three resizes erase them

The blade at final inspection on a Thursday in March is a metre and a half long. The crack the inspector finds with a torch held at a low angle is a hair's width and about the length of a fingernail, running from the root fillet along the pressure face. The display panel two buildings over is the size of a phone, and the scratch the inspector finds by tilting it under a polarised light is a few microns deep and invisible from any other angle. In the frame from each station's camera, the crack is about four pixels wide and the scratch is about two.

Both models found the large defects in the first week. Neither found these, and the reason is arithmetic rather than learning.

A crack four pixels wide does not survive three resizes

A detector has an input size, and a frame from the blade station's high resolution camera, fitted in January, is far larger than it. So the frame is resized to fit, by a factor that on the blade camera is about four in each direction. A crack four pixels wide becomes one pixel wide, which is to say it becomes a slight change in the value of one pixel, and the first convolution averages it away with its neighbours. The scratch on the display, two pixels wide, is gone before the resize is finished.

The model has not failed to learn the crack. It has never been shown one. Every training frame went through the same resize, so what the model saw at training was a blade with no crack, labeled as having one, and it learned that the label means nothing. The what a vision model can and cannot see lesson makes the point in general: a model sees what is in the pixels it is given, and a defect that is smaller than the pixel grid after resizing is not in them.

The labeler has to see it first, at full resolution

Before any of the training tricks, there is a plain question: can the person drawing the box see the March crack. On a resized frame on a laptop screen, the answer is usually no, and a labeler who cannot see the crack draws the box where the inspector's note says the crack was, which is a box on clean blade with a crack somewhere inside it. The model learns clean blade.

So the labeling happens on the full-resolution frame, zoomed until the crack is a visible line, and the box hugs it. Lexi proposes candidates on the full frame, and the inspector who found the crack with the torch checks each one, in the verification pass the labeling guide describes. Her un-boxing matters as much as her boxing: a machining witness mark on the pressure face, boxed as a crack, teaches the model that witness marks are cracks, and the blade has hundreds of them.

The display inspectors tilt the panel under the polarised light to find the scratch. The camera at the station had to be given the same angle and the same light, because a scratch that is invisible to the inspector from straight on is invisible to the camera from straight on too.

Tiles keep the pixels at training and again at inference

Tiling is the fix for the resize, and it went onto the blade line in April. The full frame is cut into tiles, each small enough to fit the detector's input with little or no resizing, and the detector trains on the tiles. The crack keeps its four pixels. At inference the same tiling runs on every frame from the station camera, the detector runs on every tile, and the boxes are stitched back onto the full frame in the coordinates the inspector's report uses.

It costs time. A frame that was one pass through the detector becomes sixteen or more, and on a station sampling about every two seconds that time has to fit. On the blade line it does, because a blade sits at final inspection for minutes. On a faster line the tile count is the number to argue about, and the argument is between coverage and speed with no third option.

The tile overlap decides whether an edge crack is found

A crack that runs across a tile boundary, as the March crack did, is two half cracks, one in each tile, and each half may be too short for the model to call. The tiles overlap for that reason, by more than the longest defect the line expects, so that every crack appears whole on at least one tile. The overlap also produces duplicate boxes where two tiles both find the same crack, and those are merged on the full frame before anything is counted.

I would spend the budget on a better lens before a bigger model, on both these lines. A detector that is given eight pixels of crack instead of four has an easier job than any architecture change can give it, and the lens is a one-time cost where the model is a recurring one. The surface defect detection use case carries the same warning about defects that are small and low-contrast on reflective surfaces: the camera decides what is possible before the model is chosen.

A missed label teaches absence, so every label gets QA

On a line where the defect is a few pixels wide, a labeling mistake is worse than usual. A missed crack in the training set is a frame of blade with a crack in it labeled as clean, and the model learns, correctly given what it was shown, that this pattern of pixels is nothing. Enough of those and the model is confident about clean blades that are not. So every label on both lines gets a second look by a person before training, and the frames with no defect get the same look as the frames with one, because the clean frames are where the misses hide.

That pass is what holding 99%+ accuracy in production means on a manufacturing line where the defect is four pixels wide: the accuracy is made in the labels, and the labels are made by someone who could see the crack.

Doubted tiles come back and the rare class grows

The model trained on the first hundred cracks will be unsure about the hundred and first, a line along the fillet that could be a crack or a witness mark. That tile comes back to the inspector with the box, full resolution, and her verdict is a label. On the display line the doubted tiles are mostly dust, and each one the inspector clears is a piece of dust the next version leaves alone.

LexData takes the station's model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the station camera, in the cloud, on your servers or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. For a defect that appears a few times a month and is four pixels wide, the review queue is the only place the training set can grow from, and the inspector with the torch is the only person who can grow it.