Surface defect detection in manufacturing, from the camera to the grade

The inspection station at the end of stamping line 3 has a light tent, a camera looking straight down, and a panel passing under it every few seconds on the way to paint. An inspector used to stand there with a torch held at a low angle, because a scratch that is invisible under the overhead lights throws a shadow when the light rakes across it. The torch is the part of that job the camera has to replace first.

Everything else in this post follows from getting that light right.

A surface defect is a small thing on a big plain surface

The panel is large and mostly fine. The defect is a scratch a few pixels wide, a dent that only shows as a change in reflection, a pit the size of a full stop, a stain that is a slightly different grey. Glass has bubbles and chips. Fabric has slubs and dropped stitches. The common shape across all of them is a small region of wrong on a large region of right, at a contrast the lighting decides.

That is why the light is the first decision. A raking light from one side throws every scratch into relief and hides the ones running parallel to it, so a station that matters gets light from two sides, or a frame under each. A diffuse dome light flattens the reflections that make a dent look like nothing. A dark-field setup, where the camera sees only the light that a defect scatters, turns a chip on glass into a bright mark on black.

The camera is fixed, the light is fixed, and the panel arrives the same way at 6 am as it does at 6 pm. Frames from a phone under the shop lights teach the model about a station that does not exist.

Object detection finds the defect and a mask says how big it is

For most stations, object detection is the right first model. Each defect gets a box and a class, and the box carries the position on the panel, which is often half the decision. A scratch in the cosmetic zone of a door skin is a different matter from the same scratch on a face that will be hidden by trim. You type the classes once, Lexi proposes a box on every defect in every frame, and an inspector checks them before anything trains.

A mask earns its extra labeling cost when the disposition depends on size. A scratch under a certain length ships, over it is reworked, and only an outline gives the length. The surface defect detection use case takes that position: severity is size and location on the part, and a box cannot supply the size. The honest answer for a new station is to start with boxes, find out from the first month of reviews which classes are being decided on size, and add masks for those classes only.

Class definitions get written before anyone labels, in the inspector's words, with an example frame for each. Scratch and scuff are different classes or they are one class, and either is fine, but the decision is made once and the QA pass on every label holds every labeler to it. A class that two labelers draw differently is a class the model will draw both ways.

The dataset has too few defects in it, on purpose

Good panels outnumber bad ones by a wide margin, which is what a working line looks like and a poor way to train a detector. The model sees a great many clean panels and a handful of pits and learns that the safe answer is silence.

The fix is in how the frames are chosen rather than in the model. Frames with defects are kept at a far higher share than they occur, and the rare classes are sampled up until the model has seen enough of them to draw them. The clean frames are kept too, since a model that has never seen a clean weld seam will box the seam. The how much footage lesson covers the arithmetic. What a quality engineer needs to hear is that the rare defect class is the one the dataset is built around, and the plentiful clean frames are the background it is built against.

Labels on this kind of footage come back at up to 99.9% accuracy when a person checks every one, and on a defect dataset that checking is where the value sits, because a mislabeled pit is a pit the model has been taught to ignore.

The grade comes from written rules and the model supplies the evidence

The model does not grade the panel. It produces boxes with classes, and a set of rules written by the quality engineer turns those into a grade. No defects in the cosmetic zone and fewer than a written count anywhere else is an A. A single scratch over a written length in the cosmetic zone is rework. A crack anywhere is scrap. The rules are in a document the plant owns, and when the customer's spec changes the document changes and the model does not.

This split is the part most first stations get wrong, by asking the model for the grade directly. A model trained to output pass or fail has learned a rule nobody wrote down, and when the rule changes the whole model has to change with it. A model trained to find defects, with the rules kept outside it, survives every spec revision the customer sends.

In our manufacturing work the inspection models hold 99%+ accuracy maintained in production, and that figure is the detection, measured against inspectors' verdicts. The grade on top of it is arithmetic.

The unsure panel goes to a person with the frame

Between the panels the rules can grade and the ones they cannot is a band: a mark the model boxed at low confidence, a class the rules do not mention, a defect that straddles the cosmetic zone boundary. Those go to a person with the frame, the box drawn on it and the rule that could not decide. The inspector looks, grades the panel, and the verdict becomes a label.

LexData takes the inspection model through its whole life. You type what to look for, Lexi puts a box on every defect in every frame, and an inspector checks each label before anything trains on it. The model then watches the station camera, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. On a defect station the corrections are concentrated in the band, which is where the model was weakest, so each version is trained on exactly the panels the last one could not decide.

One aside. The inspector at the stamping station still keeps the torch in a pocket. On the panels that come back for review the torch comes out, because a raking light held by hand is still the fastest way to settle whether a mark is a scratch or a smear.

A change in the defect mix is a threshold problem first

A new coil supplier arrives in March and the pits go up while the scratches go down. The panels look the same and the classes mean the same, but how often each occurs has moved, and every threshold that was tuned around the old rate is now wrong. A rule that sent one panel an hour for review sends one a minute. The drift catalog calls this the defect rate changed, and the important thing is what it is not: the model has not degraded, and retraining it on the same classes will not move the review queue.

The signal is the correction rate and the override rate in the review queue, read weekly. A jump in doubted frames with no jump in corrections means the thresholds are wrong for the new mix and the model is still right. A jump in corrections means the model is seeing something it was not taught, and that is the case for retraining. Reading the two apart is most of the operator's job in the first year of a station.

My own opinion, which the model people I work with do not all share, is that the quality engineer should own the thresholds and the review queue outright, and that the model should be retrained only when the engineer asks. A station that retrains itself on a schedule hides the defect rate change behind a model change, and the engineer loses the one signal that told them the supplier had changed.