Label inspection with computer vision, six checks a carton passes before the case is sealed

The print-and-apply head on the food line puts a label on each carton at 3 pm as it passes. The camera a metre downstream sees the result: a carton, a label somewhere on it, and whatever the head did to the label on the way. The question the line has is whether that carton can go into a case, and the answer is six questions in order, each one asked of a smaller piece of the frame than the last.

The order matters. A carton with no label fails the first check and never reaches the second, and the crop from the first check is what the second one looks at.

01Object detection finds the label, or the carton is kicked before anything else runs

The first model puts a box on the carton and a box on the label, and the first rule is that a carton box with no label box inside it is a carton with no label. That happens when the head misfires, when the label stock runs out, or when the label went on and fell off before the camera. Two classes and one rule make it the cheapest check on the line, which is why it runs first.

You type the two classes once, Lexi proposes the boxes on frames from the outfeed camera, and the line's inspector checks them. The label box has to hug the label, because every later check is run on the crop it defines, and a loose box carries a strip of carton into the print check.

02The label sits inside its window on the carton

Placement is the label box measured against the carton box. The label's corners are expressed as fractions of the carton's width and height, so the check holds whether the carton is a little left or right on the conveyor. The allowed window is set once, from a carton the line lead called good on the Monday the camera went in. A label outside the window is skewed on the carton or applied too low, and the customer's receiving line will reject the case for it.

The package and label inspection use case describes every check on this list the same way: a region and a rule. The box localises the region; the rule does the rest, and it is written in the plant's words.

03The label is the right way up

Orientation is a tag on the label crop rather than a box: upright, rotated a quarter turn, upside down. A head that has been re-threaded after the 3 pm stock change is the usual cause of a rotated run, and a run is what it produces, so the rule is written for a run rather than a single carton. One rotated label is a kick; a dozen in a row is an alert to the operator with the frame, because the head needs attention before it labels the rest of the shift.

04The print is clean

Print defects are small: a smear where the ribbon dragged, a void where the head missed, a streak where a print element has failed. They are boxes on the label crop, tuned for small objects, and the crop is why they can be found at all, since a void the size of a character on a carton-sized frame is a few pixels.

The operators on that line call the fading labels a ribbon prints as it runs out ghosts, and the 3 pm ribbon change produces a few every shift, and a ghost is a print defect in the taxonomy even though every character is there. The first version passed them, because the labeler had been shown voids and smears and never a whole label at half density.

05The text matches the lot record

Content is read from the label crop into fields: the product name, the best-before date, the lot code. The check is a comparison with what the line controller says is running, and it is the check that catches the expensive mistake, a correct label from the wrong job left in the head after a changeover. The date on the label is compared with the date the lot record holds, and a mismatch is a kick and an alert, because the next carton will carry the same wrong date.

Once the fields exist, the question the quality team asks afterwards is asked of LexInsight in plain words: which cartons from this lot carried a date other than the lot record, and the answer comes back with the frames. Asking that question changes nothing in the model; it reads what the model already found.

06The barcode decodes to the right code

The last check is the barcode. It is found as a box on the label crop, decoded, and compared with the code the lot record expects. A barcode that will not decode is a print defect that the print check did not catch; a barcode that decodes to the wrong code is the wrong label. Either is a kick, and the rule for the second is written as a sentence with a critical severity, approved before it goes live, because a wrong code on a sealed case reaches the customer's warehouse before anyone sees it. On the lines we run that alert goes to the line lead's Slack with the frame.

I would kick at the first failing check and not run the rest, even though the later checks produce interesting counts. A line does not need to know four things wrong with a carton it has already rejected, and the time saved goes to the cartons that are still in play.

07A new artwork is a spec change, and it is labeled before the changeover

The failure that arrives on a marketing schedule is the new label. The artwork changes in the spring, the new label is correct, the head applies it perfectly, and the model has never seen it. The placement window is off because the logo moved, the print check flags the new colour as a smear, the content reader finds the date in a different corner. Every carton fails on the morning of the changeover. The drift catalog calls this the spec changed: the pixels are correct and every label the model owns is now wrong about what correct looks like.

The date is known months ahead, so the fix is done before it. A window of frames with the new artwork, from a trial run, labeled and checked by the inspector and folded in, and the model has evidence for the new label before the first case of it is sealed.

LexData takes the outfeed model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the outfeed camera the line already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. That is how 99%+ accuracy is maintained in production on the manufacturing lines behind our numbers, and the spring artwork is one of the reasons the loop keeps running after launch.

The alert written as a sentence is what tells the operator which head to look at when a run starts failing, with the frame attached.