Chocolate box inspection with a camera over the tray line

The assortment box on the tray line holds two dozen pieces in paper cups, laid out in the pattern printed on the lid's insert. The line fills a tray, a person at the end glances at it, and the lid goes on. The glance is good for the first hour. By the end of the shift a caramel in the praline's cup goes through, and the customer who opens it in December knows the pattern better than the person who packed it, because the customer read the insert.

At a few hundred trays an hour there is no amount of glancing that holds up. A camera over the line looks at every cup of every tray, and it does not care what hour it is.

Object detection names each piece, and the template says where it belongs

The model's job is to put a box on every piece and say what it is: praline, caramel, coconut, the dark truffle, the milk one. That is object detection with a class per piece type, on a fixed camera at a fixed height, with the tray always in the same place. It is one of the more forgiving detection problems there is, on a Monday or a Friday, until two pieces look alike.

What the model does not do is know where the praline goes. That is a template, written down once: a list of rows, each row a list of piece names, left to right, exactly as the insert shows. The check sorts the detections into rows by their height in the frame, then left to right within each row, and compares each position against the template. A mismatch is reported by position: second row, third from the left, expected praline, found caramel.

My own view is that this check should stay code a line engineer can read in a minute, with no learning in it. The model is probabilistic and can be wrong. The template is the spec, and when a tray is rejected the reason should be something a person can point at.

A few dozen frames per piece type is enough on a fixed camera

Because the camera, the light and the tray position never change, the first model needs less footage than most people budget for. A few dozen labeled frames per piece type is a working first version on a line like this, and the lesson on how much footage you actually need explains why constrained scenes are the cheap case.

The exception is the lookalikes. A milk caramel and a milk praline in the same cup shape, under the same light, differ by a swirl on top, and those two classes want more frames and a person who knows the difference checking every box. You type the piece names once, Lexi proposes the boxes, and the person checking them is the one who has been packing this assortment for years and can tell the swirl at a glance.

Naming is the part that bites later. If one labeler writes "Praline" and another writes "hazelnut praline", the template check has two classes where the model has one, and every tray fails.

Damaged is a class, and the hardest one to label

A piece can be in the right cup and still be wrong: cracked, bloomed grey on top, misshapen from a warm tray, or missing its decoration. The line wants those trays held as much as the swapped ones, so damage is a class of its own, alongside the piece types.

It is also the class the labelers disagree on most. A hairline crack in the shell, a small bloom at the edge, a truffle that settled slightly off centre in its cup. The boundary between fine and damaged is a decision the plant makes, and the disagreements at labeling are where that decision gets written down, usually by the QA lead on a Friday afternoon with the two labelers and a tray of examples. The reviewer's verdicts on the doubtful frames are the boundary the model learns, and they are worth arguing over before the first version trains rather than after.

A missing piece and a missed detection look the same in the count

A tray that comes up one detection short of two dozen has either an empty cup or a piece the model did not see. The two need different responses, and the check has to tell them apart before it rejects the tray. The plain rule is to count first: if the count is short, look at the cup the template expects at the gap, and if the model was unsure about anything in that region the tray is held for a person rather than rejected.

That hold is the review queue. A frame the model doubts comes back to a person with the box drawn, usually within the same shift, and the person's answer is a labeled frame for the next version. On the tray line that person is the packer at the end of the belt, who was looking at trays anyway and now looks only at the held ones, at 10 am and again after lunch.

The line's own name for a piece that has escaped its cup and is sitting between two others is a traveller. Travellers cause most of the doubtful frames.

A new assortment is a spec change that needs a relabel

The failure that arrives on the calendar is the seasonal assortment. In March the spring box replaces two pieces with new ones, the insert changes, and the template changes with it. The camera sees a piece it has never seen and either boxes it as the nearest lookalike or boxes nothing, and every tray fails the template check for the first hour of the run.

The drift catalog calls this a spec change: the frames are fine and the right answer moved. The template update is the easy half. The new pieces need labeled frames before the run, a few dozen each, and the tell that somebody forgot is the packer overriding a held tray every few seconds until it is done.

LexData takes the tray model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the camera the line already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

The assembly verification use case is the same shape with fasteners instead of pralines: a detector that names the parts, a template that says where they go, and a person for the trays the two cannot agree on.