Preventing food recalls with a camera at the labeller and one over the belt

The ready-meal line runs the chicken curry on Thursday mornings and the vegetable korma after lunch, and the sleeves for both come off the same labeller. At the changeover a reel of curry sleeves is left in the magazine for the first few hundred kormas. The kormas are fine. The sleeves say curry, list the allergens for curry, and go into the same distribution centre as everything else. Somebody with a nut allergy buys one the following week.

That is the shape of most food recalls. In most years the regulators' figures put undeclared allergens at the top of the list, and an undeclared allergen is a label problem. The rest are things in the product that should not be there. A camera can see both, and it needs to be two cameras.

The labeller camera matches the label to the run

The first camera looks at every pack as it leaves the labeller on Thursday morning. The model boxes the label, and inside the box the product name, the allergen statement and the batch code are read into fields. The fields are then compared with the run: the line's scheduling system says korma is running, the label says curry, and the pack is wrong before it reaches the case packer.

My view is that the comparison should be against the run's recipe file and never against the previous pack. Comparing pack to pack catches a torn label and misses the changeover entirely, because every pack after the changeover agrees with the one before it. The recipe file knows what should be on the sleeve.

The package and label inspection use case describes the shape: every check is a region and a rule. The label is inside its window, the code decodes and the fields match the run.

The allergen statement is checked for presence and for legibility

Presence is a box. The allergen statement occupies a known region of the sleeve, and a sleeve where that region is blank, covered by a promotional sticker or torn is a pack that fails without reading a word. Legibility is harder and matters as much. A printer running low on ink produces a statement that is present, correctly worded and unreadable at arm's length, and a customer who cannot read the word peanut has not been told.

So the read has to return a legibility judgement as well as the words, and the frames a person labels are the faded ones, the ones with a fold across the statement, and the ones where a reflection off the film sits over the text. In LexAnnotate you type "allergen statement, batch code", Lexi puts a box on every frame, and a person checks each label before anything trains on it. On the manufacturing lines we run, 99%+ accuracy maintained in production is the figure for this kind of check, and it is maintained by the person reviewing the doubtful reads.

The allergen box is printed in bold on most sleeves for the same reason the model is asked about legibility: the words are only useful if they can be read.

Object detection over the belt boxes the foreign object

The second camera looks down on the open trays before the film goes on. Here the question is whether anything is in the tray that is not food. Object detection is the right tool for the things the line has seen before: a fragment of blue glove, a piece of plastic from a torn bag, or a bone in a product that should have none. The Thursday curry run has produced all three at some point. Each is a class, labeled from the line's own frames, and each labeled example is a frame where a person confirmed the thing was there.

The rare ones are the hard ones. A line may go months without a glove fragment, so the training set has a handful of them against a season of clean trays, and a person goes looking for those frames deliberately. A model that has never seen a particular object still returns a doubtful frame when something strange is in the tray, and doubtful frames go to a person rather than into a pack.

The alert holds the batch while a person checks

An alert in LexAlert is a rule written as a sentence, with a severity and a cooldown, approved before it goes live. For the labeller it reads: a label that does not match the run, or an allergen statement missing or unreadable, critical, to the line lead's screen and to the quality channel, with the frame. The frame shows the sleeve with the fields boxed and the mismatch named, and the line lead can see in one glance that the magazine still holds the curry reel.

The hold itself is the line's decision, made by the PLC and the line lead according to the procedure the plant already has. The alert tells them which packs, from which time, on which frame. A recall traces back to a batch, and the frame link is the evidence that the batch was checked and what was found.

LexData takes both line models through their whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the two cameras, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

A formulation change is a label update footage cannot show

The korma recipe changes in November to a paste that contains cashew. The sleeve artwork is updated, the allergen statement now lists nuts, and the new sleeves arrive on a reel that looks almost exactly like the old one. The camera at the labeller sees packs that match the previous week's sleeves perfectly, and every one of them is wrong for the run.

The drift catalog calls this the spec changed: the pixels are identical and the right answer moved. Nothing in the footage signals it, so the signal has to come from the recipe file, updated before the first run of the new formulation, and from the rule comparing against it. When the file is updated and the reel is not, the first pack fails, which is the failure the line wants.

The reviewer's overrides are the other signal. If the line lead starts overriding label alerts in a consistent direction, the recipe file and the sleeves have parted somewhere, and that pattern is worth more than any accuracy figure. Corrections at review retrain the label model on the new artwork, and the frames from the changeover are the ones it learns from.