Medical device label inspection reads the lot number only after finding the label

At the end of line 4 a handheld device comes out of its tray face up, its screen showing the self-test value from the last station and a printed label on the back with a lot number and an expiry date. An inspector picks up one device in each tray, reads the screen, reads the label, checks both against the batch record on the clipboard, and puts it back. The line runs at a pace of about one device every 3 seconds, and the inspector's sample is a fraction of what goes into the cartons.

A camera over the tray sees every device. The problem is that a camera reading a whole frame reads everything in it.

Object detection finds the label before anything reads it

A reader pointed at the full frame returns the lot number, the expiry, the stencil on the tray, the safety notice on the conveyor guard, and the lot number from the device in the next tray over. It returns all of that as one run of text with no idea which belongs to which. The pipeline that holds up on line 4 finds the label first and reads it second.

Object detection is the first stage: two classes, screen and label, a tight box on each, on every sampled frame. The boxes are the whole product of this stage. What they contain is somebody else's job.

You type the two classes once, Lexi puts a box on the screen and the label in every frame, and a person checks the boxes before the model trains. The labeling doc covers the verification pass; on this line the checker's attention goes to the box edges, because a box that clips the last digit of the lot number is a box that reads wrong on every frame afterwards.

The crop is read into fields by a question

The second stage takes the crop inside each box and asks LexInsight a question in plain words: what is the lot number on this label, what is the expiry date, what value is on the screen. The answer comes back with the frame behind it, and it comes back as fields rather than as a run of text, which is what the batch comparison needs.

A reading that is only a string is not yet a result. The lot number has a format the plant defined, eight characters with a letter in the third position, say. The expiry is a date that has to fall inside the product's shelf life from today. The screen value has a range. The fields are checked against those rules before anything is called a pass.

Somebody on line 4 writes the day's lot number on a whiteboard at the head of the line each morning, and the inspector glances at it more often than at the clipboard. The rule the pipeline checks against is that whiteboard, kept in the batch system where the camera can read it.

An implausible value goes to a person rather than into the record

A lot number that does not match the batch format is not a fail. It is a frame the pipeline is not sure about, and it goes to the person covering line 4 with the crop, the reading and the format it failed. Most of the time the person sees a smudged digit, a glare band across the screen from the light over the tray, or a label caught mid-turn as the device settled. Occasionally the person sees a wrong label, and that is the frame the whole line was built to catch.

The package and label inspection use case has the same shape, and names the constraint: line speed. The decision has to be made in the gap between devices, on frames that are sometimes blurred, and a false reject stops the line as expensively as a false accept lets a bad device through. Routing the doubtful reading to a person rather than stopping the line on it is what keeps the false reject cost down while the pipeline is young.

The two stages are scored separately

This is the part I would argue for over anything else in the design. If the pipeline is scored end to end, the number you get is how often the right lot number came out, and when it drops you cannot tell whether the detector missed the label or the reader misread it. The fixes for those are different and the frames that fix them are different.

So the detector is scored on its own: how many labels in the checked frames it boxed and how many it missed, and how tight the boxes were. The reader is scored on its own: given a correct box, how many characters it got right. A missed label is a detector correction and goes back into the detector's training set. A misread digit under glare is a reader problem, and the fix is usually the light over the tray rather than anything in the model.

On the manufacturing lines we run, the figure we hold to is 99%+ accuracy maintained in production, and it is held by keeping those two scores apart, because a single blended number hides which half is slipping.

Unreadable frames come back as corrections

LexData takes the label model through its whole life. You type what to look for, Lexi puts a box on every screen and label in every frame, and a person checks each label before anything trains on it. The model then watches the camera over line 4, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

The frames that come back in the first month have a pattern. A device rotated a quarter turn in its tray so the label is foreshortened. A screen in its power-saving dim state. A new label stock with a gloss finish that throws the tray light straight back at the lens. Each one is a box corrected by a person, and when the corrections cross the project's threshold a new version trains, with the gloss labels in it.

The alert is a mismatch against the batch record on the line

The reading that passes its format check is compared with the batch record, and the alert is written as a sentence: a lot number on line 4 that does not match the open batch, high severity, no cooldown, approved before it goes live. The monitoring and alerts doc covers how a model is attached to a feed and how the alert is described. What arrives is the frame with the label boxed, the reading, and the batch record's lot number beside it.

That alert stops the line. A misread digit routed to a person does not. Keeping those two outcomes apart, on the same camera and the same frames, is what the two-stage design is for.