How to improve computer vision model accuracy by starting with one defect class

The packing line at the coffee roaster runs bags past a station camera on their way to the case packer. The first brief for the model, written in March, listed eight things to catch: torn bag, misprinted label, missing valve, wrong weight, open seal, crushed bag, foreign object, and a wrong lot code. Two months and a few thousand labeled frames later the model found some of each, badly, and the line lead switched it off because it flagged a good bag every few minutes.

Everything about the second attempt was smaller. One class, from the right camera, with a guideline and a number per shift. That model was trusted by the end of its first week, and the other seven classes arrived one at a time afterwards.

One class first, the torn bag, with the cost of each error written down

Of the eight on the March brief, the torn bag was chosen because a torn bag that reaches a customer is the complaint the roaster hears about, and because a tear is visible from the station camera on every frame it appears in. The wrong weight was struck; it is a scale's job. The wrong lot code was deferred; it is a reading problem with its own camera position. The other five were queued behind the first.

Before a frame was labeled, two costs were written down. A missed tear costs a returned case and a phone call. A false alarm costs an operator a walk to the station, and after the third false alarm in a shift the operator stops walking. Those two sentences set the threshold, the review policy and the alert cadence. They are the reason the second model was trusted: it was tuned to be slow to flag and quick to send a doubtful bag to a person rather than to the line lead's phone.

The frames come from the station camera with the glare left in

The first attempt's training set was assembled from photographs. Some were taken by hand at the station with a phone, some came from the packaging supplier's catalogue, some from the web. None was taken by the camera that would feed the model, from its mounted height and through its own lens, under the strip light that puts a glare bar across every foil bag at the 2 pm shift change.

The second set was pulled from the station camera's own stream, across three shifts and both bag sizes, with the glare in it. The how much footage lesson is the arithmetic: distinct conditions matter more than volume, and the conditions that matter here are the line's own. A model that has never seen the 2 pm glare will flag a good bag every afternoon for as long as it runs.

Negative frames went in on purpose. Bags with no tear, in every condition, so the model learned what a good foil bag looks like under glare, rather than learning that glare means tear.

The written guideline is the ceiling on accuracy

A tear on a coffee bag can be a pinhole at the valve, a split at the seal, or a gash down the gusset, and three packers asked to label tears drew three different things. The first attempt's labels were an average of three private definitions, and no model trains past its labels.

The second attempt began with a page: what counts as a tear, the smallest one that matters, whether the box hugs the tear or the bag, and what to do when the tear is hidden by a fold. Lexi proposed a box on every frame from that page, and a person checked each label against it before anything trained. The labeling doc describes the pass; the guideline is what makes the pass mean the same thing on Tuesday as it did on Monday. Where labels are checked this way they come back at up to 99.9% accuracy, and the ceiling moves out of the way.

The one-way valve on a coffee bag exists because roasted beans give off carbon dioxide for days after roasting, and a sealed bag without one would swell. The valve is also the place most pinhole tears start, which the guideline says and the first attempt's labels did not.

A baseline before augmentation says what the frames are worth

The instinct after a poor first model is to add augmentation, flips and colour shifts and synthetic glare. The April attempt did the opposite. It trained a plain baseline first, on the station frames and the checked labels, with no augmentation at all. That number said what the frames and the labels were worth on their own, and it was already better than the first attempt's best, which was the evidence that the frames and the guideline had been the problem rather than the architecture.

Augmentation came after, one kind at a time, each kept only if the per-class number moved. Most did not. A horizontal flip helped, because a bag can lie either way on the belt. A colour shift hurt, because the roaster's two bag colours are the only two the line will ever see and a model taught to ignore colour learned to ignore a real cue.

Evaluate per class and per shift rather than on the aggregate

An aggregate accuracy figure on the first attempt hid everything. The model was fine on crushed bags and useless on tears, and the average looked respectable. The second attempt was scored on one class, so there was no average to hide behind, and it was scored per shift, because the 2 pm glare is a condition and a condition deserves its own number.

The split was by time. Train on frames up to a date, test on the fortnight after, so the score measured the model on bags it had never seen rather than on neighbours of bags it had. That number was lower than a random split would have given and it was the number the line lead could believe, because it matched what the operator saw at the station.

Most computer vision projects add the second class too early

The pressure to add the misprinted label came in week two, from the March brief. It was held until the tear model had run for a month and its doubted frames had been reviewed, because those doubted frames were where the second class would come from. Among the frames the tear model was unsure of were bags with a crooked label the model half-read as a tear, and those, corrected, were the seed of the label class.

My own view is that most computer vision projects that fail on accuracy did not have an accuracy problem. They had eight problems, each starved of the frames and the attention one problem would have received. I would rather ship one class the line trusts than eight the line switches off, and the second class is cheaper to add once the first is running.

The loop adds the second class from the frames the first one doubted

LexData takes the bag model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the station camera the line already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. Adding the misprinted label is a new sentence typed into the same project, and the frames it trains on are the ones the tear model already sent back.

The seven classes from the first brief arrived over the following months in the order the doubted frames suggested, the crooked label, then the open seal, then the missing valve. Each got its own line on the guideline and its own number per shift, and the package and label inspection use case is the same shape at any station where a bag or a carton passes a camera. The manufacturing work behind our numbers holds 99%+ accuracy in production, and the roaster's tear model is what that looks like from the station.