CPG inventory cataloging with computer vision, every package named without a model per SKU

The field merchandiser photographs the coffee bay at 9 am on the Tuesday visit, one frame, forty bags across four shelves. The brand wants to know what was on that shelf: which of its varieties, in which sizes, in how many facings, and which of the competitor's bags sat beside them. Last year that answer was typed into a form in the car park. The year before, it was a paper tally. Neither version was checked by anyone, and both are the inventory the brand's planners work from.

The photo has all forty answers in it. Getting them out is two problems, and only one of them is detection.

Object detection puts one box on every package before anything is named

The first problem is finding the packages. The coffee bay at 9 am has bags that touch, stack and lean, in two or three sizes, with the same brand's dark packaging running the width of a shelf. Object detection with a single class, package, gives a box round each bag, and the labeling rule is the one every dense shelf needs: one box per facing, tight to the front face, ending where the neighbour begins even when there is no visible seam.

The detector does not need to know what a bag is. It needs to find the rectangles, and a person checking the proposed boxes on a few hundred shelf photos is what teaches it the bay's seams. That is the same problem the shelf and planogram compliance use case describes, hundreds of near-identical classes on a grid, and the detector's job here is deliberately smaller than the compliance model's. It finds packages, and naming them is the second problem, which the detector never has to solve.

A question per crop reads the brand, the variety and the size

Each box becomes a crop, and each crop gets a question. The brand printed on the bag, the variety, the weight on the front panel, whole bean or ground. LexInsight answers a question about footage with the frames behind it, and asked per crop the question is small enough to answer well: this bag, this label, these words.

The read comes back as fields, and the crop stays attached so a person can check the field against the bag it came from.

This is where the model-per-SKU approach falls over before it starts. A detector trained to recognise a particular variety needs examples of that variety in every size and every lighting, and the brand ships forty varieties and refreshes the artwork on a quarterly schedule. A detector that finds packages and a reader that reads the front panel need neither. The package looks like a package, and the panel says what it says.

An aside from the coffee bay: the difference between whole bean and ground on the leading brand's bag is a small sticker on the bottom corner. It is the single field the reader gets wrong most often, because the merchandiser's thumb is over it in a tenth of the photos.

Normalising the read against the catalogue is where the work is

The panel says one thing and the brand's catalogue says the same thing in capitals with half the words abbreviated. The read has to become a catalogue entry. The matching has to survive a size printed as grams on one bag and ounces on another, a variety renamed last April, and a limited edition that exists on the shelf and nowhere in the master list.

Lowercase everything, strip the punctuation, keep the size and the variety, and treat a read that matches nothing as a question for a person rather than a silent guess. The unmatched crops are the review queue. A person opens them with the bag beside the fields, rules on the limited edition, and the ruling is a label the next pass learns from and a catalogue entry the planners did not know they were missing.

One detector and a reader beat a model per SKU

My own view is that a model per SKU is a maintenance contract nobody signed. Every new variety, every size change, every seasonal pack is a retraining job on that approach, and the brand's marketing calendar does not consult the model. A package detector that a person checks and a reader that answers a question per crop splits the work into the part that changes slowly, what a package looks like on a shelf, and the part that changes constantly, what is printed on it.

The retail work behind our figure, 4M+ annotations and validations, is mostly shelves boxed a facing at a time and checked by someone who knew which variety was which. The checking is the same whether the model is naming the bag or the reader is, and it is the part that does not get skipped.

The new pack design is the spec change the shelf cannot show

The refresh ships in October. The bags on the Tuesday photo are the new artwork, correct on the shelf, and the detector finds them fine because a package is still a package. The reader reads the new front panel fine because the words are still words. What breaks is the matching, if the new artwork moved the size to the back or renamed the variety, and the drift catalog calls that a spec change: the pixels are correct and every rule about them is now wrong.

The signal is the review queue filling with unmatched reads on the morning the new bags reach the shelf, and a person overriding the match in a consistent direction. That override rate is what tells the catalogue it needs the new entries, and the corrections at review are what retrain the detector on the new pack's seams. The refresh was on the brand's own calendar for a quarter, and the cheapest version is a handful of the new bags photographed in the warehouse a week before they ship.

The catalog is checked by a person before it is trusted

LexData takes the package model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the footage the merchandisers already take, in the cloud or on your servers. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The forty bags on the Tuesday photo become forty catalogue rows by Tuesday afternoon, each with the crop it came from, and the four the reader could not match are the four a person looked at.

The merchandiser stops typing in the car park. The form is still in the app, and for a month it is where the person checks the catalogue against the shelf, until the catalogue is what the form is checked against.