Computer vision for office supply retail, empty pegboard hooks and the camera already watching them

The pen aisle at the office supply store is a wall of pegboard from waist height to the top shelf, each hook holding a blister pack of one product, and at 10 am on a Monday about a tenth of the hooks are empty. Some have been empty since Friday. The store knows which ones when a customer asks, or when the merchandiser walks the aisle on Wednesday with a handheld and a list. At the end of the aisle, mounted for the door, the security camera has been recording the whole wall since the store was fitted out.

Nobody has asked it anything.

Object detection on a pegboard is a box per hook and a class for empty

The pegboard makes the detection problem unusually clean. Every hook is at a fixed position on a grid, so the boxes are drawn once and never move. The question per hook per sampled frame is whether there is a pack on it, and if so which one, and the classes are the product on the hook and "empty". Object detection with the pack classes plus the empty class covers the wall, and the empty class is the one the store wanted.

The toner shelf across the aisle is the same job on a shelf instead of a grid: a box per facing, a class per SKU on the box, and empty as a class of its own. The shelf and planogram compliance use case covers the harder version of this, hundreds of near-identical products on a grocery shelf, and a pegboard aisle is that problem with the answer partly given, because the hook tells you where the product should be.

You type "blister pack" and "empty hook" once, Lexi proposes the boxes on every frame, and a person checks them. On a pegboard the checking is mostly the hooks a customer has moved a pack onto from the hook next door, which is the aside from this aisle: customers rearrange pegboard constantly, and the planogram is a suggestion by Tuesday.

A facing that stays empty past the cooldown becomes an alert

An empty hook on one frame is a customer holding the pack. An empty hook on every frame since Friday is a stockout. The difference is time, and the rule carries it. The alert is a rule written as a sentence, "a hook on the pen wall empty for more than a day", with a severity and a cooldown, approved before it goes live, delivered to the store's morning list with the frame of the hook attached. The cooldown means one message per hook rather than one per frame, and the day-long window means the pack in a customer's hand never fires it.

The list on Monday morning names eleven hooks and two toner facings, with a frame each. The person with the list walks the aisle once. The monitoring guide covers how the rule and the delivery are set, and the store's part is to decide what "too long" means for a hook of highlighters versus a facing of the toner that sells one a day.

My own view is that the empty-hook rule is worth more than any count of pens, and that stores get this backwards. A count is a number for a report. An empty hook with a frame is a task for the person standing in the aisle.

The security camera's framing was chosen for the door, and that is fine

The camera at the end of the aisle was mounted to see the door and the tills, and the pen wall is in the frame because it happened to be between them. The angle is oblique, the far end of the wall is small, and the lens is the one the security installer chose. None of that stops the model. It means the frames the model trains on have to come from this camera at this angle, and the hooks at the far end are the ones that come back for review most.

That is the retail work in its usual shape. The retail figure behind our numbers, 4M+ annotations and validations, is mostly frames from cameras that were already on the ceiling, boxed as they were rather than as a vision engineer would have mounted them.

The second store is a new site with different pegboard and different light

The model that works in the first store goes to the second in the chain. The second store's pen wall is a different pegboard vintage, mounted a foot lower, under fluorescent tubes instead of the first store's LEDs, with the security camera on the other side of the aisle. The model's detections in the second store differ from the first from the opening day rather than degrading into it, and the drift catalog calls this a new site came online.

Compare each store against its own history. A chain-wide average of empty hooks per day will look fine, because the first store carries the second. The fix is a window of frames from the second store's camera, labeled and folded in before the store goes live, and the model that has seen two pen walls reaches the third with more to go on. The rollout that stalls is the one where the second store decides the model is bad, when it is doing exactly what a model trained on one wall does.

The store's own corrections are what the model learns from

LexData takes the aisle model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the security camera the store already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The hooks the customers rearranged on Tuesday come back as doubted frames, the person with the list rules on them, and the model that watches the wall in November has seen a year of customers.

The merchandiser still walks the aisle on Wednesday. The handheld list is shorter, and the hooks on it are the ones the camera flagged on Monday that nobody got to.