Computer vision inventory monitoring from a frame every few minutes

The dairy case in a mid-sized store is counted twice a week, on Sunday night and Wednesday night, by two people with a handheld scanner and a printout. Between those counts the store's system holds whatever the last count and the till receipts add up to. That is close on Monday morning and wrong by Wednesday afternoon, because milk gets dropped, moved to the wrong facing, and put back by customers in places no receipt records.

The camera over the aisle has been looking at that case the whole time. It was fitted for loss prevention, and it records the shelf every second of the day, and until recently nobody asked it what was on the shelf.

A frame every few minutes is enough for a shelf

The first decision is how often to look, and the answer for a shelf is far less often than the camera can. Stock on a facing changes on the scale of minutes, and a frame every two or five minutes catches every restock and every run on a product without processing a day of frames that show nothing changing. On the dairy case that is a frame at 7:00, one at 7:05, and so on until the doors lock. The record that results is a series of counts against the clock, per facing, per camera, and that is the shape the rest of the pipeline consumes.

The receiving bay wants a different rhythm. A pallet is on the floor for a few minutes and then it is racked, so the bay camera samples faster or samples on the door opening. Two cameras, two intervals, the same model of what a record is.

Nothing here needs a new camera. The aisle camera and the bay camera are the ones the store already owns, and the frames come off the recorder they already write to.

Object detection boxes the facings, and a count is a change signal

The model's job on each sampled frame is object detection at a small scale: a box on every visible unit of every product the store cares about on that shelf, with the product as the class. Two litre whole milk, one litre semi, the store-brand yoghurt in its four-pack. The labeling for that is a person typing the product names once, Lexi proposing the boxes on frames from that camera at that distance, and a person checking them. The checking matters more here than on most jobs, because the products look alike and the boxes are small.

The count that comes back is the front row. A camera cannot see the second row of a deep shelf, so a facing that reads as three units may have twelve behind them, and a facing that reads as empty is empty at the front and probably empty all the way back. I think the honest way to use the number is as a signal of change on that facing, from full to thinning to gone, and not as a stock level to reconcile against the system. Teams that try to reconcile it spend their time explaining the second row.

Every detection is a record with a time and a camera

The output of a sampled frame is a list of rows, one per box: the time, the camera, the product, and where in the frame it sat. Over a day those rows are the history of the shelf, and over a month they are the pattern of it. The whole milk facing empties at 5 pm on weekdays and at 11 am on Saturdays, and it has for six weeks.

That record is what LexInsight answers questions from. "When did the semi-skimmed facing last read empty, and for how long" comes back with the times and the frames that show it. The frames are the difference between a number the store believes and a number the store can check.

The night crew stacks the milk with the oldest date at the front, and the camera reads that wall of dates the same way it reads everything else, one box at a time.

The alert fires when a facing stays empty, with the frame attached

An alert is a rule written as a sentence, with a severity and a cooldown, approved before it goes live: a facing on the dairy case that reads empty for longer than the store's tolerance, routine, to the department's morning list. The frame arrives with the gap boxed, the aisle and the time, and a person walks to the case knowing which facing and what was supposed to be there.

The cooldown is what keeps the channel useful. A facing that is empty for an hour is one alert, and the same facing still empty two hours later is a second alert with a higher severity, and neither is a page at 3 am. The shelf and planogram compliance use case is the same rule extended to the whole bay: which facings are in the wrong place, and which have been gone the longest.

This is the retail work behind our numbers, 4M+ annotations and validations, from stores where the cameras were already on the ceiling and the question was what to do with the frames.

The receiving bay counts what came in before anyone scans it

The bay camera is a different model on the same footage path. Its classes are pallets and roll cages, and the question it answers is how many arrived and when, sampled at the door. That count against the delivery note is an early check that the store gets before the goods reach the shelf, and the record has the same shape as the shelf's, the time and the camera and the class and where in the frame it sat.

LexData takes both models through their whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the cameras the store already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

A nudged camera is the failure to expect

The failure that will happen is the one the store causes without noticing. The aisle camera gets knocked by a ladder during a ceiling tile change on a Tuesday, or is re-aimed by the loss prevention team to cover the end cap. Now the dairy case sits four degrees off where it was on the day the model was trained. The model still finds milk, because milk looks like milk, but the facings have moved in the frame and the counts per facing are now assigned to the wrong facing.

The drift catalog covers this as a camera moved, and the tell is a step. The counts on one camera jump on a maintenance date while every other camera in the store holds. The fix is the cheapest on the list: a short window of frames from the new framing, labeled by the person who checks the morning list, and the next version knows the case from where the camera now sits.

Across the wider retail picture, from queue length at the tills to the bay door, the same thing holds. A fixed camera learns the shelf as it was framed, and the person who notices the counts have gone strange is the person who tells you it moved.