Product recognition AI at the till, checking the counter against the receipt

At 5:40 pm on a Friday the self-checkout at lane 6 is the busiest place in the store. A customer scans four items, bags five, and taps pay. The till has no idea, because the till only knows what was scanned. The attendant is three lanes away helping someone with a coupon. The camera over lane 6 has been recording all of it since the store opened, and until now nobody has asked it anything.

Whether the fifth item was an honest miss or not is a question for the attendant. Whether there was a fifth item is a question the camera can answer.

Object detection at SKU level is a class per product

Recognising a product is a detection job with a very long class list. Each item on the counter gets a box and a name, and the name has to be the store's own SKU rather than "bottle" or "box", because the receipt is written in SKUs. Object detection at that level is trained on the store's own frames, from the store's own ceiling camera, at the angle and the light the camera actually has. A product photographed on a white studio background looks nothing like the same product under the lane 6 light at closing time.

The retail work behind our numbers, 4M+ annotations and validations, was built on exactly that kind of footage. You type the class list once, Lexi puts a box on every item in every frame, and a person checks each box before the model trains. The checker's attention goes to the pairs that look alike on the counter: the small and large bottle of the same drink, the two flavours with the same colour scheme, the own-brand pack that copies the branded one.

A word on scope. The class list does not need to be the whole catalogue on the first day. The first version knows the products that go through lane 6 most, and the long tail comes back as doubted frames.

The set on the counter is compared with the receipt

Once every item in the frame has a name, the pipeline has a list: what the camera over lane 6 saw on the counter between the start of the transaction and the pay tap. The till has its own list: what was scanned. The comparison is the whole product, and it is less simple than it sounds.

A multi-pack is one scan and one box. Loose produce is a weight on the receipt and a bag on the counter, so it matches on the produce code the customer chose rather than on the item. A bag for life on the counter is nothing on the receipt and should not be counted. The comparison rules are written by the store, since the store knows which of its products are sold by weight and which are bundled, and the rules are checked on a week of real transactions before anyone trusts them.

A mismatch goes to a person and never to the customer

When the two lists differ, the alert is written as a sentence: an item on the counter at lane 6 with no matching scan, routine severity, with a cooldown per transaction, approved before it goes live. What arrives on the attendant's handset is the frame with the unmatched item boxed, the lane, and the receipt beside it.

The attendant looks at the frame and decides. Most of the time it is a scan that did not register, a customer who put the item back, or a product the model has not seen. Sometimes it is what the store worried about. Either way, it is the attendant who walks over.

My own view, and I hold it firmly, is that the model should never say anything to the customer. A screen that tells a shopper an item was not scanned, on the strength of a box, will be wrong often enough to cost the store more in goodwill than the fifth item ever cost in stock. The frame goes to the attendant, and the attendant makes the call.

The same camera can tell you whether a till needs opening

The camera over lane 6 is watching the counter. The one over the lane entrance is watching the queue, and the checkout queue monitoring use case is built on that view: does a till need opening in the next five minutes. A queue is a social thing, people step out and return, and the labeling standard has to decide where browsing ends and queueing begins.

Those two questions share the ceiling and share the recorder, and a store that has done one usually does the other next, on the same footage.

A packaging refresh is what breaks recognition first

The failure to expect is not a camera fault. In March the drinks supplier refreshes its pack, and from the first delivery the most common item through lane 6 has a new label the model has never seen. The old class describes a pack that is no longer on the shelf. The attendant starts getting alerts on every transaction with that drink in it, overrides every one, and by Thursday has stopped looking at the handset.

The drift catalog files this as a spec change: the right answer changed by decision rather than by weather, and the model cannot see the decision. The signal is the attendant's override rate on one product, and it rises the week the new pack arrives. The fix is to label the new pack from the frames lane 6 already recorded and fold it in.

Somebody at the customer service desk keeps a handwritten list titled "things the camera cannot tell apart". It is a good list, and it is the class review for the next version.

Doubted frames from the counter are the next version

LexData takes the product model through its whole life. You type what to look for, Lexi puts a box on every item in every frame, and a person checks each label before anything trains on it. The model then watches the cameras over the lanes, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

The frames that come back from lane 6 in the first month are the crushed pack, the item held in a hand, the bottle lying on its side with the label down, and the new pack from March. When the corrections cross the project's threshold, a new version trains on them, and the attendant's handset goes quiet on the drink again.

The camera was already over the lane. The recorder was already writing. The question was only ever what to do with the frames.