Five computer vision applications in production, on the cameras a site already owns

The camera on the north fence of the substation yard has recorded every night for six years, and for six years the only thing that has looked at the footage is a guard glancing across a bank of monitors at 3 am. Nothing about the camera has to change for that to stop being true. What changes is the question the yard asks of it.

There are five questions a fixed camera can answer in production, and almost every project we see on a plant, a store or a yard is one of them. Where is the thing. How many of them are there. Is the thing right. Where did it go. What does it say. Each has its own label, its own rule and its own way of failing, and the sooner a site knows which one it is asking, the less it spends finding out.

Detection puts a bounding box on the thing and says where

The yard's question is whether anyone is inside the fence who should not be. That is detection: the model returns a bounding box around each person in the frame, and the rule compares the box with a zone drawn once on the image. A person in the car park is a detection with no consequence. A person inside the fence at 3 am is the frame that goes to Slack.

The label is a box on every person in a few hundred frames from that camera, at night as well as in daylight, checked by a person before the model trains. The rule is a sentence: a person inside the fence outside working hours, high severity, a cooldown so one intruder does not send forty messages. On the energy sites we run, this shape of question is where the 21,000+ hazard detections per month come from, and the substation and equipment monitoring use case is the same question with a thermal camera added for the hot bushing.

The failure to watch is the fixed background. A camera that never moves gives the model the same frame all day, and a model can learn the yard instead of the person until the day a van parks in the gap.

Counting is detection with a line drawn across the belt

On the packaging line the cartons come off the case packer at a steady rate, and the count the planner wants is how many crossed the point where the line hands over to the palletiser on each shift. Counting is detection with one more piece: a line drawn across the frame, and a rule that adds one each time a box crosses it.

The extra piece is where it goes wrong. A carton that stalls on the line, rocks back across the line and rolls forward again is counted twice unless the model keeps the same identity on it from frame to frame. So a counting model is really a tracking model with a short memory, and the labels that make it work are boxes with an instance identity carried across frames, which is a heavier label than a box on its own.

The planner on line 3 keeps a tally counter on a lanyard and clicks it once a minute for the first week, to check the camera. That is the right instinct.

Inspection asks whether the thing is right

The tile line runs glazed tiles under a light bar at the exit of kiln 2, and the inspector at the end of it has one question per tile: does it ship. A chip on a corner, a pinhole in the glaze, a crack under the light. Inspection is the third job, and the shape of its answer is different from detection because the decision is a severity call. A chip of a certain size on a visible edge is a reject; the same chip on the back is a pass.

Size and position need an outline rather than a box, which is why the surface defect detection use case labels with masks. The other thing that is different is the balance of the frames. On a healthy line the good tiles outnumber the bad by a wide margin, so the training set is unbalanced by construction, and the rare defects have to be collected on purpose over weeks rather than found in an afternoon of footage.

The inspector's own vocabulary goes into the class list unchanged. If the line calls a particular glaze fault "crawl", the class is called crawl.

Tracking follows one shopper from the end cap to the till

The store's question about aisle 4 is how many people walked down it on Saturday and how long the queue at the till got at noon. The model has to hold on to each shopper as an individual from the moment they appear at the end cap until they leave the frame, so that one person walking slowly is one person and two people crossing each other stay two.

Tracking fails at the crossing. When two shoppers pass, the model can lose one track and start a fresh one, and the count goes up by one for nobody. Staff walking the aisle every few minutes are a second source of error, filtered out with a tag on the uniform rather than a separate model. The retail work behind our 4M+ annotations and validations is largely this job, done store by store, because each store's shelving and light is its own condition.

Compared against its own history rather than a fleet average, a store's count is a number the manager can use. Compared against the average, the good stores carry the bad one.

Reading turns a pallet label into a field on the manifest

At the loading dock, under the dock light at 5 am, every inbound pallet carries a label and every label has to match a line on the manifest. Reading is the fifth job and it is two jobs stacked: detection finds the label as a box, and a reader turns the characters inside the box into fields. The fields are what get matched. A pallet whose label reads cleanly and matches goes straight through; a label the reader could not make out comes back to a person with the frame, and the person's answer is a label the next version learns from.

The dock crew still writes the trailer number on a whiteboard by the door. It has not been wrong yet, and on the morning the reader was, the whiteboard was the check.

The job decides the label, the rule and the failure to watch

LexData takes each of these models through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the cameras you already have, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

What the job decides is everything before that. Detection needs boxes; counting needs boxes with identities; inspection needs outlines and a deliberately collected set of rare frames; tracking needs identities held through occlusion; reading needs the box and then the characters. The rule is different each time, and so is the failure the site should be watching for once it is live.

My own view, which the sales side of our company does not always share, is that a site should start with the most boring of the five. A person inside a fence is a smaller model than a queue length, it is easier to check by walking outside, and the site that gets a boring camera right is the site that trusts the interesting one later. The full set of questions by industry is on the use cases pages, and the way a model moves from the first labeled frame to a camera is on the platform page.