Video analytics with vision AI, the three stages every camera system shares

The packing hall had a motion rule for years. Any change in the pixels over the outbound lane after 8 pm sent a message to the night supervisor, and every evening at dusk, when the high bay lights came on and the shadows swung across the floor, it sent one. By the second week the supervisor had muted the channel. The rule was not wrong about motion. It was answering a question nobody had asked.

The questions people ask of a camera are of a different kind. Is that carton damaged. Did anyone step inside the robot's zone. How long did the pallet sit on the dock. None of those is a pixel change, and every working system that answers them has the same three stages underneath, whatever it is called on the invoice.

A bounding box on every frame is the first stage

The first stage finds things. On every frame the model puts a bounding box around each object it was trained to look for, a carton, a person, a pallet, a forklift, with the class attached. Nothing else in the system can happen without this, because you cannot follow or judge what you have not located.

What makes this stage work in a real hall is the frames it was trained on. The labeling is a person typing the classes once, Lexi proposing the boxes on frames from the hall's own cameras, and a person checking them. A model trained on somebody else's warehouse finds cartons, mostly. A model trained on this hall's cartons, under this hall's lights, with this hall's shrink wrap, finds them at dusk.

The second stage follows the box across frames

A frame tells you a carton exists. The next frame tells you a carton exists. Without something linking the two, a carton that spends a minute on lane 3 is a minute's worth of cartons. The second stage gives each box an identity that persists across frames, so the carton in this frame is the same carton as in the last one.

That identity is what most of the useful questions depend on. Counting once. Dwell time on the dock. Whether the person who was outside the zone is now inside it. Which carton the damage belongs to, so it can be pulled rather than described. The track is unglamorous and it is where most of the tuning time on a new camera goes: how many frames before a new track counts, how long a track survives behind a post.

The third stage asks a question of the box

Once there is a carton with an identity, the third stage asks about it. Is it damaged. Is its label facing up. Is the person inside the fenced zone, and for how long. Some of these are rules on the boxes, a zone drawn on the frame and a track that enters it. Some are a second look at the region inside the box by a model trained on that question.

The alert is a rule written as a sentence, with a severity and a cooldown, approved before it goes live: a damaged carton on the outbound lane, routine, to the packing lead's channel. It arrives with the frame and the box, which is the difference between "something moved" and something a person can act on. The alert described in plain words is the same shape for a zone breach or a stalled pallet, with the class and the severity changed.

The question that was not written as a rule gets asked afterwards, in words. How many damaged cartons on the outbound lane this week, and which shift. LexInsight answers with the count and the frames.

The runner beside the recorder is part of the architecture

Where the three stages run is a decision, and in most halls it is made by the network. The cameras were installed to record to a box in the comms room, and there is no outbound video link that would carry every camera's stream to anywhere. So the runner sits beside that recorder in the comms room, on the hall's own network, from the day it was installed in May, and watches every camera in parallel, one consumer per stream, sampled about every two seconds.

Alerts fire from there first. The frames stay on the hall's network, and the only frames that leave are the ones the model doubted, going to a person for a verdict. The platform runs the same three stages in the cloud or on the hall's own servers when the network allows it; the stages do not change, the boundary moves.

The review of doubted frames is the stage people forget

The three stages produce a fourth thing nobody drew on the diagram: frames the model was unsure of. A carton half behind a pallet, a person in a hi-vis colour the training frames never had, a box at dusk with the shadow across it. Those frames come back to a person with the box drawn, and the person's verdict is a label.

LexData takes the hall's model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the cameras the hall already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

The correction rate on those frames is the health signal for the whole system. A steady rate means the model is holding. A rate climbing on one camera means that camera's world changed, and the maintenance log for that week usually says how.

The dusk rule is why the stages exist

My own view is that every hand-written pixel rule is a model somebody did not want to train. The motion rule at dusk on lane 3 knew nothing about cartons, so it could not tell a carton from a shadow, and the only fix within its vocabulary was a threshold, which moved the false alarms to a different hour. A detector trained on the hall's cartons does not fire when the lights come on, because the lights coming on is not a carton.

The packing lead's tape gun, left on the lane at the end of a shift, was the other regular trigger. It is boxed as nothing now, which is the right answer.