Running computer vision on the IP cameras a site already has

The supermarket has eleven cameras on the ceiling, bought for loss prevention, wired to a recorder in the back office that the head of security keeps the password for. The plant has a camera on every line, fitted by the safety team after an incident one January, feeding a recorder in the electrical room. The container yard has cameras on the light masts because the insurer asked for them. None of the three was installed to feed a model, and all three can.

The question that stalls the project at each site is the same one: which cameras do we need to buy. The answer, most of the time, is none.

The cameras on the ceiling were bought for security and they will do

An IP camera is a sensor with a network port. What it does with the light it collects is encode it as video and send it to whatever asks, which for the last decade has been a recorder. A model is simply another thing that asks. The camera does not know or care whether the frames end up in a week-long ring buffer or in front of a detector, and it produces the same frames either way.

What matters for the model is what the camera can see, and the security team already answered that when they mounted it. The aisle camera sees the aisle. The line camera sees the line. The mast camera sees the stacks. If the thing you want to detect is in that view at a size the sensor can resolve, the camera is enough, and if it is not, a new camera of the same kind mounted in the same place would not be either. The what vision can see lesson is the honest test: can the best person on the site make the call from the footage alone.

An RTSP stream is the only requirement

Almost every IP camera sold in the last fifteen years speaks RTSP, and so does almost every recorder. The protocol dates from 1998 and has outlived most of what was designed to replace it. A camera's stream has an address, the address takes a username and password, and anything on the network that connects to it receives the video. That is the entire integration.

Two things make it practical rather than merely possible. The first is that the recorder itself usually re-exposes every camera it holds as an RTSP stream of its own, so a model can be pointed at the recorder in the electrical room and see every line without touching a single camera's settings. The second is that most cameras carry two streams, a full-resolution one for recording and a lighter one for the wall monitor, and the lighter one is often the right one to watch, because the model is going to sample it rather than consume it.

On the sites we run, the store's eleven cameras, the plant's lines and the yard's masts were all reached this way, from the recorder, in an afternoon, with nobody on a ladder.

One consumer per stream, sampled about every two seconds

The model does not watch video. It looks at frames. Every stream gets its own consumer, a process that holds the connection, decodes what arrives and hands a frame to the model about every two seconds. Every camera is watched in parallel, and a stall on the yard's mast camera does not delay the plant's line camera, because they share nothing except the model.

Sampling rather than consuming is the decision that makes the rest cheap. A shelf does not empty between one second and the next, and a cocked cap on a bottling line stays cocked long enough to be seen twice. Decoding every frame of every stream is the expensive part of any pipeline, and it is expense that buys nothing for the questions a store, a plant or a yard actually ask.

The consumer's other job is to reconnect. Cameras reboot, recorders patch themselves, a switch in the back office loses power during a shelf reset at 2 am. A consumer that gives up on the first dropped connection leaves a camera unwatched until somebody notices, and nobody notices an unwatched camera.

Most computer vision projects stall on a hardware quote nobody needed

The quote arrives before the first frame is labeled. A camera with a chip in it, a rugged housing, a mounting survey, a network upgrade, and a schedule that puts the first detection six months out, behind a procurement gate that does not open until January. By the time the new cameras are on the ceiling the person who wanted the project has moved on.

My view, and I hold it firmly, is that a project which begins with a camera purchase has made its first mistake before it has made a decision. The existing cameras produce frames today. Label those, train on those, watch those, and find out in a fortnight whether the question is answerable from that mount. If the answer is that the aisle camera is too far from the shelf edge to read a label, that is a finding worth a camera, and it was learned for the price of an afternoon rather than a survey.

Most computer vision projects that stall never reach that finding. They stall at the quote, in a spreadsheet, before anyone has looked at a frame.

The results go to alerts and review and the footage stays where it was

A detection on its own is a row in a table. What the three sites needed was a detection in front of a person, with the frame, in the channel that person already reads. An alert is a rule written as a sentence, with a severity and a cooldown, approved before it goes live. An empty bread shelf for more than an hour is routine and goes to the store's morning list. A person inside the press guard on line 4 is critical and goes to the shift lead's phone. The frame arrives with the box drawn on it.

Frames the model is unsure of go somewhere else, to a review queue where a person confirms or corrects the box, and those corrections are what trains the next version. That second path is what makes the model better in month three than it was in month one, and it needs nothing from the cameras that the alert path did not already need.

Where the model runs is a separate decision from which cameras it watches. LexData runs in its cloud, on your servers, or on a runner beside the recorder, and with the runner the footage never leaves the building: the alerts fire on site and only the doubted frames leave. The deployment doc covers what leaves and what stays. The platform is the same loop in each case, and the cameras are the same cameras.

The model has to learn the mount it will be watched from

The one place existing cameras do impose a cost is in the training set. A model trained on photographs of shelves taken with a phone at eye level has never seen a shelf from a ceiling camera at a steep angle through a wide lens, and it will do badly on the first day. The frames that train the model have to come from the mount that will feed it, in the light that mount sees, across the shifts and the seasons the site runs.

That is cheaper than it sounds, because the mount is already producing frames. A window of footage from each camera, labeled and checked, is the training set. LexData takes the model from there through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the cameras you already have; the frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

The head of security at the store still owns the recorder's password, and still should. The model is a reader of that recorder, and the fewer things that change about the room in the back office, the better.

A patch to the recorder is the failure to watch for

The thing that breaks a pipeline built on existing cameras is rarely a camera. It is a Tuesday night update to the recorder that drops the sub-stream's bitrate, or a firmware push that shifts the white balance on every camera at once. The frames still arrive. They are no longer the frames the model was trained on, and the drift catalog files this under a firmware or encoding change because it looks like drift and is a configuration diff.

The signal is a step on every camera at the same hour, and the check is the recorder's change log before anybody retrains anything. That log is one more thing the head of security already keeps.