Computer vision on multiple video streams from one runner

The distribution building has twenty cameras on its recorder: fourteen inside over the aisles and the docks, four on the yard, two on the gatehouse. One runner sits beside the recorder and watches all of them with the same model, which boxes people, forklifts and pallets and fires a routine alert when a person is inside the forklift lane. For six weeks it worked. Then the yard camera on the far fence, which reaches the recorder over a wireless link, started dropping to a crawl in the rain, and on a wet Tuesday night every alert in the building stopped.

Nothing inside the building had changed. The pipeline had been written so that one slow stream could hold up the rest, and the far fence camera found the weakness.

Every camera is watched in parallel, one consumer per stream

The first principle is that streams do not share a reader. Each camera gets its own consumer, pulling its own RTSP feed from the recorder, decoding its own frames, and handing sampled frames to the model on its own timeline. If the fence camera is slow to deliver a frame, its consumer waits. The other nineteen consumers do not know the fence camera exists.

That sounds obvious and is not how the first version of most pipelines is built, because the simple loop that reads camera one, then camera two, then camera three, works fine on a bench with three cameras and fails on the first slow one. The deployment guide describes the runner's shape, and one consumer per stream is the part of it that keeps the building's alerts independent of its worst camera.

Sampling about every two seconds is what makes twenty fit

Twenty cameras producing full-rate video is more frames than one box can put through a model, and more than any rule in the building needs. A person entering the forklift lane is in it for many seconds. A pallet on dock 3 is there for minutes. Sampling each stream about every two seconds gives the model one frame per camera in that window, which is a complete record of every event that matters and a small fraction of the frames the recorder holds.

The sampling also decouples the model from the camera. A camera that runs at a high frame rate and one that runs at a low one both deliver a sample every couple of seconds, and the model's load is the number of cameras rather than the sum of their frame rates.

The building's recorder keeps everything at full rate for playback. The runner reads what the model needs, which is a much smaller thing.

The scheduler takes turns so a busy camera cannot starve a quiet one

Twenty consumers each produce a frame every couple of seconds, and one model has to serve them. The queue in front of the model is where fairness is decided. A first-come queue lets a camera whose frames arrive in a burst, the gatehouse at shift change, fill the queue and push the quiet aisle cameras to the back. The aisle where a person just stepped into the lane then waits behind a dozen frames of the car park.

The runner takes turns instead. Each camera gets one frame in the queue at a time, and a new sample from a camera that already has one waiting replaces it rather than joining behind it. The model always sees the newest frame from each camera, and the oldest frame in the queue is never more than one round old. Under load the sample interval stretches evenly across all twenty rather than collapsing on the ones that happen to be busy.

The gatehouse cameras at 6 am, when the day shift's cars arrive, are the busiest in the building, and the forklift lane alerts on the far side do not slow down for them.

A slow camera drops its own frames and stalls nobody else

Back to the fence camera on the wet Tuesday. Its consumer, on a wireless link at a crawl, gets a frame late, then later, then not at all for a while. In the version that failed, the consumer blocked waiting for that frame and the loop it belonged to blocked with it. In the version that works, the consumer times out, drops the sample it was waiting for, logs that it did, and tries again on the next interval. The fence camera's own record has gaps on wet nights. Nobody else's does.

Dropping is the right policy for a live rule. A frame that arrives twenty seconds late describes a yard that no longer exists, and an alert raised on it would be wrong by the time it lands. The recorder still has the full footage for anyone who wants to look back at what the model missed.

One unplugged camera is one silent heartbeat

On the Thursday after the rain, the electrician unplugged camera 14 to move it two bays along, and left it unplugged over lunch. The consumer for camera 14 reconnected every few seconds, failed, and carried on trying. The other nineteen streams never noticed. What noticed was the heartbeat.

Every camera on the runner has a heartbeat: the timestamp of the newest frame the model received from it. In LexInsight the building shows as twenty timestamps, nineteen of them seconds old and one of them, over lunch on Thursday, forty minutes old and climbing. An alert on the heartbeat itself, written as a sentence in the monitoring and alerts guide's terms, tells the site that camera 14 has gone quiet, which is a different message from every alert stopping at once.

My own view is that the heartbeat per camera is the single most useful number a multi-stream runner reports, ahead of any accuracy figure, because it is the only one that distinguishes a building where nothing happened from a building nobody is watching.

The model updates once and every stream gets it

LexData takes the building's model through its whole life. You type what to look for, Lexi puts a box on every person, forklift and pallet in every frame, and a person checks each label before anything trains on it. The model then watches the cameras the building already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. On the runner that means one model swapped once, and all twenty consumers feeding the new one from the next sample on.

The wet-night frames from the fence camera, the ones that did arrive, came back for review because the model had not seen the yard under that much rain. They were the first corrections of the model's second version.

The electrician who moved camera 14 wrote the new bay number on a strip of insulating tape stuck to the camera's back, which is where the site's camera map lives now.