Running computer vision on RTSP camera streams from the recorder you already have

The plant has a dozen IP cameras, one over each line and a few on the doors, all wired to a recorder in the electrical room that has been quietly writing a fortnight of video to its disks since it was installed. The engineer who has been asked to run a model on the lines has the recorder's address, the password on a label on its lid, and a Friday afternoon.

Everything the model needs is behind that address. The work is getting frames out of it steadily, for every camera at once, without the plant's network noticing.

The recorder re-exposes every camera as an RTSP stream of its own

An IP camera speaks RTSP, and so does the recorder that stores it. The recorder holds a connection to each camera and, on request, offers each one back out as a stream with its own path, so a consumer that connects to the recorder sees every line without ever touching a camera's own settings. That matters on a plant where the cameras were fitted by the safety team and the camera passwords are nobody's to change.

Most recorders offer two versions of each camera, the full-resolution stream that goes to disk and a lighter sub-stream meant for the wall monitor. For a model that is going to sample rather than consume, the sub-stream is usually the one to take: less to decode, less on the wire, and enough pixels for any question that a camera at that mount could answer in the first place.

RTSP itself carries no video. It negotiates, and the pictures travel over RTP beside it, which is why a firewall rule that opens the RTSP port alone produces a connection that succeeds and then shows nothing.

One consumer per stream and a stall on one never touches the others

The shape that survives contact with a real plant is one consumer per stream. Each is a process that holds its own connection to one camera's path, decodes what arrives, and hands a frame to the model when it is asked for one. Twelve streams, twelve consumers, all watched in parallel, sharing the model and nothing else.

The alternative, one loop that polls every camera in turn, fails the first time one camera stalls. A camera on the loading door reboots at 3 am, the loop blocks waiting for it, and the eleven line cameras go unwatched until it comes back. With a consumer each, the door camera's consumer reconnects on its own schedule and the lines never notice.

Reconnecting is most of a consumer's job. Cameras reboot, the recorder restarts after a patch, a switch in the electrical room drops for a second when the compressor starts. A consumer that gives up after one dropped connection leaves a camera silently dark, and the only thing that reports a dark camera is a consumer that keeps trying and logs each attempt.

A frame about every two seconds is enough for almost every question

The model does not need every frame. A pallet does not leave a bay between one second and the next, a person inside a guard is inside it long enough to be seen twice, and a bottle missing its cap stays uncapped all the way to the case packer. Each consumer decodes the stream and hands the model a frame about every two seconds, and every question the plant has asked so far is answered at that rate.

Decoding is the cost. Pulling every frame of every stream through a decoder is the most expensive thing in the pipeline, and at two seconds most of that work is thrown away. Anything that lets the consumer decode fewer frames, taking the sub-stream, seeking to the nearest keyframe, is worth more than a faster model. My own view is that the first tuning to reach for on any RTSP pipeline is the decode path, and that model speed should be looked at last, because a model that is fed one frame every two seconds has all the time it needs.

TCP when the switch is busy and UDP when it is not

RTSP will carry its video over UDP or over TCP, and the plant's network decides which. UDP is lighter and drops packets under load, and a dropped packet in a compressed stream is a corrupted frame, a smear of grey blocks across the bottom of the picture that a model will happily put a box on. TCP retransmits, arrives intact, and falls behind when the network is congested, so the frame the model sees is older than the clock says.

On a plant network that also carries the line PLCs and a shift's worth of office traffic, TCP is the safer default, and the consumer should ask for it explicitly, since many cameras and recorders default to UDP. The cost is latency under load. At a sample every two seconds, latency measured in fractions of a second is not a cost the plant will notice.

On the sites we run, the consumers ask for TCP, and the one plant that started on UDP switched after a week of grey-block frames from the camera nearest the compressor.

Timestamps come from the stream rather than the clock the consumer read

A frame handed to the model needs a time, and the tempting one is the moment the consumer received it. That moment includes the decoder's queue, the network's lag, and the recorder's own buffering, and on a busy afternoon it can be seconds behind the moment the camera saw the frame. Two cameras on line 4 will then disagree about when the same pallet passed.

The stream carries its own timestamps, and the consumer should keep them. When a dwell time, a cycle time or an alert's cooldown is computed from those, the numbers agree across cameras and survive a network that was slow for an hour. When they are computed from arrival time, every congested afternoon shows up as a change in the line's behaviour that never happened.

A bitrate change on the recorder is the step to rule out first

The pipeline runs for a month. Then, on a Tuesday, detections on every line camera drop at the same hour, and the doubted frames pile up. Nothing on the lines changed. What changed was the recorder, which took a firmware update overnight and reset its sub-streams to a lower bitrate, so the frames the consumers pull are now blockier than the frames the model was trained on.

This is the first thing to check on any sudden drop, before anyone retrains anything. The drift catalog files it under a firmware or encoding change and calls it a pipeline event rather than drift, because the world did not move and a configuration diff reverses it in an afternoon. The clue is the shape: a cliff on every camera at the same hour is the recorder, a cliff on one camera on its service date is that camera, and a slope over weeks is the world.

The model that runs on these frames goes through the same loop as any other. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the recorder's streams, in the cloud, on your servers, or on a runner beside the recorder, and with the runner the footage stays in the electrical room and only the doubted frames leave. The deployment doc covers what leaves and what stays. The consumers in this post are the part of that sentence that sits between the recorder and the model, and they are the part the plant's network will test first.