Cloud vs on-device inference for computer vision, and why the answer is usually both

The substation sits at the end of a rural feeder with a thermal camera on the fence line. Its link back to the control room was sized for telemetry, a few kilobytes a second, one October years before anyone thought of sending video down it. The plant, on the other side of the same utility, has a fast link and a security team whose written policy is that no camera footage leaves the building. Both want the same thing: a person to be told when something is on the fence at night, with the frame.

The cloud versus on-device question is usually argued as a choice. On these two sites, and on most of the sites I have seen, the honest answer is a split, and the interesting decision is which frames cross the fence.

Cloud inference gives every site one model and asks for the stream in return

Running the model in the cloud has one large advantage: there is one model. Every camera at every site is watched by the same version, a new version reaches all of them at once, and the review queue, the corrections and the training runs all happen in one place with nothing to synchronise. For a utility with dozens of sites and a team of three on the Monday shift in the control room, that is the difference between a fleet and a collection.

What the cloud asks in return is the stream. The frames have to travel from the camera to wherever the model is, and on the substation's link they cannot. A sample every couple of seconds from one thermal camera is more than the telemetry link was built for, and a second camera would be out of the question. At the plant the link is fine and the policy is not: the stream leaving the building is exactly what the security team's document forbids. Either constraint alone rules cloud inference out for that site, and both are common.

On-device inference keeps the footage in the building and the alert on time

Putting the model beside the camera answers both. A runner beside the recorder at the substation watches the thermal camera's stream on the local network, samples it, draws the boxes and applies the rule on site. The footage never touches the telemetry link. A person on the fence at 2 am fires an alert locally, and what goes down the link is the message and one frame, which the link can carry. If the link is down, the alert still fires on site and is delivered when it returns.

At the plant the same runner satisfies the policy by construction. The stream stays on the recorder's network, the model reads it there, and the security team can see on the network diagram that no video path leaves the building. The security page describes that boundary: what crosses the perimeter is the result, and the evidence stays home.

What on-device inference gives up is the fleet. Each runner has its own copy of the model, and left alone, each drifts on its own site with nobody able to compare them. That is the gap the split has to close.

The fence camera is thermal because the switchyard has no lighting at night, and adding lights would mean trenching power across the yard, which is the one job nobody at a substation will approve.

The split that works is the runner deciding and the cloud learning

The arrangement that holds on both sites is a division of labour rather than a choice of venue. The runner decides: it watches the stream, draws the boxes, applies the rule, fires the alert, all on site, with no dependency on the link being up. The cloud learns: it holds the review queue, counts the corrections, trains the next version when the threshold is crossed, checks it per class against the running one, and sends it back to every runner.

Neither half works without the other. A runner with no cloud behind it is a model frozen on the day it was installed, drifting through its first January with no way to be corrected. A cloud with no runner in front of it cannot reach the substation at all and is forbidden at the plant. The deployment doc puts the same arrangement in the product's terms: the runner is where footage is processed, and what leaves the site is the answer.

Only the doubted frames cross the fence

The split has one traffic rule. The runner keeps every frame it is sure about, and sends up only the frames the model was unsure of. A thermal blob at the fence that might be a person or a deer, a vehicle half out of the frame, a shape it has weak evidence for. Those are the frames a person needs to see anyway, and they are a small fraction of what the camera produces. On the substation's link that fraction is what makes the arrangement possible; on the plant's network it is what makes the policy team sign.

The doubted frames are also the training set. A person reviews each one, confirms or corrects the box Lexi proposed, and the correction counts against the project's threshold. When the threshold is crossed, a new version trains on the original set plus every correction from every site, so the substation's deer and the plant's forklift both improve the model the other one runs. The override rate rising at one site, and not the others, is the drift signal that says a runner's world has moved, which is how a fleet of runners avoids becoming a collection.

Vision language models and training runs stay on the cloud's side

Some work is cloud-shaped whatever the policy says. Training a new version is one: it needs every site's corrections in one place and hardware no runner carries. Answering an open question about a month of footage is another. The kind of question that needs vision language models, a sentence asked of the frames and answered in a sentence with the frames behind it, is heavier than anything that belongs on a box beside a recorder. Those stay on the cloud's side of the split, fed by the frames and results the runners already send, and they never need the stream.

The runner's side is the narrow, fast, repeated work: the same boxes on the same fence, every couple of seconds, all night. A model sized for that is small enough to live beside the recorder and fast enough on a modest board. The device it is built for, a Jetson, a small industrial PC, a GPU server in the plant's rack, is described when the model is exported rather than assumed.

The runner and the cloud run the same version or the split breaks

The failure that undoes the arrangement is quiet. The cloud trains a new version, the plant's runner receives it, the substation's runner is behind a link that was down that night and does not. For the next month the two sites run different models, the corrections from the substation are made against a version the cloud no longer has, and the training run folds them into the wrong baseline. Nothing alarms, because each site's model works, on its own.

So the rollout is a rule rather than a push. A runner reports the version it is running, the cloud refuses to count corrections against a version it has retired, and a runner that has missed a rollout is listed on the same screen as a camera that has gone dark. The replacement itself has no downtime, the runner loads the new version beside the old and switches, but the bookkeeping around it is what keeps the fleet a fleet.

LexData takes the fence model through its whole life across that split. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the cameras the sites already have, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime, on the substation's runner as soon as its link comes back.