What end-to-end computer vision has to mean for a camera fleet

The utility's third vision project has a handoff meeting every second Tuesday, and the calendar invite is still named after the second project, because someone copied it. Four teams attend. A contractor labels the frames from the substation cameras. A data science team trains on them. IT deploys what data science hands over. Operations watches the alerts and, when the model is wrong about a transformer bushing, tells someone in the meeting.

Each team is good at its stage. The project is failing anyway, in the fortnight between meetings, because nothing that operations learns on a Wednesday reaches the contractor until the Tuesday after next, and by then the frame has been overwritten on the recorder.

Computer vision projects fail between the stages

Every stage of a vision project has a page in every vendor's deck: collect, label, train, deploy, monitor. Computer vision projects rarely die inside one of those boxes. They die in the arrows. Frames exported from the recorder to a drive for the contractor on a Friday. Labels exported from the contractor's tool to a folder for data science. Weights exported from a training run to a ticket for IT. An operator's correction on a live alert exported, if it is exported at all, to a spreadsheet nobody trains on.

Each arrow is a person carrying something, and each person has another job. The utility's project is a relay, and a relay is only as fast as the slowest runner and the longest gap between them.

A loop is a workflow where the correction comes back

The difference between a relay and a loop is the direction of one arrow. In a relay, everything flows forward and stops at operations. In a loop, what operations learns flows back to the start: the frame the model doubted, the box the operator corrected on Wednesday, becomes a labeled frame the next version trains on, without leaving the system it was corrected in.

That backward arrow is the whole meaning of end to end, and it is the arrow the deck never draws, because it is the hard one. It means the review queue and the training set are the same store. It means the operator at the substation desk and the contractor's labeler are doing the same job in the same place. It means the model version that watches the cameras tonight can name the corrections it was trained on.

The test is one correction, followed to the next version

So here is the test, and it takes an afternoon. Pick a doubted frame from a substation camera, one the model sent back because it was unsure whether the mark on the insulator was a crack. Have the operator correct it. Then follow that correction. Does it land in the training set on its own, or does someone export it. Does the next training run include it, or does someone have to remember. Does the new version reach the camera without IT scheduling a change, and does the old version stay where it can be rolled back to.

If every answer is yes, the workflow is end to end. If any answer is "someone does that on Tuesdays", it is a relay with a good deck. My own view is that the phrase should be earned by that demonstration on the buyer's own frames, rather than claimed on a slide, and that a team evaluating platforms should ask to watch the correction travel before anything else.

LexData takes the substation model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the cameras the utility already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The platform is that loop, and the test above is what it is built to pass.

The old frames stay in, or the loop eats itself

A loop can go wrong in a second way, and it looks like diligence. Accuracy dips after a cold snap in January, the team rebuilds the training set from last month's corrections, and the model gets better at frost and worse at everything it knew before. Six months later the same thing happens with summer haze, and the project is rebuilding twice a year.

The field guide's lesson on retraining without starting over makes the case in full: corrections on frames the model got wrong are the most valuable labels a team will ever get, and they are added to the set rather than swapped for it. Versions keep what they were trained on. When corrections cross the project's threshold, the new version trains on the corrections and on everything before them, and the model that learned frost does not forget the bushing.

A fleet multiplies the seams

One substation is one loop. The utility has dozens, with cameras of different vintages, mounted at different heights, some facing morning sun. Each one is a place where the model can be quietly wrong on its own, and a fleet average hides it, because the good sites carry the bad one.

End to end for a fleet means the loop runs per camera. The correction rate is visible per site, so substation 14, whose queue fills after a bracket was re-tensioned, stands out from the ones that held. The new version reaches every runner in the order the utility chooses, and rolls back the same way. And a new substation coming online gets its own labeled window before it goes live, rather than the fleet model and a hope.

The handoff meeting at the utility still happens. It is shorter now, and the invite has finally been renamed, because the thing it used to exist for, carrying corrections from one team to the next, is no longer something a person does.