Infrastructure asset management with computer vision from a truck at road speed

The patrol truck leaves the depot at 7 am and drives the rural feeder at road speed, the same loop it has driven every quarter for years. The camera on the dash sees a pole, a crossarm, a hazard sign, a run of guardrail, a marker post, each for a fraction of a second, and then the next. The asset register in the office says what should be on that loop: the pole numbers, the sign types, the guardrail runs by start and end. The two have never been compared except by the patroller's eye and a clipboard, and the patroller has a loop to finish by noon.

The camera has recorded every loop since it was fitted. The register has never seen the footage.

Object detection at road speed sees each asset for a fraction of a second

At road speed on the 7 am loop an asset enters the frame small, grows to a few hundred pixels wide as the truck approaches, and leaves. The model sees it on a handful of frames, blurred on the nearest ones, and has to make its call across that handful rather than on any single frame. Object detection with the register's own classes, pole, sign, guardrail, marker, gives a box per asset per frame, and tracking across the handful of frames turns the boxes into one asset with a best frame.

The best frame is the one that matters. It is the frame where the sign face is largest and least blurred, and it is the frame that goes to the register and to the crew. The rest of the handful are evidence that the asset was passed.

The frame with the asset is matched to the register by position

The register holds an asset's position, and the truck's camera knows where it was when it saw the sign. Matching the two is a position problem with a tolerance: the sign the model boxed at this point on the loop is the sign the register has within a few metres of it. Where the match lands, the register entry gets a fresh frame and a date. Where the model boxed a sign and the register has none, or the register has a sign and no frame showed one, the mismatch is the finding.

The tolerance is the part the office has to settle. A pole is where it is. A marker post can be a few metres from where it was surveyed and still be the same post. The loop's first pass produces a long list of mismatches that are really survey tolerance, which a person rules on once, on the Monday after the first pass.

An aside from the loop: the pole numbers are painted on the poles, and the paint fades on the south face first. The model that finds the pole is the same one that comes back with a frame the office can read the number off before it is gone.

A discrepancy against the register is a ticket for a crew with the frame

The alert is a rule written as a sentence, "a guardrail run in the register with no detection on this pass", with a severity and a cooldown, approved before it goes live, delivered to the maintenance channel with the frame from the best pass attached. The energy work we do has this shape across grid assets, a detection on a patrol becoming a ticket with the evidence, and the alert path is what produces the 21,000+ hazard detections per month behind that page.

A crew with a frame drives to the guardrail and finds it flattened by a truck in the night. A crew with a register row spends the morning finding the guardrail.

Rain and low sun are the days recall drops

The model that shipped in June was trained on dry roads under high sun. In November the loop is driven in rain with the wipers going, and at 7 am the sun is low enough to sit directly behind the signs on the eastbound leg. On those days the model boxes fewer signs, the register shows more mismatches, and the mismatches are the model missing rather than the assets missing. The drift catalog covers this as rain, fog and dust: the share of frames where nothing is found rises for a week, then returns.

Keep those days. The rainy loop and the low-sun loop are rare in the training set and disproportionately valuable in it, and the frames are already recorded. The corrections a person makes on them, a sign the model missed behind the glare, are the retraining set for the next winter.

My own view is that recall is the number to watch on a patrol and precision is cheap to fix. A false detection costs a person a glance at a frame. A missed guardrail costs a crew a quarter, because the next pass is the next quarter. Tune the model to find too much and let the review queue sort it.

The pole the register has and the camera never sees is the finding

Some mismatches are not the weather. A pole the register holds at a point on the loop, and no pass since January has boxed a pole there, is either a pole that was removed and never struck from the register, or a pole a tree has grown in front of. Either way it is a finding for the office, and neither is a job for the model. The power line and grid inspection use case makes the same distinction from the air: the model's job is to say what it saw, and the dispatch decision is a person's.

The patrol's own frames retrain the model

LexData takes the patrol model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the dash camera the truck already carries, on a runner beside the recorder or in the cloud once the truck is back at the depot. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The November loop's glare frames, boxed by the person who ruled on them, are what the model driving in December learned from.

The patroller still drives the loop with the clipboard. The list on it is shorter, and the register in the office has a frame beside every row for the first time since the survey.