Thermal infrared imaging for computer vision on a transformer yard

At 2 am the colour camera over the transformer yard shows a black rectangle with two sodium lamps in it. The thermal camera on the same pole shows the yard as clearly as it did at noon: the transformer tank a warm grey, the radiators cooler. The bushing on the far phase is a small bright spot, hotter than its two neighbours, and it has been getting hotter for a week. Behind the fence a figure is walking the perimeter, and the figure is the brightest thing in the frame.

Neither of those is visible to the colour camera until morning, and by morning the person is gone and the bushing is the same temperature it was, which is the problem.

Thermal frames are temperature, and the model has to learn that look

A thermal camera on a pole in a yard in Cumbria does not see light. It measures the infrared each surface gives off and maps the temperature to a grey level, so the frame is a map of how warm things are, drawn at a lower resolution than a colour sensor and with no colour at all. Edges are soft where a warm object meets warm air. A person is a bright shape with a cooler head where the hair is. A transformer is a texture of grey blocks with the heat sinks showing as ribs.

A model trained on colour footage has learned edges, textures and colours that do not exist in that frame. It will find some people, because a person's outline survives, and it will find almost no hot bushings, because a hot bushing in colour is an insulator like any other. The thermal look has to be learned from thermal frames.

Object detection on thermal needs its own labels and its own set

The practical consequence is a second training set, labeled on the thermal frames themselves. The classes are drawn from what the yard actually asks: a bushing, a hot bushing, a person, a vehicle, the fence line. Lexi proposes the boxes on the thermal frames, and the person checking them is doing a harder job than on colour footage. A bushing at the edge of the radiator bank is a grey blob against grey blobs, and the "hot" in hot bushing is a judgement about brightness relative to its neighbours rather than a shape.

That judgement is the label that matters. Object detection on a thermal frame is asked to find the hotspot as a position, and the substation equipment monitoring use case is a box for this reason. A hot bushing is found by where the thermal signature sits relative to the equipment, and an intruder by whether a person lands inside a zone.

Frames should come from the camera as mounted, through the day and the seasons, because a yard in January and a yard in July are different maps. The bushing that is a bright spot against a cold tank in winter is a slightly brighter spot against a warm tank in summer, and the model needs to have seen both.

The hot bushing and the person at the fence are one camera's two jobs

The alert on the bushing is routine: a hotspot on any phase warmer than the margin the utility chose, sent to the maintenance channel in Slack with the frame attached, so the engineer sees which bushing and how bright. The alert on the person is different in kind. A person inside the fence at 2 am is an intrusion, with a higher severity and a shorter cooldown, and it goes to whoever is on call.

Both alerts come from one camera and one model, and the second one is what the colour camera has never been able to give. On the energy sites we run, the same alert path across grid assets produces the 21,000+ hazard detections per month our energy work reports, and the thermal cameras carry the night shift of that number.

With a runner beside the recorder the alert fires on site first. The yard's footage stays where it was recorded, and only the frames the model doubted leave.

Emissivity turns one temperature into different pixels

Two surfaces at the same temperature in the Cumbria yard do not look the same to a thermal camera. A painted steel tank gives off most of its heat as infrared and reads warm. A polished aluminium fitting beside it reflects the sky and reads cold, at the same temperature. A bushing with a porcelain skirt reads differently from one with a polymer skirt. The property is emissivity, and it means the grey level in the frame is a function of both the temperature and the material.

For a detector this mostly does not matter, because the model learns what a hot bushing looks like in this yard, on these bushings. It matters the day the utility replaces a porcelain bushing with a polymer one and the new unit reads cooler than the old one at the same temperature. The model has learned the old material's brightness, and the new one has to be labeled in.

The engineer who does the thermography round by hand carries a laminated card of emissivity values in the van, and still checks the card before writing a temperature down.

A swapped thermal sensor is a calibration to redo

Thermal cameras get replaced, and the replacement is rarely the same unit. A different sensor has a different response curve, a different noise floor and often a different resolution, and to the model the yard it knew now reads as a stranger's yard. The drift catalog calls this a camera was replaced, and on a thermal camera the effect is larger than on colour, because the whole frame is the sensor's opinion of temperature rather than a picture of reflected light.

The signal is the correction rate. Frames come back for review the week after the swap, and the person correcting them finds the hot bushings marked as ordinary and the ordinary ones marked as hot. The override rate on that camera climbs while every other camera on the site sits where it was. A short window of frames from the new sensor, labeled and folded into the next version, is the fix.

LexData takes the thermal model through its whole life. You type what to look for, Lexi puts a box on every bushing and every person in every thermal frame, and a person checks each label before anything trains on it. The model then watches the yard cameras the utility already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

The two cameras answer two questions and should be two models

My own view is that a team should resist the urge to fuse the colour and thermal feeds into one model to save effort. The colour camera answers what a thing is. The thermal camera answers how warm it is and whether anyone is there in the dark. Those are different questions with different labels, and a model asked to answer both from a stitched frame is worse at each than two models on their own feeds.

Where they meet is the alert. A person the thermal model finds at the fence at 2 am and the colour model confirms at 6 am, when the light comes up, is the same event with two frames behind it. The utility's control room keeps both.