Vehicle damage localization with instance segmentation, so the estimate names the panel and the area on it

The claimant took four photos in a supermarket car park at dusk, with the phone at chest height, of the front-left corner where a trolley bay had won. The first is close enough that the dent fills the frame. The second is the whole corner. The third is blurred. The fourth is the number plate, because the app asked for it. The adjuster who opens the claim at 9 am the next morning has to decide from those four frames which panel is damaged, how much of it, and whether that is a repair or a replacement. She has a few hundred more claims like it in the queue.

The question the estimate needs answered is where the damage is on the car, in the parts language the repair shop uses. A model that says "dent" with a box around it has answered the question the claimant asked. The estimate needs the panel.

A bounding box says where the dent is and a mask says how much of the panel it covers

A bounding box on the dent puts it in the frame, and that is useful for finding the damage in the second of the four photos when the first is all dent. What a box cannot do is say how much of the bumper the dent occupies, because the box is a rectangle and the dent is a dent, and a rectangle around a long shallow crease is mostly undamaged paint. The auto damage estimation use case puts it plainly: the estimate is built from the affected area per panel, and only a mask carries the area.

So the damage label is a mask. Dent, scratch, crack, crush and missing part are the classes on the car-park photo from Tuesday, each an outline rather than a rectangle, and the area inside each outline is the number the estimate uses.

One mask for the dent and one for the panel it sits on

A dent mask on its own floats in the 9 am photo. It says a dent of this shape and this size is here, in pixels, and the estimate cannot price pixels. The second mask is the panel: bumper, wing, bonnet, door, sill, and the rest of the parts list, each outlined on the same photo. The dent mask intersected with the panel mask gives the two things the estimate wants: the panel's name and the share of it the dent covers.

That is why the two are separate labels rather than one class called "dented bumper". A dented bumper class would need a training example for every panel and every damage type in every combination, and the archive does not have them. Two layers, damage and panel, each learned on its own, combine into every pairing the estimate will ever need.

My own view is that the panel layer should be labeled first and from undamaged cars. A bumper is a bumper whether or not a trolley has met it, and the archive is full of clean cars in the photos claimants take of the other side.

Phone photos have no capture standard, so the frames are the claimants' own

There is no camera to mount and no lighting to control. The photos come from whatever phone the claimant owns, at whatever angle they held it, in whatever light the car park had, and there is no second chance to reshoot. A model trained on a manufacturer's studio photographs has strong evidence for a studio and almost none for a car park at dusk.

The training frames are the claims archive. Lexi proposes the masks from the class names, and a person checks every one in LexAnnotate before anything trains. The checker's time goes on the boundary of a shallow dent under a dusk reflection, where the outline is a judgment, and on the panel edges where the bumper meets the wing, which two people would draw differently unless the guide settles it. Claimants photograph the damage close and forget the whole car, so the panel model sees far more bumpers than it sees cars.

Borderline severity goes to an adjuster with both masks drawn

A dent that covers a share of the bumper near the repair-or-replace line, a crease that may or may not cross onto the wing, a photo too blurred to outline anything: these are the claims the model doubts, and they should say so. The frame goes to the adjuster with the damage mask and the panel mask drawn on it, she confirms or corrects the outlines, and her call is recorded against the photo.

That is what lets the desk handle the queue. The insurance work behind our 6,000+ cases resolved per month is built this way, with the model clearing the claims whose masks are clear and the adjuster's morning spent on the ones that are not. Her corrections are the labels the next version learns from, and they are labels on exactly the claims the model found hardest.

A revised repair guideline is a spec change for every label

The repair network revises its guideline in the spring: a dent covering more than a certain share of a bumper is now a replacement rather than a repair, and the parts schedule has moved with it. Every photo in the archive looks the same as it did in March, and every severity label the model learned for that boundary is now wrong. The drift catalog calls this a spec change, and the signal is the one it names: adjusters overriding the model's repair verdict in a consistent direction on the borderline claims, while the masks themselves stay right.

The two-layer design is what keeps this cheap. The damage mask and the panel mask are unchanged by the guideline, since a dent is still a dent and a bumper is still a bumper. What changes is the rule that turns an area share into a verdict, and the rule is edited rather than retrained. Only where the guideline redefines what counts as a crease versus a dent do the labels themselves need a pass.

LexData takes the damage model through its whole life. You type what to look for in the estimating language, Lexi puts a mask on every photo, and a person checks each label before anything trains on it. The model then watches the claims as they arrive, in the cloud or on your servers. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The wider insurance picture, from the roof photo to the fleet, is built on the same two questions, what is damaged and on which part.