Computer vision in automotive manufacturing, from the body shop to final inspection

A car plant is three factories in a row that happen to share a roof. The body shop welds bare steel into a shell, the paint shop coats it, and general assembly bolts several thousand parts into it, and each of the three has its own light, its own cycle time and its own idea of what a defect is. The camera that inspects welds at framing station 3 is no use at the paint booth, and the model that finds a run in the clear coat has never seen a trim clip.

What the three share is the clock. A station has its cycle time, in seconds, and an inspection that takes longer than the cycle is an inspection that stops the line.

The body shop inspects welds on every shell rather than one an hour

The framing station puts a few hundred spot welds into a shell and the weld engineer used to pull one shell an hour to check them. A camera at the station sees every weld on every shell as the robot moves off. The bead is boxed first, so that later questions are asked inside it rather than across the whole fixture, and inside the bead the classes are the engineer's own: porosity, undercut, a crater crack at the end of the seam.

The weld quality inspection use case sets out why the labels have to follow the plant's own procedure: the acceptance criteria are code-driven, and a labeler working from general practice will draw a different undercut from the one the engineer means.

Good welds outnumber bad ones by a wide margin on a working line, so the frames with defects are kept at a far higher share than they occur, and the rare classes are sampled up until the model has seen enough to draw them. In our manufacturing work the inspection models hold 99%+ accuracy maintained in production, and on a body shop line that is the detector agreeing with the engineer on the shells the engineer pulled.

The paint shop is a mirror and the defect hides in the reflection

A painted door skin reflects the booth lights, the conveyor and the inspector. A run in the clear coat, a sag, a fibre under the paint, a crater from a drop of silicone: each is a small change in how the reflection bends, and each is invisible under flat light. Paint inspection tunnels have known this for decades and light the panel with stripes, so that a defect is where the stripe breaks. The camera looks at the stripes, and the model learns what a broken stripe looks like on a curved panel.

Two things make the paint shop its own problem. The first is that the same defect looks different on a black car and a white one, and on a metallic finish it looks different again, so the frames for training come from every colour the plant runs, at the booth, under the stripes. The second is that a panel's own design lines, the swage along the door and the crease on the bonnet, bend the stripes on purpose. A detector tuned to miss nothing will flag them on every car until it has seen enough shells to learn the difference. The inspectors' corrections on those flags are what teach it.

Labels on paint frames come back at up to 99.9% accuracy when a person checks every one, and on paint the person is an inspector who has spent years learning that a sag and a run are graded differently.

Final assembly is a completeness check at cycle speed

At trim, the question changes from "is there a defect" to "is everything there". A clip that holds the wiring loom, a grommet in the firewall, a badge on the tailgate, a wheel nut cap: each is present or absent, in the right place or the wrong one, the right way round or not. The assembly verification use case makes the point that absence is harder to score than presence, since a missing small part in a cluttered assembly is the signal and the clutter is everything else.

The station camera sees the assembly at the end of the operator's cycle, and the model has a box for every part on the check list. A box that should be there and is not is a stop, with the frame and the empty position drawn on it sent to the station screen and to the line lead in Slack. The rule is written as a sentence, with a severity and a cooldown, approved before it goes live: a missing loom clip at station 12, high, to the station and the lead.

An aside from the line. The operators at station 12 still tick the paper check sheet, and the sheet is what the model's stop gets compared with when the two disagree. That is how the plant found out the model was right about a clip the sheet had been ticking for a year.

The cycle time decides where the model runs

A station cycle is seconds long, and the model's answer has to arrive inside it. That decides where the model runs. A round trip to a data centre and back may fit in a cycle on a good day and not on a busy one, and a stop that arrives after the shell has moved on is a stop at the wrong station.

On a car plant the model runs on a runner beside the recorder, on the plant's own network, and fires the stop locally. The footage stays in the plant and only the doubted frames leave.

LexData takes each station's model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the station camera, on your servers or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime, which on a line that runs three shifts is the only kind of rollout there is.

A model-year change is an equipment change, labeled before changeover

The new model year arrives in August with a different door skin, a new supplier for the loom clips and a revised bracket in the body shop. Every camera on the line is now looking at something it has never seen. The paint model flags the new door's crease on every car. The trim model reports the new clip as missing because it is a different shape. The drift catalog calls this the equipment changed: a new generation of the part, a new supplier's component, and the model has never seen the thing it is now looking at.

The signal is the correction rate stepping up on the morning of changeover, and on a car plant that morning is scheduled months ahead. So the frames are labeled before it. Pre-production shells go through the body shop and the paint booth weeks before launch, and the frames from those runs are labeled and folded in so that the model meets the new door already knowing it. A supplier change is smaller and more frequent, and the same rule holds: the first box of new clips goes under the station camera for an afternoon before it goes on the line.

My view, having argued it with plant teams who wanted to start in the paint shop, is that an automotive rollout should start at trim with a completeness check. The check sheet gives a ground truth from day one and a wrong stop costs a recount rather than a rework. The paint shop is where the value is and where the model is hardest, and a plant that has learned the review queue on clips is a plant that will keep the paint model alive. Where the models run, and how a runner sits beside the recorder on the plant network, is in the deployment doc.