Custom vision models for industrial robots, trained on your own castings

Cell 4 at a machining plant has a robot that picks aluminium castings from a bin and loads them into a lathe. The integrator's vision system was tuned on the casting the plant was running the year the cell was commissioned, and it has worked since. In March a second foundry starts supplying the same casting to the same drawing, with a parting line a millimetre off where the old one was and a slightly different surface finish. The robot misses one in ten picks by the second shift, and the cell stops with the integrator's number on the screen.

The drawing did not change. The casting the camera was tuned for did.

Object detection trained on the cell's own castings

A generic model has been shown bottles and people. It has not been shown this plant's casting, or the connector that goes into it, or what a good one looks like against the bin liner in cell 4 under the cell's own lights. Object detection for a robot cell is trained on frames from the cell's own camera, at its mount, on the parts the cell actually handles.

You type the classes once, casting, connector, and the features the gripper needs, and Lexi puts a box on each in every frame. A person checks the boxes before anything trains. The frames come from a few shifts of the cell running, with the bin at every fill level, since a casting on top of a full bin and a casting alone on the liner are different pictures of the same part.

The integrator left a laminated sign on the camera housing reading "do not touch camera". It is still there, and it is still good advice.

One cell asks three questions of the same frame

Cell 4 needs more from a frame than "there is a casting". It needs three answers, and they are three label types on the same frames in the same project.

Is the casting good? A defect class, boxed on the casting where the flash or the void is, so a bad part is rejected before the gripper spends a cycle on it. Which way is it facing? Two keypoints on the casting's axis, giving the angle so the wrist can rotate to meet it. Where does the gripper go? A grasp point, a keypoint on the feature the gripper closes on, placed the same way on every casting by the labeling rule.

The manipulation and grasping use case names why the third is hard: parts arrive in bins, overlapping and at arbitrary angles, and the pose has to be read from a partial view of a thing that is partly under other things. Two keypoints and a grasp point answer it for a casting lying on a bin liner. A casting half under another needs the full position and rotation the use case describes, and that is where a cell graduates from boxes to six degrees of freedom.

The model runs in the cell on the hardware the cell has

My own view, and it is the one I would argue longest for, is that a robot cell's model belongs on a box inside the cell, and the cloud is for training it. The robot cannot wait on a link to anywhere, the plant's footage should not leave the plant, and a model that is exported once and runs on a compute module beside the controller has no network to fail.

That is how the cell 4 model is deployed. It leaves the platform as a file, PT or ONNX, through a device preset that describes the hardware in the cell rather than an architecture, a Jetson beside the controller, say. The deployment doc covers how a model leaves the platform and how pixels never do. A runner beside the cell's recorder watches the camera, the model runs on it, and only the frames it doubts leave the plant for review.

The cell's cycle time does not change when the model is updated, because a new version replaces the old one in place.

A new part variant is labeled rather than reprogrammed

Back to March and the second foundry. The old approach is to call the integrator, who re-tunes the vision system for the new casting, in a week, for a fee, and the cell runs one variant until the next call. The plant now has two foundries and will have three.

The better approach treats the new casting as a labeling job. Frames from the first shift of the new supplier's parts, the same class list, boxes and keypoints proposed by Lexi and checked by a person, folded into the next version. The old casting's frames stay in the training set, since the old foundry is still shipping, and the cell now handles both without knowing which bin it is picking from.

The drift catalog files what happened as a spec change: a definition moved, and every label that assumed the old definition is now wrong. On cell 4 the definition was "a good casting looks like this", and the second foundry changed it without changing the drawing. The labels for "good" had to be widened by a person who looked at the new parts, and no amount of reprogramming would have done that.

Overlapping castings and oil sheen are the frames that come back

LexData takes the cell's model through its whole life. You type what to look for, Lexi puts a box, two keypoints and a grasp point on every casting in every frame, and a person checks each label before anything trains on it. The model then watches the cell's camera, on a runner beside the controller. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

The frames that come back are two castings overlapping on the liner, oil sheen on a part fresh from the foundry, the gripper's shadow across the bin, and the March casting itself, which the first version had never seen. Each is a correction, and when the corrections cross the project's threshold a new version trains. In our robotics work the figure we hold to is 99%+ safety-critical accuracy, and in a cell it is held by the operator correcting the model on the frames it sent back, one foundry at a time.

The robot's miss rate is the honest measure. When it climbs on one bin and not the others, the frames from that bin say why, and the sign on the camera housing has usually not been obeyed.