Vision guided robotics, giving the robot a reason to look before it moves

The tray of castings arrives at the machine tending cell at 6 am, and one casting is sitting a few millimetres off its nest because the tray was set down hard on the forklift. The robot's program does not know that. It goes to the position it was taught, closes the gripper on the edge of the casting instead of the boss, and either drops it on the way to the chuck or crashes the gripper into the fixture. The cell stops. The setter comes over, re-seats the casting by hand, and presses cycle start.

That stop happens on most cells that load from trays, several times a week, and it is the ceiling of blind automation: the program assumes the part is where it was yesterday.

The blind program assumes the part is where it was taught

A taught program is a list of positions. The robot goes to each one and does what it was told there, and the whole cell is engineered so that the part is always at those positions: a fixture that locates it, a tray with nests, a feeder that presents it the same way. That engineering is most of the cost of the cell, and it works until the part is not where it should be.

Setters call the taught pick position the nest, and a tray whose nests have worn is the commonest reason a part arrives off. Nobody replaces trays until the cell has stopped enough times to be noticed, which on the 6 am tray took most of a winter.

The camera-guided pick finds the part, estimates its pose, and corrects the path

A camera over the tray changes the assumption. Before each pick, the cell takes a frame, the model finds the casting, outlines it, and places its keypoints: the boss, the two lugs, the parting line. From those a pose is fitted, a position and a rotation in the camera's frame, and the difference between that pose and the taught one is applied to the pick position. The robot goes where the casting is rather than where it was.

The manipulation and grasping use case describes the label this needs: a gripper wants a position and a rotation, the graspable features are specific points on the part, and neither a box nor an outline carries orientation. You type the classes once, Lexi proposes the boxes and outlines on frames from the tray camera, and the setter places the keypoints against them and checks each one, because the setter is the person who knows which lug the gripper has to clear.

The loop's speed is the limit on what the camera can correct

The correction has to arrive between the frame and the pick, and the cycle time sets how long that is. On a machine tending cell it is comfortable, a second or two while the chuck opens. On a high-speed pick from a moving belt it is not, and the design changes: a lower-resolution frame, a smaller model, a pose fitted from fewer points.

The model runs on the cell's own compute for that reason, exported as ONNX or TorchScript for the device the cell has, and the frame never leaves the cell to get its answer. The review, the retraining and the versions live elsewhere, on the plant's servers or in the cloud, and only the frames the model doubted go there.

A failed pick is routed back for review with the frame and the pose

When the gripper closes on nothing, or the chuck rejects a casting that was loaded askew, the cell has the frame, the pose the model proposed, and the outcome. That triple goes back to the setter, in Slack or on the pendant, with the keypoints drawn on the frame, and the setter's verdict, wrong pose, wrong part, or the tray moved, is a label the next version learns from. On the cells we run, the failed-pick queue is the training set, because it is the model being wrong with the evidence attached.

I think every cell should keep the blind program as a fallback, selectable from the pendant, and log how often it is used. A cell that falls back to blind picks twice a week is a cell whose camera has drifted, and the log says so before anyone notices.

A bumped camera mount shifts every pick by the same offset

The failure that arrives without a message is the mount. The camera over the tray gets knocked by a forklift mast on a Friday, or the bracket is loosened during a clean, and the lens now points a few degrees from where it pointed when the camera was calibrated to the robot. The model still finds every casting. The pose it reports is in the camera's frame, and the camera's frame has moved, so every pick is off by the same amount in the same direction. The first pick on Monday drops a casting; the second drops another.

The drift catalog calls this a camera moved and notes that detection accuracy does not change at all, which is why the setter's first guess is the model and the right guess is the bracket. The signal is the shape of the failure: every pick wrong by the same offset from one moment, rather than some picks wrong at random. The fix is to re-run the calibration between the camera and the robot, with the target plate the cell already has, and it takes a few minutes.

On the cells that run well, that calibration check is part of the 6 am start: the robot presents the plate to the camera, the offset is compared with Friday's, and a difference over the limit stops the cell before the first tray.

The corrections keep the pose model on the castings it actually meets

LexData takes the tray model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then runs on the cell's own compute, with the camera the cell already has, or on a runner beside the plant's recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. That is how the robotics work behind our numbers holds 99%+ safety-critical accuracy across a change of tray supplier, and the failed picks from the worn trays are what the second version learned from.

The setter still keeps a spare tray by the cell. It has not been needed since the spring, and nobody has moved it.