Computer vision for robotics, what the cameras have to answer before the arm moves

At 6 am the arm in the picking cell is holding position over a bin of zinc-plated brackets, and at the far end of the aisle a mobile robot is stopped at a junction with its indicator on. Neither has failed. Both are waiting on the same thing, which is an answer from the cameras about what is in front of them, and the answer has to be right to a degree that the rest of the plant does not demand of anything else.

The questions come in an order, and each one is a different model with a different label behind it.

Four questions in order, and each is a different model

The first question is what is there. That is detection, a box on each bracket in the 6 am bin and on each person and pallet in the aisle. The second is where it ends, because a box on a bracket half under another bracket includes the other bracket, and the gripper needs the outline. That is segmentation.

The third is how it is oriented, because a bracket lying on its side and the same bracket on its back present the same outline and want different grasps. That is pose estimation, and it is the question most stacks get to last and should get to first. The fourth is how far away it is, because a person at the end of the aisle and a person three steps from the robot are the same size box at different depths, and the stopping decision needs the depth.

The robotics work on this site starts from the same list and the observation that every environment adds cases the list did not cover.

Detection takes boxes, pose estimation takes keypoints, and the planner takes cuboids

Each question needs its own label, and the labels are where the cost sits. Detection and segmentation are boxes and outlines, and on the bin camera a person checks them frame by frame because a bracket's outline under another bracket is exactly where a proposed label goes wrong. Pose estimation needs keypoints: the two holes, the folded edge and the tab on every bracket, placed so that the model learns where those features sit however the part landed. From the keypoints a pose in six degrees of freedom is fitted, and the pose is what the gripper is given.

The mobile robot's planner wants none of those. It wants a cuboid in the robot's own frame, with a position, an extent and a heading in metres, so that the camera, the lidar and the radar all describe the same person in the same coordinates. The sensor fusion labeling use case is that label: one cuboid valid for every sensor at once, which only works if the sensors are calibrated to each other and aligned in time before the first one is drawn.

You type the classes once, Lexi proposes boxes and outlines on the bin frames and the aisle frames, and a person checks each one. Keypoints and cuboids are placed by hand against the proposals, and the checking is slower than on any other footage we label, because the plant will act on the label at speed.

The lighting engineer on that cell hung a diffuser over the bin in the first week, because zinc plate under a point source produces a highlight the size of a bracket, and the model had learned to pick highlights.

The mobile robot's question is distance, and a box does not carry it

The person at the junction is the safety-critical class. The acceptable miss rate is effectively nil, and it has to hold for a person crouching by a rack, half hidden behind a pallet, or stepping out at the edge of the frame. An image box says a person is present. The stopping decision is computed from range, and range comes from the cuboid, or from depth fused in from the second sensor.

That is why the robotics work behind our numbers reports 99%+ safety-critical accuracy separately from everything else, and why the people-in-aisle frames are reviewed by a person before any version is allowed onto a robot. A frame the model doubts, a person half behind a pallet at the junction, is a frame that comes back, and the reviewer's verdict on it is a label the next version learns from.

A firmware update that shifts a timestamp is the first failure to check

The stack that worked on Friday fails on Monday. The picks are off by a fraction, the mobile robot brakes late at the junction, and the reaction is to retrain. Look at the change log first. A driver or firmware update over the weekend that shifts one sensor's timestamp by tens of milliseconds puts the camera and the lidar out of register at speed, and each stream looks correct on its own. The drift catalog files this under a firmware or encoding change, and calls it what it is: a pipeline event rather than drift, the commonest cause of a sudden drop and the cheapest to reverse.

The signal is the shape of the failure. Drift from the world is a slope over weeks. A step on the exact hour of an update, confined to the robots that received it, is a configuration diff, and retraining against it bakes the mistake into the weights.

I would put a timestamp agreement check in the same script that starts each robot in the morning, and refuse to move a platform whose sensors cannot prove they agree. Most teams will think that is excessive until the first Monday.

Where the model runs is decided by how fast the loop has to close

The picking cell needs its answer between one pick and the next, and the mobile robot needs it faster than that. Neither can wait on a round trip to the cloud, so the model runs on the robot's own compute, exported as ONNX or TorchScript for the device the robot carries, a Jetson on most of the platforms we see. The review, the retraining and the versioning run elsewhere, in the cloud or on the plant's servers, and only the doubted frames leave the cell. The deployment page covers the presets.

LexData takes the perception models through their whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then runs on the robot, with the cameras it already carries, or on a runner beside the plant's recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The failed picks from the 6 am bin, each with its frame, are what the next version of the pose model learned from.

A failed pick is the most useful frame the cell produces

Every time the gripper closes on nothing, as it did four times on the 6 am bin, the cell has a frame, a proposed pose and a known outcome. That triple is worth more than any labeled frame from a clean run, because it is the model being wrong with the evidence attached. Route those frames back for review as a matter of course, and the pose model improves on exactly the brackets it could not pick, which are the ones that were lying at the angle the training set had fewest of.