Instance segmentation for robotic manipulation from twenty-five labeled frames

The robot at the bench has one job this week, set on Monday morning: pick up the torque wrench by its handle and put it in the rack. On the first attempt it closes the gripper on the head. On the second it closes on the handle, at an angle that has the wrench swinging out of the fingers before the arm reaches the rack. The detection was right both times. There was a wrench, and there was a box around it.

The box was the problem, and so was where the camera sat. Both are cheaper to find out on day three than in month three, which is the case for starting small.

A bounding box finds the tool, a mask finds the handle

A bounding box on a wrench tells the planner roughly where a wrench is. It does not say which end is the handle, how the handle is lying, or where its edges are, and a gripper needs all three. That is the case for instance segmentation: a mask for each object, pixel by pixel, so the planner has a shape and not a rectangle.

The labeling follows from that. You type the classes once, Lexi proposes the masks on every frame, and a person checks each one before it trains. For a wrench the classes are the handle and the head as separate instances, because the grasp is on one and never the other. The manipulation and grasping use case is built on that split: parts as classes, chosen by what the planner needs to know.

Twenty-five labeled frames make a first model worth being wrong with

The mistake most perception teams make is labeling hundreds of frames before the first model exists. The first model is where the surprises live, and every one of those surprises is cheaper to meet with twenty-five frames labeled than with three hundred.

So the first batch is twenty-five frames from the camera position the team thinks will work, labeled with the scheme they think is right, trained by Tuesday afternoon, and tested on the actual grasp. The first model on the bench failed on the wrench head, and it failed on the wrench when it lay across the rack rail, and both failures were visible within an hour. The lesson on how much footage you actually need makes the general case: constrained scenes need less than people budget for, and the first batch's job is to find out what the second batch should contain.

I would rather have a wrong model in the cell on day three than a right one in month three. The wrong one tells you what to label.

The camera on the wrist sees what the planner needs

The first camera was on the robot's body, looking down at the bench at a slant. From there the wrench handle and head are foreshortened, the rack rail hides one end, and the mask the model produces is a fair guess at a shape the camera cannot really see.

Moving the camera to the wrist changed the problem. The camera now looks at the wrench from the direction the gripper will approach, and the mask it produces is the shape the planner has to close on. The body camera frames were kept, because a segmentation model absorbs a second viewpoint without complaint, but the wrist frames are the ones that made the grasp work.

Once the camera is chosen it becomes part of the model. A wrist camera knocked a few degrees on a Thursday collision, a bracket re-seated after maintenance, and the masks are subtly off on every frame while the detection still finds the wrench. The drift catalog covers this as a camera moved, and on a robot it happens more often than on a wall, because the robot moves.

The label scheme is designed backward from the grasp

The question to ask before labeling a single frame is what the planner needs. If it needs the handle, the handle is a class. If it needs to fit a plane to the handle's flat, the flat is a class. If it needs the pivot of a lever, a small mask on the pivot gives it that directly, and no amount of whole-object labeling will.

On the bench the scheme went through three versions between Monday and Friday: whole wrench, then handle and head, then handle, head, and the flat on the handle the gripper pads sit on. Each version was twenty-five frames, and each was tested on the grasp rather than on a metric.

The test tool for the first week was a kitchen spatula, because the team did not want to drop a torque wrench forty times.

The mask goes into the depth frame as a shape, one pixel is never enough

A mask is two dimensional. The gripper closes in three. The wrist camera is an RGBD unit with a depth channel, and in principle any pixel in the mask can be projected out to a point in space. In practice the depth on a cheap sensor is noisy and full of holes, and a single pixel's depth is as likely to be missing as right.

The working approach projects the whole mask and treats the result as a cloud of points, so the geometry of the handle comes from hundreds of readings rather than one. A plane fitted to the bench gives a reference the handle is resting on, and the intersection of the cloud with that plane is a grasp line the planner can trust. Deformable things, a hose or a cable, get the same treatment across frames, tracking the cloud rather than the pixel.

Frames the robot doubts are the next batch

The cell is where the interesting frames come from. A wrench lying at an angle the batch never showed, a second tool overlapping the first, a shadow from the arm itself across the handle. On each of these the model produces a mask it is unsure of, and those frames come back to a person with the mask drawn, to be corrected and folded into the next version.

LexData takes the segmentation model through its whole life. You type what to look for, Lexi puts a mask on every frame, and a person checks each label before anything trains on it. The model then watches the wrist camera the robot already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. In the robotics work we run that is how 99%+ safety-critical accuracy is kept: the doubted frames from the cell are the batch, every time.

The twenty-five frames were never the dataset. They were the question.