Robotic pick and pack with a bounding box and a mask on every item, and why the arm still misses

The tote arrives at cell 4 at 11 pm with nine items in it. Three paperbacks, a bag of coffee in foil, a boxed phone charger, two tubes of hand cream that have rolled under the books, a bottle of shampoo lying on its side, and a soft toy. The arm picks the paperbacks cleanly, hovers over the coffee, descends, comes up with nothing, tries again, and after the third miss the tote is pushed to the exception lane for a person. The failed-pick counter on the cell's screen ticks up by one.

Every one of those misses has a frame behind it. The frame usually says what the camera had to do with it.

A bounding box finds the item and a mask finds where the cup lands

The first thing the camera has to do is say what is in the tote and where. A bounding box on each of the nine items in the cell 4 tote does that, and for a paperback lying flat on top, the box is nearly the whole answer: the suction cup lands in the middle of the box and lifts.

The coffee bag is where the box stops being enough. Its box is mostly empty tote on two sides and the bag is a soft pillow with a seam, and a cup placed at the centre of the box lands on the seam and leaks. The mask, the outline of the bag itself, lets the planner find the flattest patch inside it, which is the grasp point.

The manipulation and grasping use case goes further, to keypoints on the handle or the rim for items that must be gripped rather than lifted, and to orientation, because a bottle on its side has to be picked differently from one standing up. Boxes for where, polygons for the grasp surface, and points for the features a gripper closes on.

Lexi proposes all three from the class names, and a person checks them. On a returns tote the classes are whatever came back that week, and the checker's time goes on the items under other items.

Shiny packaging punches holes in the depth map

The depth camera over the cell returns a distance for every pixel, except where it cannot, and it cannot on foil. The coffee bag comes back as a hole in the depth map with a few wrong values around the rim, and a planner that takes those values at face value plans a grasp a centimetre above or below the surface. That is the hover and the empty cup.

The fix is partly in the label. Foil and clear plastic are their own classes, so the planner can switch to a strategy that suits them: a cup at the edge of the mask where the depth returns are sound, or a fallback to the fixed height of the tote floor. It is also partly in doubt. A frame with a good box and no depth inside it is a frame the model is unsure of, and it should be sent for review rather than picked at, because the third miss costs more than the exception lane.

A tote of returns on a Monday contains at least one thing nobody in the building can name.

The camera on the wrist sees the tote up close and the fixed one sees it whole

Two cameras is the usual answer on cell 4. A fixed camera over the cell sees the whole tote at once and plans the order of picks. A camera on the wrist sees the chosen item close, after the arm has moved and the stack has shifted, and corrects the grasp point on the way down. Each has a calibration to the arm, and the two have a calibration to each other, and the picks depend on all three.

That is a lot of geometry resting on brackets.

A nudged mount is the drift that shows up as failed picks

On Thursday the arm clipped the fixed camera's bracket on a fast retract. Nobody logged it, because nothing looked broken. The camera still returns a clear picture of the tote, the model still finds the coffee bag, and the box is where the box should be in the frame. But the calibration that turns a pixel in that frame into a point in the arm's coordinates was written for the bracket's old position, and every grasp point on cell 4 is now a few millimetres off in one direction.

The failed-pick rate on cell 4 diverges from its own history the same evening, while the cells either side of it hold steady. The drift catalog covers this as a camera moved, and the signature is the same as on a fixed camera anywhere: one camera against its own history, a step on a maintenance date, the rest of the site unchanged. The fix is to re-verify the calibration from the new framing, re-label a short window of frames, and retrain on footage the cell already recorded.

My own view is that the failed-pick counter is the best model monitor a cell has, better than any metric computed from the model's own outputs, because it is measured by the world. A pick that fails is a label, and the world does not need a review queue to produce it.

The failed-pick frames come back for review, and the person names the cause

Each miss is a frame with the box and the mask on it and the planned grasp point drawn, and it goes to a person. The person says why: wrong box, grasp point on the seam, depth hole, item shifted after the frame was taken, or item the model has never seen. That reason is a label, and the frames tagged with it are the next version's training set for exactly the picks this cell finds hardest.

LexData takes the pick model through its whole life. You type what to look for, Lexi puts a box and a mask on every item on every frame, and a person checks each label before anything trains on it. The model then watches the cameras on the cell, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The 99%+ safety-critical accuracy our robotics work reports is held that way, by the frames that failed being the frames that train.

The same camera checks the pack on the way out

The last camera in cell 4 looks at the carton before the lid goes on. Is the charger in the box that ordered a charger, is the label on the carton the label for that order, is anything missing from the nine. Those are boxes and a rule, and they catch the pick that went to the wrong carton, which no failed-pick counter records because the arm lifted cleanly. The wider robotics picture, from the tote to the truck, is built on cells like this one, where the camera is asked three questions and the arm is judged on all of them.