Computer vision for autonomous mobile robots that pull pallets from dark trailers

The trailer backed onto dock 4 has been on the road for two days and the pallets inside have shifted. The first one is a wooden pallet with a split stringer, wrapped in stretch film that has gone milky, sitting a few degrees off square against the trailer wall. It is dark, because the trailer's interior light is a rectangle of daylight at the far end. A forklift driver reads that scene in a second. A robot with a forward camera has to be taught it.

Vision-first robots do this without a scanner on the pallet or a reflector on the wall, and the question of how they keep doing it, in the second warehouse and the third, is the more interesting one.

The forks need the pallet's pose, and pose estimation gives it from one frame

Finding the pallet is the easy half. A box around it says there is one and roughly where. The forks need more: the pallet's centre, its width, and the angle it sits at relative to the robot's approach, since a pallet a few degrees off square is a pallet the forks miss the pockets on. Pose estimation supplies that from the forward camera, without a second sensor: the pallet's position, its rotation, and the height of the fork pockets, from the frame the robot has anyway.

The labeling is oriented boxes on pallets, drawn with the angle, and keypoints on the pocket openings, from the robot's own cameras at the robot's own height, in trailers, with the film and the split stringers and the dark. You type the classes once, Lexi proposes them on every frame, and a person checks each one, with the rotation angle the thing most often wrong. A labeler who squares up every box teaches the model that pallets are always square, and the forks find out otherwise at the dock.

The frames come from the fleet. A pallet photographed on a bench under studio light teaches the robot about a warehouse that does not exist.

People and obstacles are the classes the robot stops for

The second job of the forward camera is everything that is not a pallet. A person stepping into the trailer, a dropped strap, a loose cable across the dock plate, a puddle the robot's planner would otherwise drive through. These are the classes the robot stops or steers for. The navigation edge cases use case describes why they are labeled as dense regions rather than as a list of objects: the edge case is usually a scene structure, an unmarked ramp, a surface that reflects, rather than a thing with a name.

People are boxed as people, always, with a written rule that a partial view counts, since a leg behind a pallet is a person the robot has to stop for. In our robotics work the figure that matters here is 99%+ safety-critical accuracy, and the person class is where it is earned.

The stop is a rule written as a sentence, with a severity and a cooldown, approved before it goes live. A person inside the trailer envelope while a robot is engaging, critical, to the robot's controller first and to the dock supervisor in Slack with the frame attached. With a runner on the site network beside the recorder the stop fires locally, which on a dock is the only place it can fire in time.

The model runs where the robot is, and the doubted frames leave

A robot in a trailer has no reliable link to anything. The model that finds the pallet and the person runs on the robot or on a runner on the site network, and the footage stays where it was taken. What leaves the site is the doubted frames: the pallet the model was unsure of, the shape at the trailer door it could not classify, the pose it estimated at low confidence and the forks then had to correct.

That is the loop, stated for a fleet. LexData takes the perception model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the robot's cameras, on a runner beside the recorder, and frames it is unsure of come back to a person. The corrections retrain it, and the new version replaces the old one with no downtime.

Across a fleet the doubted frames arrive from every robot on every dock, which is the only way the long tail gets collected, because a rare scene cannot be found by sampling and has to be mined from the fleet's own uncertainty.

The override rate is the number to watch. When the forks correct the estimated pose more often this week than last, or when drivers take over more often at one dock, the model is drifting there whatever the fleet average says. Fleet averages hide a bad dock behind eleven good ones.

Where the fleet fails is milky film, low sun and a new pallet supplier

The failures are not exotic. Stretch film that has gone milky hides the pocket openings the keypoints sit on. Low sun through the trailer door at 4 pm puts the pallet in silhouette. A new pallet supplier ships plastic pallets with pockets in a different place, and the pose the model estimates is right for a wooden pallet that is not there.

Each of these is a doubted frame before it is a failure, and the review queue is where they show up first. In our robotics work the fleets hold 95%+ navigation reliability in changing environments, and "changing" is the operative word: the environments that stayed the same never needed the loop.

One aside from the dock. The forklift drivers who used to pull the trailers still walk the dock at shift start, and the thing they check is the dock plates, because a lifted plate is the one obstacle nobody has yet labeled enough of.

The second warehouse is a new site, with its own window of frames

The fleet that ran clean at the first warehouse for a year is moved to the second, which has the same robots and the same pallets and a different everything else: the dock height, the trailer mix, the lighting, the floor paint, the customer's film. The drift catalog calls this a new site came online: the model was trained at one site and deployed at another, and the second site does not look like the first. Every new deployment site is a fresh distribution of edge cases, and the long tail solved at the first facility is a different long tail at the second.

The fix is a window of frames from the second site's own docks, labeled and folded in before the fleet goes live there, and the site compared against its own history from the first week rather than against the first warehouse. A fleet that goes live on the first warehouse's model and waits for the override rate to climb has chosen to learn the second site's long tail from its failures.

My own view, as the person responsible for where our models run, is that a fleet operator should insist on the runner beside the recorder from the first robot, even where the site has good connectivity. A robot that needs a link to stop for a person is a robot that will one day have no link. How a model gets to the robot's own hardware, as PT, ONNX or TorchScript through a device preset, is in the deployment doc.