Sensor fusion with computer vision, the tag says which pallet and the camera says what happened to it

At 2 pm the RFID portal at dock door 6 reads a tag as a forklift carries a pallet through it. The warehouse system now has a record that the pallet number went out of the door. It has no record of whether it went onto the trailer at bay 6 or was set down on the dock apron because the trailer was not there yet. It will not have one until somebody scans it again or the driver phones from the depot to say the load was one short. The camera above the door watched the whole thing.

Neither sensor is wrong. Each answers a different question, and the fusion is the part that lets one question be asked of both.

The tag answers which and the camera answers what

A tag read is an identity with a timestamp and a location, the portal. It is precise about which pallet and vague about everything else: a pallet passing through, a pallet sitting beside the portal for an hour, a pallet on a forklift that reversed and went out again all produce reads. A camera above the door is the opposite. It cannot read the tag. It can put a box on the pallet, a box on the forklift, track both across frames, and say that the pallet was lifted from the staging lane, carried through the door, and set down on the trailer bed, with the frames behind each step.

Fusing the two is joining an identity to a track. The tag read at 2:00 belongs to the pallet track that crossed the portal's polygon in the same second, and from then on the track carries the pallet number, and every event the camera sees for that track is an event for that pallet.

The join is a timestamp problem before it is a model problem

The sensor fusion labeling use case makes the point that sensors do not agree by default: the extrinsic calibration and the timestamp alignment have to be right before a single label is drawn, and both degrade quietly over the life of a platform. At the dock the calibration is the portal's polygon drawn on the camera frame, and the alignment is the two clocks. A portal that reports in its own clock and a recorder that timestamps in another, a few seconds apart, will join every read to the wrong track on a busy afternoon.

The check is cheap. Walk a tagged pallet through the door, note the read time, find the frame where the box crosses the polygon, and compare. Do it again the Monday after the recorder's firmware update, because that is the day the two clocks drift apart without anyone associating a routine patch with the dock.

The camera catches the tag that fell off

Tags come off. A stringer gets forked through the label, a wrap is cut and the tag goes with it, and the pallet that crosses the portal at 3:10 produces no read at all. The warehouse system does not see a pallet with no tag. The camera does. A pallet track that crossed the portal's polygon with no read joined to it is an event on its own, and it is the event the depot's phone call was really about.

The alert is a rule written as a sentence, "a pallet through dock door 6 with no tag read", with a severity and a cooldown, approved before it goes live, delivered to the dock lead with the frame of the pallet crossing the line. The monitoring guide covers the rule and where it lands. The lead walks to bay 6 and finds the tag on the apron.

An aside from the dock: the tags are stapled to the stringer at the wrap station, and the forklift drivers can tell you which wrap station operator staples low enough to get forked, by name.

A forklift passing behind racking is where the fused track survives

A dock is full of things that block a camera for a moment. The forklift carries the pallet behind the end of a rack, a second forklift crosses in front of the door, the trailer's tail lift swings up. A vision track on its own breaks at each of these and starts a new identity on the other side. With a tag joined to the track, the track can be picked up again on the far side of the rack, because the pallet that reappears is the pallet with that number and the join carries over the gap.

The frames at the gap are the ones the model doubts, a pallet half behind a rack upright, and they come back to a person to rule on. The rulings are what teach the model the dock as it is at 2 pm with three forklifts moving rather than the quiet demo at 10 am.

Something in the way for a month is a maintenance ticket

There is a second kind of blocking, and it does not pass. A new stack of empty pallets goes up beside door 6 in March and stays. The camera still returns a good picture of whatever it can see, and the polygon at the portal is now a third hidden behind blue plastic. Detections at the door fall for the tracks in that region of the frame and hold everywhere else, which the drift catalog covers as something is now in the way.

Check the frame before anyone retrains. Retraining around a blocked view teaches the model to stop expecting pallets in the part of the door it cannot see, and the fix is a forklift moving the stack. The catalog calls this the most misdiagnosed condition on its list, and on a dock it is the easiest to see: open the camera, look at the door, count the pallets in the way.

Fusion belongs at the event level

LexData takes the dock model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the door camera the warehouse already has, on a runner beside the recorder, so the frames stay on site and what leaves is the event and the doubted frames. The corrections retrain it, and the new version replaces the old one with no downtime.

My own view, from the architecture side, is that the fusion should happen at the event level and nowhere lower. The portal emits a read, the camera emits a track crossing a polygon, and a small layer joins them by time and place into one record: pallet number, lifted from lane 3, through door 6, onto the trailer at 2:41, frames attached. Fusing lower than that, at the pixel or the feature, ties the warehouse system to two sensors' internals and breaks on the first firmware update. The robotics work we do treats every sensor pair the same way: each answers its own question, and the join is a record.

The driver's phone call from the depot at 5 pm gets a different answer than it used to. The pallet went onto the trailer at 2:41, and the frame shows it.