Event-driven object detection, turning a bounding box into an alert someone acts on

The camera over the packing area sees cartons all day. Packers assemble them, fill them, tape them and slide them onto the outfeed, and a detector that boxes every carton on every frame produces a stream of detections that nobody needs. What the supervisor needs is to know when cartons are being stacked more than three high on a bench, because the bottom one crushes and the stack comes down on somebody's hands, and the site rule since a near miss in March has been three.

A detector on its own cannot tell her that. It draws boxes. Turning boxes into the one message she wants, and no others, is a separate piece of work, and it is the piece most detection projects stop before.

Detect, evaluate, act is the shape of every alert

Every alert has the same three parts, whatever the camera is looking at, and the packing area's alert was written on a Monday afternoon with the supervisor at the desk. The model detects: on a sampled frame about every two seconds, it returns a box around each carton it finds. Something evaluates: a rule reads those boxes and decides whether this frame describes the event the supervisor cares about. Something acts: a message goes to a person, with the frame, in a channel they already read.

Most of the effort in a detection project goes into the first part, and the first part is the one that is now nearly routine. The second and third are where the packing area's alert lives, and they are written in plain terms rather than trained.

The rule is geometry on the bounding box and three stacked is a stack

The rule is written on the boxes. A stack is a set of carton boxes whose horizontal centres line up within a tolerance and whose vertical extents touch or overlap, and the event is a stack whose count reaches three. That is arithmetic on the bounding box coordinates the model already produced, and it does not need a second model trained to recognise a stack. A person can read the rule, argue with the tolerance, and change three to four if the rule written after the March near miss is ever relaxed.

Geometry rules are worth writing for the same reason in every scene. A person inside a marked zone is a bounding box whose bottom centre falls inside a polygon, which is the whole of the hazard zone intrusion use case. A pallet that has not moved is a box whose position has not changed across a run of frames. A missing cap is a bottle box with no cap box inside it. The model supplies the boxes; the rule supplies the meaning, and the meaning is the part the supervisor can check.

The packing bench is the one place in a warehouse where the cartons are empty when they arrive, which is why they stack so easily and why the rule exists.

The alert carries the frame and goes where the supervisor already is

The rule fires and a message is sent. On the sites we run, the alert is a rule written as a sentence, with a severity and a cooldown, approved before it goes live: a stack of three or more cartons on any packing bench, routine, to the packing supervisor's Slack channel. What arrives is the frame with the carton boxes drawn on it and the stack marked, the camera, and the time from the stream. The supervisor looks at it on a phone, sees which bench, and walks over.

The frame is the part that makes the alert usable. A message that says stack detected on the third bench is a claim the supervisor has to go and verify. A message with the frame attached is evidence she can judge in one glance, and if the model has boxed a stack of empty tote lids as cartons she can see that too, and say so. Slack, email and a webhook into the site's own system are the three places an alert can land, and the monitoring doc describes how a rule is written and where it is routed.

A cooldown makes one stack one alert

A stack of three cartons stays a stack of three cartons for as long as it takes a packer to notice or the supervisor to walk over. At a sample every two seconds, a rule with no memory fires on every frame for the whole of that time, and the supervisor's channel fills with the same stack thirty times in a minute. By the second day she has muted the Slack channel.

The cooldown is the rule's memory. Once it has fired for a bench, it does not fire again for that bench until the cooldown has passed. The cooldown is set to the time it reasonably takes to act: a few minutes for a stack, longer for a pallet that has not moved. One stack is one alert. If the stack is still there after the cooldown, the second alert is a real second alert, and the supervisor reads it that way.

My own view is that the cooldown is the setting to tune first, before the detection threshold, because most alert systems that get switched off were switched off for repetition rather than for being wrong.

The runner beside the recorder does the work and the alert fires on site

The camera over the packing area streams to a recorder in the warehouse office, and the model watches that stream from a runner beside the recorder. The frames are decoded, sampled and evaluated on site; the rule fires on site; the message to Slack goes out over whatever link the warehouse has, and if that link is down the alert still fires locally and reaches the supervisor when it comes back. Nothing about the footage leaves the building. The only frames that leave are the ones the model was unsure of, so a person can review them, and the only thing that arrives from outside is a new version of the model when the corrections have earned one.

That is also where the model gets better. A frame the model doubted, tote lids that might be cartons, comes back to a person; the person corrects the box; the correction counts; and when enough have landed a new version trains and replaces the old one with no downtime. LexData takes the carton model through its whole life this way.

You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the packing area camera, on the runner beside the recorder or in the cloud or on your servers, and the frames it is unsure of come back to a person whose corrections retrain it.

A rule that fires too often is switched off by Friday

The failure mode of alerts is a channel nobody reads. A rule that fires on every stack of two because the tolerance was loose, or on every carton that passes behind another because the vertical overlap was miscounted, teaches the supervisor within a week that the channel is noise. Once it is muted, the stack of five that the rule was written for arrives in a channel nobody is looking at.

So the rule is approved before it goes live, by the supervisor, on a day's worth of frames she can look at, and it is tuned in the open: the tolerance, the count and the cooldown are settings she can read. The severity is routine, because a stack is a walk across the floor rather than a page at 2 am, and a supervisor who is paged at 2 am for a stack is one who turns the alerts off. The site rule is three. The alert says three, in words she wrote, and that is why it is still on in December.