Dwell time detection in video, measuring how long rather than how many

The camera over the dairy case has counted shoppers for a year. The merchandising lead does not want a count. She wants to know how long people stand in front of the new yoghurt range before they pick one up or walk away, because the range was moved to eye level in September and the count has not changed. A count says the same number of people passed. It says nothing about whether any of them stopped.

The same gap exists at the machine guard on a plant floor, where the safety lead wants to know how long a worker stood inside the fence. It exists at the dock, where the shipping office wants to know how long each trailer sat at door 4 with nobody unloading it. In each case the detector already draws the boxes. What is missing is time.

Dwell needs identity and a bounding box on one frame has none

A detector looks at a frame and returns a bounding box for each person it finds. On the next frame, two seconds later, it does the same, and nothing in the output says whether the box at the dairy case is the same shopper or a different one who has stepped into the same place. Counting survives that. Dwell does not, because dwell is a property of one person across many frames, and the detector's boxes are properties of one frame each.

So the first thing a dwell system adds is a tracker: a component that matches each frame's boxes to the boxes on the previous frame and gives each shopper an identity that persists. Matching is done on position and appearance, and it fails in the ways a person would guess. Two shoppers who cross in front of the case can swap identities. A shopper hidden behind a cart for a few frames may come back as somebody new. A dwell measurement is only as good as the identity under it, which is why the merchandising lead's minutes are checked against a person watching the footage before anyone trusts them.

The retail work behind our numbers, 4M+ annotations and validations, includes exactly this kind of checking, because a box that says person is easy and a box that says the same person is not.

The zone is a polygon on the frame and its edge is where the trouble is

Dwell is measured against a zone, and the zone is a polygon drawn on the camera's own view: the floor in front of the dairy case, the area inside the guard fence, the apron at dock door 4. It is drawn in the frame's pixels, which is convenient and is also the weakness this post returns to at the end.

Whether a box is inside the zone is decided by a point on it, usually the bottom centre, where the feet would be. A shopper whose head is over the zone but whose feet are in the aisle is browsing from the aisle, and the bottom-centre rule says so. At the guard fence the choice of point matters more: a worker leaning over the fence with feet outside is a different event from one who has stepped in, and the zone rule should be written with the safety lead rather than for them.

Store planners put milk at the back on purpose, so that a shopper who came for milk walks past everything else. The dairy case zone therefore sees more passing traffic than any other in the store, and most of it is not dwelling.

Hysteresis keeps a flickering box from resetting the clock

A shopper standing at the edge of the zone produces a box whose bottom centre lands inside on one frame and outside on the next, because the box itself wobbles by a few pixels from frame to frame. A naive rule sees enter, exit, enter, exit, and records four visits of two seconds each on a Saturday afternoon where a person would see one shopper standing still for a minute.

The fix is hysteresis: a box has to be inside the zone for several consecutive frames before the clock starts, and outside for several before it stops. The entry delay filters the shopper who brushed past. The exit delay bridges the wobble and the cart that hid her for a frame. My own preference is to make the exit delay longer than the entry delay. A shopper who steps back to compare two prices has not left, and a worker who leans out of the guard for a moment is still inside it for the purposes of the safety lead.

Timestamps come from the stream so the minutes are real

The minutes have to be computed from time the camera saw, not time the pipeline received. On a busy Saturday afternoon the frames from the dairy case camera can arrive seconds late, in bursts, and a clock read on arrival would stretch every dwell on that camera by the delay while the dock camera on a quieter switch reports true minutes. The stream carries its own timestamps, and dwell is computed from those, so a congested afternoon shows up nowhere in the merchandising lead's numbers.

The other timing decision is the sample rate. About one frame every two seconds is enough for a shopper, a worker or a trailer, none of which arrives and leaves between samples, and it is what keeps a dozen cameras affordable to watch at once. The hysteresis delays are set in frames at that rate, and are re-set if the rate changes.

A moved camera moves the zone and the dwell numbers with it

The zone was drawn in pixels, and the pixels belong to the camera's position on the day it was drawn. A crew cleans the housing over the dairy case one Tuesday, the bracket is re-tensioned a few degrees off, and the polygon that used to cover the floor in front of the yoghurt now covers half the floor and part of the shelf. Shoppers who stop in front of the range stand half outside the zone, dwell falls, and the merchandising lead concludes the eye-level move failed.

The drift catalog files this under a camera moved, and the signature is the same as for any detector: one camera's numbers step away from their own history on a maintenance date while the other cameras hold. The fix is to redraw the zone from the new framing and to re-label a short window of frames, which is the cheapest correction in the catalog. The check is cheaper still: a reference frame from the day the zone was drawn, compared against a frame from today, by a person, once a week.

The answer belongs in a question about the footage

A dwell per zone per hour is a table. What the merchandising lead wants is to ask whether people stopped longer at the yoghurt after the September move, and to see the frames of the ones who did. In LexInsight that is a question typed against the footage, and the answer comes back with the frames behind it, so that a number about the dairy case can be checked against the shoppers it was computed from.

The same table drives the alerts. A rule written as a sentence, with a severity and a cooldown, approved before it goes live. A person inside the guard zone for longer than the safety lead's limit is critical and goes to the shift phone with the frame attached. A trailer at door 4 with no forklift movement for forty minutes is routine and goes to the shipping office. The checkout queue monitoring use case is the same measurement pointed at a till, where dwell in the queue zone is what decides whether to open another lane.

LexData takes the dwell model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the cameras the store already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The tracker, the zone and the hysteresis sit on top of that, and the moved bracket is what the person with the reference frame is there to catch.