Computer vision for workplace safety, six things one camera can watch for

The safety lead at a distribution yard walks the site at 7 am with a clipboard and a list: hard hats on in the yard, vests on, nobody on foot in the forklift lane, the wet patch by the wash bay coned off. The walk takes forty minutes and covers the site once. The camera on the corner of the warehouse covers the same yard all day, and the list is six questions it can be asked, provided they are asked in the right order.

Every one of them starts with a person.

The person comes first, and everything else is asked of the person crop

The first model finds people. That is the whole of the first stage, and it is built to be good at it on this yard from the 7 am walk onward. People in hi-vis and people without it, people half behind a pallet, people in the cab of a forklift and people on foot. Every other safety question is then asked of the crop around each person rather than of the frame, because a hard hat on the floor of the yard is not compliance and a hard hat on a head is.

Attributing the hat to the head is the subtle part. A helmet box and a person box in the same frame do not say the helmet is on that person. The site safety and PPE compliance use case puts keypoints on the head, shoulders and waist so the gear can be tied to a body, and that is the design that holds up when two people stand close together.

A stock model calls a cap a hard hat, and a bounding box is only as good as its label

The question a safety lead asks on the first day is why a general model cannot be used. It can find a person. It cannot tell a hard hat from a baseball cap from a hood at the distance and angle of this camera, because it was never shown the difference on this yard. A bounding box labeled "helmet" by someone who never stood in the yard is a box around whatever was on a head.

The fix is frames from this camera, with the classes the site uses, checked by the person who runs the walk. You type the classes once, hard hat, vest, no hard hat, no vest, Lexi proposes the boxes on every person crop across a week of frames, and the safety lead checks them. The lead is the person who knows that the contractor's hats are white and the site's are yellow, and that the vests are orange on Monday and grey by Friday.

That last one is the false negative the first version produces. A model that learned orange vests loses them on Thursday, and the correction rate on Thursday frames is what tells you the vest class needs the grey ones too.

Vehicles and zones are the same rule from the other side

The forklift lane is a polygon drawn once on the frame. A person box with its feet inside the lane while a forklift is in it is the alert; a forklift box inside the pedestrian route is the same alert from the other side. The hazard zone intrusion use case is that shape: a box and a line to cross, with the effort in the zone definition rather than the model. On the yards we run, the lane is walked with the forklift driver watching the frame before the rule goes live, so the polygon matches the paint.

The vehicle is its own class, and it has to be, because the rule for a person in the lane is written for a person in the lane while a vehicle is moving in it. A lane with a parked forklift and a person leaning on it at the 10 am break is a yard at rest.

Posture is the person box changing shape and then not moving

A fall is a person box whose height collapses and whose width grows, in a moment, and then stays that way. The rule is written on the shape change and on the time after it: a box that goes flat and is still flat after a set interval is a person on the ground. A person kneeling to tie a boot at the 7 am start goes flat and gets up, and the interval is what separates the two.

The alert is critical and it carries the frame, because the person who receives it needs to know which corner of the yard before they start running. Of the six things on the list, this is the one where the time between the frame and the response matters most, and the one where a false alert costs least, since a person who was tying a boot is fine.

A spill is a region on the floor that was not there at 7 am

The wet patch by the wash bay is a region rather than a box, an outline on the floor with no fixed shape, and the model learns it against the same camera's dry floor. The comparison is with the yard's own baseline, the floor as it looked at 7 am from this camera, so a new region that is darker and reflective is a spill until someone says otherwise. The rule is routine, to the yard supervisor, with the frame, and the cone goes out.

The oil sheen from a leaking forklift and the puddle from the wash bay are the same class to the first version. The safety lead's corrections are what teach it the difference, and the difference matters, since one is a mop and the other is a maintenance ticket.

The alerts go to the safety lead with the frame, and the lead's verdict is a label

Every alert is a rule written as a sentence, with a severity and a cooldown, approved before it goes live. A person in the forklift lane while a forklift moves is critical and goes to the supervisor's radio and Slack. A person without a hard hat in the yard is routine and goes on the safety lead's daily list with the frames. A spill is routine to the supervisor. Each arrives as the frame with the boxes drawn, so the person reading it judges a picture rather than a count.

I think PPE alerts should never go to the individual. They go to the lead, as a daily count with the frames behind it. A camera that polices people one by one gets a rag hung over it by Wednesday, and a lead with a week of frames can fix the reason the vests come off at the loading door.

LexData takes the yard's models through their whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The models then watch the corner camera the yard already has, in the cloud, on your servers, or on a runner beside the recorder. Frames they are unsure of come back to a person, the corrections retrain them, and the new version replaces the old one with no downtime. The grey Thursday vests from the first week are what the second version learned from.

The lead still does the 7 am walk. The clipboard now has the overnight frames clipped to it, and the walk goes to the places the frames came from.