Computer vision for construction site safety, a warning before the worker and the excavator meet

At 7 am the excavator on the east side of the site is opening a drainage trench. The banksman is at the front of the machine where the operator can see him, and a groundworker walks behind the counterweight to fetch a shovel from the van. The operator slews to dump the bucket. Nobody is hurt, because the groundworker stepped back a moment early, and nobody writes it up, because nothing happened.

The camera on the pole over the compound saw all of it. What it could have done is send a frame to the supervisor's phone at the moment the groundworker's boots crossed into the slew radius, with the machine and the person boxed and the zone drawn between them.

Object detection gives the zone a machine to follow

The classes are people and machines, with the machines split by type because the zone depends on the type. An excavator's hazard is its slew and its bucket. A dumper's is its blind rear and its tipping body. A telehandler's is the boom and the load. So the model finds a person, an excavator, a dumper and a telehandler as four classes, and the zone logic reads the class before it draws anything.

Object detection here is the cheap part. A pole camera over a compound gives clear, well lit frames for most of the day, and people in hi-vis against mud are among the easier things a model is asked to find. The work is in what happens after the boxes exist.

You type the classes once, Lexi proposes the boxes on frames from that camera, and a person checks them before anything trains. The person's attention goes to the machine boxes, because a box that stops at the cab and leaves the bucket out has moved the zone away from the part of the machine that does the harm.

The danger zone is drawn from the machine's box, so it moves with it

A fixed exclusion zone works for a fixed hazard. A crane lift has one, drawn once on the frame around the swing radius, and the hazard zone intrusion use case is built on exactly that shape: a person box and a line to cross. An excavator digging a trench moves every few minutes, and a zone drawn around where it was at 7 am is a patch of empty ground by nine.

So the zone is computed from the machine's box on every frame: a margin around the box, wider for an excavator than a dumper, widest behind the counterweight where the operator cannot see. As the machine tracks across the frame the zone tracks with it. When the machine leaves the frame, the zone goes with it.

The banksman stands inside that zone all day. That is his job, and he is the reason the rule is written for a person entering the zone rather than a person inside it.

A pole camera measures distance in pixels, and pixels lie about depth

The hard part on a pole camera is perspective. A person standing well behind the excavator, on the far side of the trench, can sit inside the machine's box on the frame while being several metres from it on the ground. The camera has flattened the site, and a zone drawn in image coordinates inherits the flattening.

Two things help. The first is the mount: high and steep, looking down on the work area rather than across it, so that distance on the ground maps more honestly to distance on the frame. The second is the test point. The bottom of the person box is the feet, and the feet are on the ground plane, so the rule asks whether the feet are inside the zone rather than whether any part of the box overlaps it. A person's head overlapping a machine box from behind is perspective; their feet inside the slew radius is a person in the slew radius.

On the sites where this is set up properly, someone walks the zone with the operator watching the frame on a tablet before the 7 am dig, and the margin gets adjusted until the drawn zone matches the cones on the ground. The cones will be under mud by lunch. The zone will still be there.

Motion is the trigger, because a parked machine has no slew radius

An excavator with the engine off at the 10 am break, the operator in the welfare unit, and a groundworker leaning on its track eating a sandwich is a site at rest. A rule that alerts on presence would page the supervisor every break. The rule alerts on a person entering the zone of a machine that has moved in the last few seconds, and tracking the machine box from frame to frame is what tells the rule whether the machine is working.

That one condition removes most of the noise. It also means the first version has to track well, because a machine box that jumps between two identities looks like a machine that has just moved, and a jumping identity is the commonest cause of a false alert in the first week.

The alert carries the frame, and the supervisor sees who and where

The alert is a rule written as a sentence, with a severity and a cooldown, approved before it goes live: a person entering the zone of a working excavator, critical, to the site supervisor. What arrives on the phone, by Slack or by text, is the frame with the person and the machine boxed and the zone drawn, so the supervisor knows which machine and which person without opening anything. With the cooldown set to the length of a slew, one entry produces one message rather than a message per frame.

I think the alert should reach the operator as well as the supervisor, on a screen in the cab, and that a site which will not fit that screen is not ready for the camera. The supervisor is somewhere else. The operator is the one who can stop.

With a runner beside the site recorder the alert fires on site first, over whatever connection the compound has that morning, and the footage stays in the cabin. The alert written as a sentence is the same mechanism on every camera on the site.

Dust and rain are the frames that come back for review

The camera that gave clear frames at 7 am is looking through a dust cloud by eleven on a dry August day, and through rain on the lens by November. A person in a dust cloud is a shape with soft edges; a person in rain is a person behind a smear. The machine boxes hold better than the person boxes, because a machine is large and a person is not, and the days the person boxes fail are the days the trench is slippery and the site is loud.

These are the frames the model doubts, and they come back to a person on the safety team to confirm or correct. The drift catalog calls this rain, fog and dust: the conditions arrive for a week, detection thins, then recovers, and the frames from that week are rare in any training set and worth keeping.

LexData takes the site model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the pole camera the site already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The dusty frames from August are what the version running in November learned from.

The same camera can watch for the rest, one rule at a time

Once the people and machines are boxed on the east side camera, the other things a safety lead asks about are rules on the same boxes. A person without a helmet is a second model on the person crop. A person on the ground for longer than a fall would take is the person box changing shape and then not moving. A vehicle in the pedestrian route is a machine box inside a fixed zone. Each is a rule written as a sentence and approved before it goes live, and each arrives with the frame, so the supervisor is judging a picture rather than a count.

Start with the slew radius, as the site in this post did in August. It is the one where the time between the boots crossing the line and the harm is measured in seconds, and where the frame arriving on a phone can still change what happens next.