Process compliance tracking with a camera, each step in order and for long enough

The handwash station at the entrance to the ready-meal hall has a laminated sign above it. Wet, soap, scrub for twenty seconds, rinse, dry, sanitise, in that order. At 5:55 am the day shift comes through it in a queue, and the sign is followed exactly by the first few people and less exactly by the last few, because the line starts at 6 am and the supervisor is already on the floor. The auditor who visits in March will ask for the handwash records, and the records are a sheet on a clipboard that everyone initialled.

The camera over the station has watched every shift change since the hall was refitted. It was put there for the door.

Object detection reads each step of the procedure per frame

The model's job on each sampled frame is to say which step is happening. Hands under the tap, hands on the soap dispenser, hands scrubbing, hands under the dryer, hands at the sanitiser. Each is a class, and on the frames from the 5:55 queue the classes are readable from the position of the hands relative to the fixtures, which are boxed once and never move. Object detection on hands and on the fixtures gives the model the two things it needs to name the step: where the hands are, and what they are near.

There is no step called "nothing". A frame with no hands in it is the station empty, and that state has to be in the model too, because the sequence for one person begins when the station stops being empty and ends when it is empty again.

A timer per step turns frames into a duration

A class per frame is not a compliance check. The sign says twenty seconds of scrubbing, and twenty seconds is a count of consecutive frames in the scrub class. The layer above the model keeps a timer per step: it starts when the class first appears, accumulates while the class holds, and closes when the next step's class takes over. What comes out per person is a list of steps with a duration each, in the order they happened.

The order is checked against the sign. Scrub before soap is a skipped step even if every duration was long enough. A rinse that lasted two seconds is a step cut short. The sequence, the durations and the frames that produced them are the record, and the record is what the auditor in March was really asking for.

An aside from the sign: the twenty-second guidance on most of these laminated cards says to sing a well-known birthday song twice, and the timer is the first thing at the station that has ever measured whether anyone did.

The rule fires when a step is skipped or cut short, with the frames attached

The alert is a rule written as a sentence, "a scrub step under twenty seconds at the hall entrance", with a severity and a cooldown, approved before it goes live, delivered to the hygiene lead in Slack with the frames of that person's sequence attached. The monitoring guide covers how the rule is written and where it lands.

My own view, from running review queues, is that the alert should never name a person. It names a station and a time, 5:58 at the hall entrance, and what the hygiene lead does with that is a conversation on the floor rather than a record against a name. A system that names people gets switched off by the second month, and a system that names stations and times is still running when the auditor arrives.

Two people at the station at once is the case a per-frame model gets wrong

At 5:55 the queue is long enough that two people are at the double station together, and a per-frame classifier sees four hands and one soap dispenser. Which scrub belongs to whom is a tracking question, and it is the case the demo footage never contained because the demo was shot with one person at 10 am.

The fix is an identity per person across frames, so the timers run per track rather than per station. The frames where the tracks cross, one person reaching past another for the dryer, are the frames the model doubts, and they come back to a person to rule on. Those rulings are what teach the model the shift-change version of the station rather than the quiet one.

The assembly bench is the same problem with torque instead of soap

Move the camera to an assembly bench and the shape does not change. The procedure is fit the gasket, seat the cover, drive four screws in a cross pattern, connect the cable, and the model classes each step from where the hands and the tool are relative to the housing. A cover seated before the gasket is a skipped step. Three screws is a step cut short. The assembly verification use case covers the presence side of this, is every part there and the right way round, and the compliance side adds the order and the time.

The bench version has an advantage the handwash station lacks: the unit itself. Each housing has a serial, and the sequence record can be written against it, so the housing that reaches test with a loose cover has a record of the step that was cut short at 10:14 on station 5.

The review queue is where the procedure's edge cases get ruled on

LexData takes the station model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the station camera the plant already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The lines in our manufacturing work hold 99%+ accuracy maintained in production on that basis, and a compliance record is only worth keeping if the model behind it is being checked the same way.

The edge cases are the procedure's, not the model's. A person who rinses with one hand while reaching for a towel. A left-handed assembler who drives the screws in a different order that is still a cross. Each comes back as a doubted frame, the hygiene lead or the line lead rules on it, and the ruling becomes a note in the class definitions and a label the next version learns from.

The clipboard stays by the door through March. The auditor initials it, and then asks to see the record with the frames.