A collaborative data annotation workflow run as a pipeline

The bottling plant has three cameras on line 2, one over the filler, one over the capper and one at the labeler, and a month of footage from each. Four people are going to label it: two in the plant's quality team, two brought in for the month. On the first morning, without a plan, two of them opened the same folder of filler frames and started boxing the same bottles. By lunch the folder had two sets of boxes on half its frames, drawn to two different ideas of where a bottle ends, and the afternoon went on deciding which to keep.

A labeling effort with more than one person in it is a pipeline whether or not anyone designed it. The five stages below are the ones that, designed on purpose, keep four people producing one dataset.

01Batch the frames by camera and shift

The month of footage is sampled to frames, one every couple of seconds, and the frames are grouped into batches before anyone labels one. A batch is one camera, one shift, one day: the capper camera on the Tuesday night shift is a batch, a few hundred frames that share a lens, a light and a crew. Batches are the unit everything else in the pipeline moves in.

Grouping this way means a labeler sees the same scene for a whole session, which is faster and more consistent than a random scatter from three cameras. It also means a problem with a batch, a fogged lens on the night shift, a camera nudged during a wash-down, is contained in one batch and visible in its numbers.

02Assign each batch to one person

Every batch has one labeler, and a frame is in exactly one batch, so no frame is labeled twice by accident. The assignment is written down where all four can see it. The two plant staff, who know what a missing cap looks like at line speed, take the capper batches. The two who joined for the month take the filler and labeler batches, where the classes are simpler.

Once a batch is assigned it moves through states: open, labeled, in review, approved or sent back. A batch that has sat in "labeled" for two days is a batch nobody has reviewed, and the state is what makes that visible. The alternative, a shared folder, has no states, and the answer to "is the Tuesday night capper batch done" is a walk across the office.

03Attach the instructions and the bounding box rules to the batch

The guideline is a document the labeler can open from inside the batch. It carries the class list in the plant's own words (bottle and cap and missing cap, label and skewed label) and a ruling with a picture for each case the first morning turned up. A bounding box on a bottle stops at the shoulder, because the neck is where the cap class lives. A cap that is on but crooked is a cap. A bottle cut off by the frame edge is boxed to the edge.

The labeling guide says to keep classes stable and treat the definitions as a living document. Attaching the document to the batch is how the living part works: when the reviewer rules on a new case on a Wednesday, the ruling goes into the guideline, and every batch opened after that carries it.

The instruction the four people are labeling against is always the current one.

Lexi proposes boxes on every frame before the labeler opens it, so the labeler's work on the capper batch is confirming, fixing or rejecting proposals rather than drawing from blank. The guideline still governs what a correct box is. A proposal that stops at the neck rather than the shoulder gets fixed by every labeler every time, because the picture in the guideline says so.

04Review every label and send the batch back or approve it

One of the plant staff is the reviewer for the month, and every batch passes across her screen before it is approved. She opens each frame with the labels drawn on, against the guideline, and either approves the batch or sends it back with the frames marked and a reason on each. A batch sent back goes to the labeler who did it, with the reason, and comes back through review again.

Most of the rejections in week one were the shoulder rule, applied three different ways by three labelers. By week two they were the rare cases, a bottle lying on its side after a jam, a cap on the belt beside its bottle. Each became a picture in the guideline. The rejection reasons, kept per batch, are the record of what the guideline was missing.

Reviewers anchor on drafts, and a plausible wrong box is accepted more often than a blank frame gets a missed one drawn in. So the reviewer also takes a small random sample of frames from each approved batch and redraws them from scratch, and the disagreement between her drawings and the approved labels is the honest number for the week. That pass, on every label, is what gets labels to up to 99.9% accuracy, and it is the stage teams cut first when the month runs short.

My own view is that the reviewer should never also be a labeler on the same batches. A person cannot review their own anchoring.

05Train only on the approved set

The training set is the union of approved batches and nothing else. A batch in "labeled" or "in review" or "sent back" is not in it, however good it looks, and a batch approved on Thursday is in the training run on Friday. The state of the batch decides, so the question of what the model trained on has an exact answer: these batches, approved by this reviewer, on these dates.

That exactness is what makes the next version explainable. The retraining lesson argues that corrections from the live line are the highest-value labels a project gets. Those corrections arrive through the same pipeline: a doubted frame from the capper camera comes back as a batch of its own, and it is labeled and reviewed and approved before it trains. When the retrained model is better on the night shift, the approved night-shift batch is why, and when it is worse, the batch that went in is the first place to look.

The bottling plant's four labelers finished the month with one dataset, one guideline that had grown by eleven rulings, and a reviewer's folder of sent-back frames that the next person to join the team will read on their first morning. The shared folder from the first day was deleted.