AI data labeling workflows, three ways to label footage and when each one pays

A pipeline operator has a season of drone flights along a desert right-of-way between pump station 4 and the compressor and wants a model that finds three things: the concrete cradles the pipe rests on, rust on the pipe surface, and a leak. The cradles are in nearly every frame. The rust is in many. The leak is in none, because there has not been one on this section, and the whole point of the model is to be watching when there is.

Those three classes want three different ways of being labeled, and a team that picks one workflow for the whole dataset pays for the mismatch on at least one class.

A right-of-way survey has three classes and one that never appears

The cradles are easy and numerous. A person can draw a box on a cradle in 1 second and there are hundreds per flight, which makes them tedious rather than hard. The rust is harder: it is a patch of colour on a curved surface, its boundary is a judgment, and two people will draw it differently unless a rule says where a rust patch ends. The leak is not in the footage at all.

That is the shape of most industrial datasets. A common class that costs time, a judgment class that costs consistency, and a rare class that costs invention. The labeling guide starts from the prompt and the class list, and the class list for this survey is where the three workflows get decided.

The surveyor's field notes for the season mention rust by kilometre marker, in pencil, with a sketch. They do not mention the cradles at all, because nobody counts cradles.

Manual boxes are slow and they are the reference set

The first workflow is the one everybody starts with: a person opens each frame and draws every box. On the cradles it is slow and accurate. On the rust it is slow and inconsistent until the boundary rule is written. On the leak it is nothing, because there is nothing to draw.

Manual labeling earns its place as the reference set. A few hundred frames from across the season, labeled by the surveyor who has walked the section since March, with the rust boundary rule applied carefully, become the set every other workflow is checked against. They are also the frames the first model trains on, before there is a model to propose anything. The cost is the hours, and the hours are the reason nobody labels a whole season this way.

Lexi proposes each bounding box and a person checks it

The second workflow is the one that labels the season. You type the classes once, cradle and rust, and Lexi puts a bounding box on every cradle and every rust patch in every sampled frame. A person then checks each one: confirm the box that is right, tighten the one that is loose, reject the one on a shadow that looked like rust from altitude, and add the cradle at the edge of the frame that was missed.

Checking is faster than drawing by a wide margin, and it is also better, because the person's attention goes to the frames and boxes that need it rather than being spread evenly across hundreds of identical cradles. The proposals on the rust are where the checking matters most, since the boundary rule is applied by the person on every patch, and the person's corrections are what teach the next round of proposals where a rust patch ends.

This is how the survey's labels come back at up to 99.9% accuracy: every proposed box passes a person before it trains anything. The 12k+ precision image annotations delivered in our oil and gas work were made this way, on footage that looks a great deal like this right-of-way.

My own view is that the checking pass is where a team's labeling skill actually lives, and that the teams who draw every box by hand are practising the wrong skill.

Synthetic frames cover the leak the drone has never filmed

The third workflow exists for the leak. There is no footage of one, and there will not be until the day the model was supposed to catch it. So a leak is put onto real frames from the flights near pump station 4. A wet stain on the ground beside the pipe, or a dark plume, rendered at a range of sizes and under the light the desert flights already have, with the box placed where the leak was put.

Those frames go into the training set alongside the real cradles and the real rust, and they never go into the evaluation set, because a model that finds its own rendered leaks has proved nothing. The first time a real leak appears, or a staged one from a controlled release the operator schedules, it becomes the first real example, and the synthetic frames start to retire.

The synthetic leak is the only way the model has a leak class on day one. It is a poor class, and a poor class that exists can doubt frames, and doubted frames are how it improves.

Each workflow pays at a different stage of the same project

The three are not competitors. Manual boxes pay first, on the reference set, before any model exists. Proposals with a checking pass pay across the whole season, once there is a model to propose. Synthetic frames pay on the one class the footage cannot supply. A project that used only the first would never finish; only the second would have no reference set and no leak class; only the third would have a model trained on rendered stains and nothing about cradles.

LexData takes the right-of-way model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the footage the drones bring back, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. After the first season, most new labels are corrections on doubted frames, which is a fourth workflow that the other three exist to reach.

The wider oil and gas work follows the same order on every asset from a well pad to a pipe rack. A reference set by hand, the bulk by proposal and check, and the rare hazard by whatever means gets it into the training set before it happens.

The QA pass is the same in all three

What the three workflows share is the pass at the end. Whether a box was drawn by a person, proposed by Lexi and confirmed, or placed by a renderer, it is checked against the class rule before it trains anything. Is it the right class, does it hug the object, was the rust boundary applied the way the rule says. The pass is the same on a hand-drawn cradle and a rendered leak.

That pass is also where the reference set earns its keep for the second time. A sample of checked proposals is compared against the manual frames, and the agreement between them is the number that says whether the season's labels are trustworthy. The surveyor with the pencil notes was the reference on the rust, and the checkers learned the boundary rule by disagreeing with the proposals until they agreed with him.