Automated image labeling, where a model proposes and a person makes it true

The drone flew the right-of-way on a Tuesday and came back with four hours of footage: a pipe on concrete cradles across scrub, the same view every second. Rust shows in a handful of places, a crew of two stands near one of the cradles, and an excavator is parked closer to the line than anybody would like. Somebody now has to turn that into labels. Boxes on the pipe and its cradles, on the rust and the people and the machine, on enough frames that a model can learn to find them on the next flight.

Done by hand, that is weeks of dragging rectangles across nearly identical frames. Done by a model alone, it is fast and confidently wrong in the places that matter. The working answer is the model proposes and a person decides.

Grounding DINO puts a box wherever the sentence points

The models behind automated labeling, Grounding DINO among them, take a sentence rather than a fixed list of classes. Give an open-vocabulary detector the words "pipe and concrete cradle and rust, person, excavator" and it returns a box for each thing it can match to a word, on every frame, without ever having been trained on your footage. A segmentation model can then turn each box into an outline where the shape matters, which for rust it does.

The name is a small joke on a lineage: DINO was a detector paper whose title abbreviated to a dinosaur, and the grounded version added the sentence.

What these models are good at is common, visible things described in common words, a person or a machine or a pipe. What they are weak at is anything the site's own vocabulary names and the internet's does not, a specific fitting, a specific failure, and anything small or partly hidden. On the right-of-way the cradles come back well and the rust comes back as a mix of rust, shadow under the pipe and the rusty scrub beside it.

The prompt is a labeling standard, calibrated on a sample

The words in the sentence decide what comes back, so the sentence deserves the same care as a written labeling standard. "Rust" and "corrosion on the pipe surface" produce different boxes. "Person" finds the crew; "worker" may not. My view is that the prompt should be written by the person who will review the labels, because that person will otherwise spend the review pass correcting the consequences of somebody else's word choice.

Calibrate before the run. Take a few hundred frames that cover the whole flight, from the shaded cradle and the crew to the bright open stretch, and run the sentence against them. Look at what came back for each word. Tighten a word, drop a word, add a threshold for the class that returns too much. Then run the whole four hours. The labeling with Lexi guide makes the same point in its first step: write the prompt like a work order, in the words the field crew would use.

A person checks each label before anything trains on it

In LexAnnotate you type what to look for, Lexi puts a box on every frame, and the boxes arrive as drafts. The review pass is where they become labels. A reviewer confirms what is right, fixes what is close and rejects what is wrong, frame by frame, and a frame nobody has looked at is not yet labeled.

Reviewers anchor on what is already drawn. A plausible wrong box gets accepted more often than a blank patch gets a missed box drawn in, which is why the review is done against the frame and against a written standard rather than against the model's output alone. The rust box that sits half on the shadow is the case to watch: it looks like a label, and it teaches the next model that shadows are rust.

Labels come back from that pass at up to 99.9% accuracy, and that figure belongs to the pass, never to the proposals.

The proposals are weakest exactly where the labels matter most

Three things on Tuesday's flight are rare: the rust, the crew and the excavator. The pipe and the cradles are in every frame. A dataset built from what the model proposed confidently would be a mountain of pipe boxes and a thin sample of the classes anybody flew the drone to find.

The how much footage lesson puts the rule plainly: count instances of the thing you care about, never hours. For the right-of-way that means a person goes looking for the rust frames and the excavator frames and labels those deliberately, including the ones where the proposal was empty. The same person draws the edge cases the model has no word for, the cradle with the pipe lifted off it, the patch of rust seen at a glancing angle, the crew member half behind the machine.

Keypoints and poses are outside what these proposal models do at all, so a job that needs the excavator's boom position is a hand-labeled job from the start.

Corrections compound on the next batch

The corrections from the first review are training material for the next proposal pass on the same footage. The second batch arrives cleaner than the first, the reviewer spends the pass on judgement rather than on dragging corners, and the dataset grows at the pace of the reviewer's attention rather than the pace of the flight.

Exports carry the verified labels only. The dataset leaves as COCO JSON, ready for any training pipeline, or stays in the platform and trains there. Either way, the frames still marked as awaiting review are settled first.

LexData takes the right-of-way model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the next flight's footage, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The 12k+ precision image annotations delivered in our oil and gas work were built this way, one reviewed batch at a time.

The four hours of Tuesday's flight are still on the drive. The labels, and the model that came from them, are what the next flight is measured against.