Image dataset search and filtering to find the frames a model never saw

The dataset for the yard camera at the distribution centre has a few thousand labeled frames and looks healthy from the outside. Forklift, truck, person and pallet are all well represented, the boxes have been through review, and the model trained on them scores well. Then the night shift supervisor asks why the forklift count on the yard dashboard drops to almost nothing after 9 pm, when she can see three of them out of her window.

The answer is in the dataset, and it takes a search to find it. Almost every labeled forklift was boxed on a frame captured between 7 am and 6 pm. After dark the yard is lit by two sodium lamps and a truck's headlights, a forklift is a pair of lights and an orange smear, and the model has seen perhaps a dozen of those. The frames exist on the recorder. Nobody ever labeled them, because nobody chose them.

Filtering by class and count shows where the labels thin out

The first pass is structural. Filter the dataset to frames that carry at least one forklift box, then look at when they were captured. On the yard set the distribution had a wall at 6 pm. Filter to frames with no boxes at all and the night frames appear in bulk, hundreds of them, captured and stored and never opened. Filter to frames with a truck box and no forklift box and the gap narrows further: trucks were labeled at night, because a truck under sodium light is still obviously a truck, and forklifts beside them were left blank.

Those blanks are worse than an absence. A night frame with a truck boxed and an unboxed forklift beside it trained the model that a forklift at night is background. The footage lesson says the unit of a dataset is distinct conditions, and the filter is how a team sees which conditions it has. On the yard set the condition "forklift, after dark" was present in the footage and absent from the labels.

Label count per frame is the other filter worth running. Frames with one box are the easy ones. Frames with eight boxes, a truck at the bay, two forklifts, a cluster of people, are the frames the yard actually produces at shift change. If the set has few of them the model has been trained on a quieter yard than the one it watches.

Asking for frames that look like a dock at night

The structural filters find gaps a person can name. The second pass finds the ones they cannot. Instead of filtering on a field, the team describes what they are looking for and asks the platform for frames that look like it: a loading dock at night, a forklift under a sodium lamp, a person walking between parked trailers in the rain. What comes back is a set of frames ranked by how closely they match, drawn from everything the camera recorded, labeled or not.

On the yard set the first such search, run on a Wednesday, returned a few hundred night frames with forklifts in them that no filter could have found, because nothing about them was labeled. A second search, for frames that looked like the one night frame the model had doubted most that week, returned its neighbours: the same lamp, the same bay, the same orange smear at the same angle.

That is what search is for on a dataset. The frames a model has never seen are, by definition, not in the labels, so no query over the labels finds them. A query over what the frames look like does.

My own view is that a dataset should be searched before every labeling batch is chosen, and that choosing frames by hand from a folder is the single biggest waste of a labeling budget I have seen. The frames a person picks from a folder are the frames that look like the frames already labeled.

The gap becomes the next labeling batch

The night frames the search returned were labeled as a batch of their own, the forklifts boxed under the sodium lamps by a person against the same guideline as the day frames. A few hundred labels, one Thursday evening's work for two people, and the class that had a dozen night examples had a few hundred. The model retrained on the corrected set and the night count on the dashboard came up to what the supervisor could see from her window.

The batch was chosen to be different from what the set already had, which is the opposite of how most batches get chosen. It is worth saying why that matters: a model learns from variety, and a labeling budget spent on frames that resemble the ones already labeled buys almost nothing. The footage lesson puts it as ten hours across twenty conditions beating a thousand hours across two, and the search is how the twenty conditions get found.

One aside that turns out to be true on every yard: the frames from the worst weather are the rarest in the set and the most valuable in it. A search for frames that look like the yard in heavy rain usually returns a handful, and every one of them is worth labeling.

Coverage is checked again after the batch lands

After the night batch was labeled, the filters were run again on the Friday. Forklift boxes by hour of capture: the wall at dusk was gone. Frames with a truck and no forklift at night: down to the frames where there was genuinely no forklift. The search for a dock at night, run again, returned frames that were now labeled, which is how a team knows the gap is filled rather than moved.

The doubted frames from the live camera are a running version of the same check. Frames the model is unsure of come back to a person, and if they cluster in a condition, a new lamp on the far side of the yard, a bay that was repainted white, that cluster is the next gap. A search for frames that look like the doubted ones finds the rest of the condition before the model has doubted all of them, and the labeling batch is chosen from that.

The supervisor now looks at the dashboard on her night shift and sees three forklifts. The dataset that produced that had the frames all along.