Writing a class ontology for defect labeling before the first box

Two inspectors on a bracket welding line look at the same fillet on a Monday morning. One calls the pitting along the toe porosity and rejects the part. The other calls it spatter, cosmetic, and passes it. Both are experienced, both are following the quality sheet, and the sheet says "porosity" without saying how many pores, how large, or whether pores under the fixture's shadow count at all.

Now imagine the same two people labeling frames for a model. Half the pitted toes are boxed as porosity and half are not, under the same class name, and the model learns the average of two opinions. The precision figure at the end of training looks like a model problem, and it was a definition problem, decided before the first box was drawn.

Each class needs four decisions written down before labeling

A class list is the easy part: crack, porosity, undercut, spatter, incomplete fusion. On the bracket line that list fits on one line of the quality sheet, and it is where the sheet stops. What the model actually learns is the set of decisions behind each name, and those get made differently by every person who labels unless they are written down.

For every class on the weld cell, four questions. Where does it end: does undercut include the discoloured band beside the groove, or only the groove? What is inside it: is a single pore a defect, or does porosity start at a cluster? What happens when it is partly hidden: a crack running under the fixture clamp, boxed to its visible edge or to its likely extent? And which side it is on: the same undercut on the show face and on the hidden face are different decisions on the line, so they may need to be different labels.

None of these have a right answer. They are conventions, and the only thing that matters is that everyone applies the same one. A page of examples, with a good box and a bad box for each hard case, does more than a paragraph of prose. I would rather have five classes with a page of examples each than fifteen with a line each.

The quality sheet on that line spells porosity two ways, and one of Monday's two inspectors has a firm view on which is correct.

Label at the finest level two inspectors can agree on

The instinct is to start broad, one "defect" class, and refine later. It is backwards. Merging classes is cheap: crack, porosity and undercut become "weld defect" with a remap and nothing is lost. Splitting is expensive, because the information was never captured. If the cell ever wants to route porosity to re-weld and spatter to the grinder, that decision has to be recoverable from the labels.

The limit is agreement. If two inspectors cannot reliably tell light spatter from surface porosity on the frames the camera produces, forcing the distinction produces noise. The right granularity is the finest level at which people agree, and the way to find it is to have two people label the same hundred frames in LexAnnotate and measure where they part.

Clean welds go into the set on purpose

On a good shift on line 3 the clean fillets outnumber the bad ones by a wide margin, and a training set that contains only frames with a defect teaches the model something false: that every frame has one. Deployed on the cell, that model finds defects that are not there rather than say nothing.

So frames with no defect at all go into the set, in roughly the proportion the line produces them, and "nothing here" is an answer the model has seen. Frames with several defects at once go in too, because the occlusion and crowding rules only get tested when a crack runs through a porosity cluster.

When precision and recall fall, check the definition before the model

Agreement between labelers is the health check on the definitions, and it is worth measuring per class rather than overall. When agreement on porosity drops in the Monday batch, one of three things is true: the definition is unclear, the distinction is genuinely hard on this camera's frames, or the tool makes it awkward to draw. Each has a different fix, and none of them is a bigger model.

The same logic applies after deployment. When precision and recall on one class slide while the others hold, the first question is whether the people correcting the model agree with each other. If they do not, the model is being trained toward two answers at once, and retraining on those corrections will not converge. Fix the definition, re-cut the disputed labels, then retrain.

A tightened tolerance makes every old label wrong

Quality moved the threshold in March. A pore cluster that passed last quarter is a reportable defect this quarter. The frames look identical, the model looks healthy, and it is confidently applying last quarter's rule to every bracket. The drift catalog calls this a spec change, and it is the most expensive failure on the list, because the model passes defects that are now reportable, consistently, at scale.

There is no input signal. The first evidence is people overriding the model in one direction, which is why we watch the override rate even when the accuracy figure looks fine. And there is no shortcut in the fix: the ground truth changed, so the old labels are wrong labels, and they are re-cut against the new rule before anything trains on them. This is the one case where keeping the old frames in the mix is a mistake, and retraining without starting over says so. Versions keep what they were trained on, so the before and the after both still exist when the auditor asks.

The definitions are the sentence you type first

On the platform the definitions are where the work begins. You type what to look for, in the words the line uses, and Lexi puts a box on every frame; a person checks each label before anything trains on it. The model then watches the cell cameras the plant already has, on a runner beside the recorder or on your servers. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The labeling doc covers the prompt, the classes and the verification pass in detail.

The weld quality inspection use case is the one this cell lives in, and the surface defect detection use case is its general form: disposition is a severity call, severity is size and location, and only a mask carries the length of a crack. Which is one more decision that belongs in the definitions before the first label rather than after the thousandth.