Bounding boxes, polygons or masks, pick the label by the question it has to answer

Three cameras, three questions. A thermal camera on the fence of a substation is asked, on a Tuesday night, whether anyone is at the fence after dark. A drone over a pipe rack is asked how much of the run has rusted since the last pass. A tractor camera at the edge of a field is asked where the crop stops and the headland begins. Each of those questions has a label type that answers it, and the label types differ in what they cost to draw and to check.

The expensive labeling mistake is rarely a bad label. It is the right label for the wrong question. A team buys the richest label type it can afford and draws a shape around every person at the fence, then finds a year later that the model it needed was answerable with a box at a fraction of the labeling time.

Bounding boxes answer where and how many

The fence camera's question is presence and count. Is there a person at the fence, how many crossed the line between midnight and six, is there a vehicle in the gate lane. A box answers all of that. It says where a thing is, roughly how big it is, and lets the model count instances, and it is the fastest label to draw and the fastest to check. What a box teaches at training time and reports at run time, and why it has to be tight, is in bounding boxes in computer vision.

For most monitoring and alerting work that is the whole question. A rule written as a sentence, "a person at the fence line after dark", sent to Slack with the frame, needs to know that a person exists and where, and nothing about the outline of their coat. The hazard zone use case is built on boxes end to end, and it should be, because the decision it supports is a yes or a no with a frame attached.

Boxes are also what most detection models predict. Label with boxes and the model trains directly on what it will output, which is the least surprising path from dataset to deployment.

Polygons answer how much and exactly where

The pipe rack's question is different in kind. The inspector does not want to know that rust exists; the last pass already established that. She wants to know how much of the run is affected and whether the patch under the second cradle has grown since March. That is an area, and a box around a rust patch is mostly not rust, so measuring from it gives a number that means nothing.

A polygon traced around the patch gives the area. Compare this pass's polygon to last pass's and the growth is a number the inspector can put in a report. The corrosion use case is where polygons earn their time, because the decision downstream is about extent rather than presence.

Polylines are the polygon's cousin for linear things: the pipe itself, a power line, a crop row. A single line along the asset with a width is cheaper than a polygon around it and answers the question the same way.

A polygon costs more than a box because it takes longer to draw and longer to check, and the price of a label follows the time. On the pipe rack the inspector still carries a tape measure for the patches she can reach, and the polygons agree with it.

Masks answer what every pixel is

The field edge is a question about boundaries. The tractor's navigation model has to know, for every pixel in front of it, whether that pixel is crop, headland, ditch or track, because the decision downstream is where to put the wheels. A box cannot say that. A polygon can approximate it, and for many fields a tight polygon is all the model needs. When the boundary itself is the product, the label becomes a mask, one class per pixel, and it is the most expensive label of the three by a wide margin.

Masks belong to questions where the edge is the answer: the field boundary a tractor steers by, a surface defect whose exact shape decides whether a panel is scrapped, medical imagery. If a team finds itself measuring things off the mask, it was worth drawing. If the team is only checking presence, it bought a scalpel to open the post.

In LexAnnotate the labels are boxes, polygons, points and tags, and a shape question is answered with a polygon drawn tightly enough to measure from. A project that needs true per-pixel masks is a different budget, and it is better to know that on day one than to discover it after a few thousand frames of polygons that were never going to be dense enough.

The label follows the decision downstream

A useful test before the first frame is labeled in LexAnnotate: write down the decision the model's output will feed, and ask what shape of answer that decision needs.

• A rule that fires or does not, a count at the end of a shift, an alert with a frame attached: a box.
• A measurement, an area, a comparison between two passes, a length along an asset: a polygon or a polyline.
• A boundary the machine acts on directly, or a shape that is itself the finding: a mask, if you can afford it.

The lesson on what a vision model can see draws the same line from the model's side: presence is a detection problem, extent is a segmentation problem, and the two train differently. Choosing the label type is choosing which problem you are asking the model to solve.

My own view is that most monitoring projects never need anything past a box, and the ones that do know it on the first day, because someone in the room has to measure something and says so.

Start with boxes and upgrade the classes that prove they need more

The honest default is boxes on every class, a first model trained on them, and a look at what it cannot answer. The fence camera will be finished at that point. The pipe rack will show that the rust class needs an area for the comparison with March, and that class alone gets polygons on the next pass while the cradle and the pipe stay as boxes. The field edge will show that a box was never the right shape, and that project moves to polygons or masks with the evidence in hand.

That is also how the loop treats label types afterwards. The live model doubts the frames it cannot read, and those frames come back to a person. The correction on a rust patch is a polygon because the project's ruling for that class says so, while the correction on a person at the fence is a box. When the corrections cross the project's threshold a new version is trained, and the new version replaces the old one with no downtime. A class is upgraded when its corrections show it needs upgrading, and the classes that never needed more than a box keep the cheapest label there is.

The first model's list of questions it cannot answer is the polygon budget. Until it exists, the budget is a guess.