Writing a computer vision problem statement before anyone labels a frame

The request from the store operations team was one line: count the items on the shelf. The bread aisle camera had been on the ceiling for three years, the team had seen a demo of a model drawing boxes on products, and counting seemed like the obvious first thing to ask for. A labeling team could have started on Monday. What they would have labeled is the question, and the request did not answer it.

A problem statement is the document that answers it, and it is short. It names the classes, the geometry, the action and the target, and it is written before the first frame is labeled, because every one of those four changes what a label is.

The class is the thing the store will act on, so name it

Count the items sounds like a class, and it is not. Items on the bread shelf are loaves, rolls, bagged and unbagged, in a dozen product lines, and a count of all of them together answers no question the store has. What the store acts on is an empty facing: a slot on the shelf with nothing in it that should have something. The class is empty facing, and a second class, occupied facing, so the model has something to tell it apart from.

That reframing, agreed on the Monday the labeling was meant to start, cuts the labeling to a fraction of the original. Nobody has to draw a box around every loaf, or teach the model to tell a bloomer from a tin loaf. The what vision can see lesson has the test that should be run on every proposed class: could the best person in the store make the call from this camera's frame alone. Empty facing passes. Product line, from a ceiling camera at that angle, mostly does not.

Bread shelves empty from the front because shoppers take the nearest loaf, so a gap at the front of a facing is not an empty facing, and the merchandiser faces up the shelf twice a day for exactly that reason. The guideline says where the gap has to reach before it counts, and that sentence came out of the problem statement rather than from a labeler's guess.

The question decides the geometry, a box, a polygon or a point

Once the class is empty facing, the geometry follows. Where is it, and how many, is a box question, and a box on each facing is what a person can draw quickly and check quickly. If the store had asked how much of the shelf is empty, as a fraction of its length, the answer is an area, and that is a polygon along the shelf edge. If it had asked which product is out, the answer is a label read off the shelf edge, which is a different camera position and a different project.

Choosing the geometry before labeling matters because the geometry is the most expensive thing to change afterwards. A dataset of boxes cannot be turned into polygons without drawing them all again, and a project that discovers in October that the stakeholder wanted an area has a box dataset and no budget. The use cases catalogue lists, for each question, the label geometry it forces, which is a useful check that the statement and the geometry agree.

The action a detection triggers is part of the statement

A detected empty facing is a row in a table until the statement says what happens next. For the bread aisle the answer was a line on the store's morning list, with the frame, so that whoever faces up the shelf at 7 am starts with the gaps the camera saw. An alert as a rule written as a sentence, with a severity and a cooldown: an empty facing on the bread shelf for more than an hour, routine, to the morning list.

Writing the action down changes the labeling too. Because the action is a morning list rather than a page to a manager's phone, the model can be tuned to be slow to flag and quick to send a doubtful facing to a person, and a doubtful facing at 3 am costs nothing. Had the action been a page, the false alarm cost would have been a manager woken for a loaf, and the threshold, the review policy and the guideline would all have been written differently. The action is what sets the cost of each kind of error, and the costs are what the labeling has to be built around.

The target is a sentence with a number in it, written before labeling

The last line of the statement is the acceptance sentence: what the model has to do, on the store's own frames, for the project to be called done. For the bread aisle it read, in effect, that the morning list should carry nearly every empty facing the merchandiser would have found on the first walk, with few enough false alarms that the list is still read by the second week. The number in that sentence belongs to the store, and it is measured on a fortnight of frames held out by time from the store's own camera, per class, never on a demo set.

My own rule, and I hold to it against the pressure to start, is that no frame is labeled until that sentence exists. A statement with no target is a project with no end, and a labeling team pointed at it on Monday will label until the budget runs out and then be asked why the model is not done.

The problem statement decides more than the YOLO version does

The architecture is the question every first project asks first, and it is the one that matters least. A YOLO model of one generation or the next, trained on boxes around empty facings drawn against a clear guideline from the ceiling camera's own frames, will find empty facings. The same model trained on boxes around items, with no guideline and no target, will produce a count nobody acts on, and no change of architecture recovers it. The statement is where the outcome is decided, and it is decided before a single training run.

That is also where most computer vision projects that fail actually failed. Scoping the visual signal and the decision it informs is the first conversation we have with every buyer, and it is the cheapest mistake to prevent, because it costs a page and an hour, and the alternative costs a dataset.

A statement survives contact with the camera or it is rewritten

The bread aisle statement was tested against the camera before labeling began, with a dozen frames from the ceiling camera at 7 am, at noon and at 9 pm, in front of the merchandiser. She could call empty facings on every one, which settled the class and the geometry. She could not tell the wholemeal from the seeded from that angle, which settled the question the store had also half-asked, and moved it to a different camera and a later project.

LexData takes the shelf model through its whole life from there. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the aisle camera the store already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The words typed at the start of that sentence are the problem statement, and everything after them is built on whether it was written down.