Automated sorting with computer vision, from the camera over the conveyor to the diverter

In the first week of the harvest the grading conveyor at the packhouse runs from 6 am, and the apples come down it faster than a person can look at them. Three graders stand along the belt anyway, pulling the bruised ones the optical sorter missed and the split ones it called bruised. The sorter was set up for last year's fruit. This year's crop came off the trees a week early after a hot August, and the russet on the shoulder of the fruit sits where the sorter expects a bruise.

The camera over the belt sees every apple. What it does with that depends on what it was taught and on who checks it when it is wrong.

Object detection puts a box on every apple before anything is judged

The first job is finding each piece of fruit as it passes. On the packhouse's belt with cups, that is easy; on a flat belt where the apples touch, it is a small-object problem with occlusion, and a box on each apple has to stay tight to the fruit rather than the pair. Object detection with one class, apple, gives the model something to grade, and the box's width across the belt gives a size before any defect is considered.

Tracking carries the box along the belt. The same apple appears in a dozen frames between the camera and the diverter at station 3, and a grade decided on one frame and lost on the next is a diverter firing at the wrong apple. Instance tracking across frames turns those frames into one apple with one grade.

The defect and the size are two answers from the same box

Size is read off the box. The defect is a second question inside it: bruise, split, russet, sunburn, insect damage, or clean. On the packhouse belt in September the classes are the grader's words, and the grader's ruling on the shoulder russet, "that is leather, it goes to juice", is the ruling the class list has to carry.

Whether the defect needs a mask depends on whether the grade depends on how big it is. A split is a reject at any size. A bruise is a reject above a certain area and a second grade below it, and that area comes off an outline rather than a box. The surface defect detection use case makes the same argument for machined parts: disposition is a severity call, severity is size and location, and only a mask carries the size. On fruit, the mask goes on the bruise class and the box does for the rest.

An aside from the belt: the graders call russet "leather" and sunburn "blush", and neither word appears in the sorter's manual.

The diverter acts on the model's call within the belt's travel

Between the camera and the air jet at station 3 the belt travels for well under a second, and the grade has to exist before the apple reaches the jet. That is the reason the model runs on a runner beside the belt rather than anywhere else. Frames stay in the packhouse, the call goes to the diverter's controller as a signal per apple, and what leaves the site is the count per grade and the frames the model doubted.

The controller's side of this is a timing problem the packhouse already solved for the old sorter. The camera's side is a class and a position per apple, delivered to the controller in the time the belt allows. The deployment guide covers why the enclosure and the decode matter more than the benchmark for a box that sits beside a belt all day.

The doubtful apple goes to the grader's table rather than the reject bin

A model asked to grade every apple will be unsure about some of them. A blush that could be sunburn, a mark that could be a bruise or a stem scar. The question is what the line does with the doubt, and the answer that costs least is a third lane. Clean goes on, reject goes to juice, and doubtful goes to the grader's table at the end of the belt where a person makes the call.

That table is the review queue. Each apple the person grades there is a frame with a label on it, and the label is what the next version of the model trains on. On a hot week in September where the russet is new, the table fills, the graders rule on it, and the corrections are the signal that the model needs to see this year's fruit.

My own view is that the sort should always be tuned so the model's doubt goes to a person and never to the reject bin. Fruit sent to juice by a model that was unsure is revenue lost quietly, and nobody ever counts it.

A new cultivar and a new season are when the model needs retraining

The sorter that was set up for last year's fruit was not wrong last year. The crop changed. A new cultivar arrives with a different ground colour, an early harvest ships fruit with more russet, and the light in the packhouse changes when the roof panels are cleaned in October. The drift catalog covers the slow version as the season turned, and on a packhouse belt it arrives in a week rather than over months, because the whole crop changes at once.

The signal is the grader's table. Corrections landing in a consistent direction, the graders overriding "bruise" to "leather" thirty times before lunch, is the model asking for this year's fruit. The frames are already recorded. A short window of them, labeled by the graders who already ruled on them, is the retraining set, and the new version replaces the old one on the runner with no downtime, mid-harvest, while the belt keeps running.

The graders' corrections are the next version

LexData takes the grading model through its whole life. You type what to look for, Lexi puts a box on every apple in every frame, and a person checks each label before anything trains on it. The model then watches the belt camera the packhouse already has, on a runner beside the conveyor. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The graders' table in the first week of the harvest is where the second week's model came from.

That is the shape of the agriculture work behind our numbers, 80k+ image annotations in production, and most of them are fruit that a person looked at and ruled on. The graders do not leave the belt when the camera arrives. They move to the end of it, and what they see there is the fruit the model was honest enough to ask about.

By the third week the table is quieter. The russet is in the model, the cultivar is in the model, and the graders are back to pulling the splits the belt hides at the seam.