Tomato leaf disease detection from the greenhouse camera and the scout's phone

At 6:30 am the scout walks row 14 of a tomato house with a phone in one hand and a bag of clothes pegs in the other. On a plant two thirds of the way along, the lower leaves have yellow-brown spots with a ring pattern in them. The scout pegs the plant, photographs the leaf, and moves on. By the time the agronomist sees the photo at 10 am, the spots have a name and the row has a spray plan, and the plants on either side have the same spots.

A camera over the rows sees row 14 every day at 6:30 am and every other hour as well. What it can tell the agronomist depends on how the labels were drawn.

Object detection on a leaf needs a healthy class too

The obvious class list is the disease list, with early blight and late blight at the top, then leaf mould, then septoria spot, then damage from spider mites. A model trained on that list and nothing else has learned that every leaf it is shown has a disease, because every leaf it was shown did. Object detection on a healthy house then produces boxes on healthy leaves, and the scout stops trusting it by the end of the first week.

So the first class is healthy leaf, and it gets more labels than any disease. You type the classes once, Lexi puts a box on every leaf in every frame, and a person checks each box before anything trains. The person checking is not a labeler in the abstract; on a tomato house it is someone who has walked the rows, because the difference between an early lesion and a splash of nutrient solution is not something a labeling guide can teach.

The agriculture work behind our numbers, 80k+ image annotations in production, was built on frames like these: crops on the grower's own cameras, at the light and the angles the house actually has, checked by people who know the crop.

Early blight and late blight look alike at the stage that matters

The two blights are different organisms with different treatments, and at the stage the scout wants to catch them on row 14, a small lesion on a lower leaf, they look alike in a frame. Two labelers shown the same leaf will disagree, and a model trained on labels that disagree learns the disagreement.

My own view is that a class the labelers cannot agree on should not exist yet. Merge the two blights into one class, "blight lesion", until the agronomist has confirmed enough of them from the pegged plants to split the class with confidence. The model is more useful with one class it finds reliably than with two it guesses between, and the agronomist was always going to confirm the species in person before a spray plan.

The crop health and disease detection use case names the underlying problem: early symptoms are exactly what you want to catch and exactly what looks like ordinary variation, sun scald or dust, and the distinction is often a few shades of colour under uncontrolled light.

The clothes pegs turn out to be the best labeling tool in the house. A pegged plant is one the agronomist has looked at, and the frames of pegged plants are the labels the model can be checked against.

Mite damage is a texture rather than a spot

Spider mite damage does not look like a lesion. It is a fine stippling across the upper surface of the leaf, a pale speckle that spreads until the leaf looks dusty, and a box drawn around a "spot" makes no sense for it. The labeling rule is a box around the affected region of the leaf, edge to edge, with the stippling inside it, and the person checking the frames from row 14 looks at whether the box stops where the stippling stops.

That rule is written down and kept, because a model trained on some mite boxes that hug a patch and others that take the whole leaf learns a blurred idea of the class. The rule is the same for mould on the underside, which is a bloom rather than a spot and boxed as a region.

The scout's phone and the fixed camera are different footage

The scout's photo is taken at arm's length, in the row, with the leaf filling the frame. The fixed camera over row 14 is three metres up, sees a whole bay of plants, and a leaf in its frame is a few dozen pixels across. A model trained on the phone frames and run on the camera is looking for lesions at a scale the camera never shows it.

Both sets of frames are useful, and they train different things. The phone frames teach the model what a lesion looks like up close and are the ones the agronomist confirms. The camera frames are what the model will actually watch, and they need their own labels, at their own scale, from the camera's own angle. The scout's photos go into the same project as a second camera, tagged as such, and the person checking labels knows which is which.

A new season's light is what has to be relabeled

The tomato house in July is lit by the sun through the roof. In October the supplementary lamps come on at 4 am, the sun sits low and comes through the side walls, and the whole house is a different colour. Leaves that were green under July light are a warmer green under lamp light, and a lesion that was yellow-brown is now closer to orange. The model that was right in August is unsure in October, and it starts sending frames back.

The drift catalog files this as the season turned: foliage, sun angle and lamp light rewrite the background the model learned, and accuracy slides over weeks rather than falling over. On a greenhouse the slide is on a calendar, and the relabeling can be scheduled for it. A window of frames from the first week of lamp light, checked by the scout, folded in before the model has a month of doubt behind it.

LexData takes the disease model through its whole life. You type what to look for, Lexi puts a box on every leaf in every frame, and a person checks each label before anything trains on it. The model then watches the cameras over the rows, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The October frames are in the version that watches the house in November.

The agronomist's confirmations are the corrections

The signal that the model needs work is the agronomist. Every frame the model boxed as blight that the agronomist looked at and called nutrient splash is a correction. Every pegged plant the model did not box is one too. The rate of those, per house each week, is what says whether the model still describes the crop, and it rises in the first week of October before anyone looks at a metric.

Asking the footage which rows had lesions this week gives the agronomist an answer with the frames behind it and changes nothing in the model. The corrections at review are what the next version learns from, and the pegs are how the agronomist decides which frames to look at first.