Computer vision in agriculture, from the sprayer boom to the packing line

The sprayer leaves the yard at 5 am because the wind drops at dawn, and the cameras along the boom are looking at sugar beet at the four-leaf stage with fat hen coming through between the rows. Every camera sees a strip of soil, two crop rows and whatever has germinated since the last pass. The decision it has to make, at the speed the boom is travelling, is whether the green thing under nozzle twelve is beet or weed.

That is one camera on one farm. The same farm has a fixed camera over the packhouse belt, and the contractor's orchard down the road has one on the front of a tractor. Each is a different question on different footage, and the footage changes every week by design.

The sprayer aims at a region, so the weed needs an outline

A box around a weed is mostly soil and, when the leaves overlap, some crop. The nozzle fires at whatever is in the box, and the saving from spot spraying comes from hitting the weed and missing the beet beside it. So the label is an outline, drawn where the weed's leaves end, and the model learns the boundary between two green plants at ground level under the same wind. The weed identification use case is built on that geometry, and on the fact that the machine has to decide while it is moving.

The class list uses the names the operator uses. On the beet farm that is fat hen, redshank and volunteer potato, and an alert or a report that says "broadleaf weed" means nothing to the person who will walk the field afterwards.

You type those classes once, Lexi proposes the outlines on frames from the boom cameras, and a person who knows the field checks them. At the four-leaf stage a beet seedling and a fat hen seedling are close enough that the checking is where the accuracy comes from.

Disease shows as a region on the leaf, and the early symptom looks like dust

The second question on the same crop arrives in July, when the agronomist walks the headland and finds the first lesions of leaf spot low on the canopy. The treatment decision is driven by how much of the block is affected, so the output is a share of leaf area and the label is again a region rather than a box.

The difficulty is that the symptom worth catching is the early one, and the early one looks like sun scald, dust or a splash of soil. The distinction is a few shades of colour under whatever light the morning brought. Frames the model is unsure of are the right frames to send to the agronomist, because a person with the frame and the field in front of them can settle in a moment what a model trained on last year's lesions cannot.

The packhouse camera grades what the field camera could not see

Back in the yard the fixed camera over the packhouse belt has the easiest footage on the farm. The light is controlled, the belt speed is known, the fruit is presented one layer deep, and the grader's standard is written on a sheet by the line: bruise over a given size, russet over a share of the skin, a split stem, a misshape. Each is a class, each is a box on the fruit, and the model runs on the same frames the grader looks at.

My view, which is not the one most growers start from, is that the packhouse camera should be the first model a farm trains rather than the boom. It proves the loop on footage that behaves, and the grader who checks its labels is the same person who will trust or distrust the field cameras later.

Across the agriculture work we run, 80k+ image annotations are in production, and a large share of them are this kind: a controlled belt, a written standard, and a person checking the model's boxes against it.

Under the canopy the tractor camera is what keeps the machine on the path

The orchard down the road has a different problem. Under a full canopy the satellite fix drops out, and the tractor pulling the mower or the harvest platform needs to know where the row is from what the camera in front of it sees. The classes are the tree row on each side and the drivable path between them, and the label is a region of ground that the machine may drive on.

This is the footage where the season shows most. The rows in the picture at the top of this post are in leaf; the same lane in February is bare wood with sky between the branches, and the path is the same strip of grass under a completely different picture.

Labels go stale on the calendar, so retraining follows the season

Everything above shares one property: the crop changes shape, colour and density every week of the season, on purpose. Beet at four leaves in May is a closed canopy by July. A weed model trained on the May pass meets a different field on the June pass, with the same weeds in it. The drift catalog calls this the season turned, and on a farm it is the normal case rather than the exception.

The signal is the correction rate at review. When the agronomist is overriding the model on the June frames in a consistent direction, the labels from May have gone stale, and the June frames the model doubted are exactly the ones to label and fold in. Compare each camera against the same week of its own history, or the monitoring will alarm every time the crop grows.

LexData takes each of these models through its whole life. You type what to look for, Lexi puts an outline or a box on every frame, and a person checks each label before anything trains on it. The model then watches the cameras the farm already has, in the cloud, on your servers, or on a runner beside the recorder in the packhouse. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The June frames are what the July version learned from.

The first season is the expensive one

A model that has seen one full season of a crop, from emergence in April to harvest in October, is in a different position from one that has seen a fortnight of it. The first year is where the corrections matter most, because every growth stage arrives for the first time, and the second year is mostly the weather and the new fields. The wider agriculture picture on this site, from yield counts to livestock, is built on the same footing: cameras the farm already runs, a person who knows the crop checking the labels, and retraining that keeps pace with the calendar.

The operator on the beet farm still walks the field after the sprayer, with a notebook. He writes down where the model was wrong in the same notebook his father used for the same purpose, before the cameras.