Fish size measurement on a grading line with a fixed camera

At 7 am on a trout farm the grading channel is running. Fish come down a shallow flume one or two at a time, water moving fast enough to keep them straight, and a grader at the end of the channel with a hand net flicks each one into a tank by size. The grader has a wooden board with three notches cut in it and has not needed the board for years.

A camera over the flume can size every fish that passes without anyone lifting a net. The catch is that a camera does not know how long anything is. It knows how many pixels long it is, and the whole job is the conversion.

Among computer vision applications, sizing needs a scale in the scene

Detection gives a box around each fish. The box is in pixels, and pixels become centimetres only when the camera's view has been calibrated: a fixed height over the flume, a known width of channel, and a marker painted on the channel floor at a known length. Most computer vision applications can skip this. Anything that measures cannot.

The marker matters more than the camera. A line 1 metre long painted on the flume floor, in view of the camera, is read on every frame, and the pixels-per-centimetre for that frame come from it. If the camera drifts, the marker drifts with it and the scale is corrected without anyone knowing it happened. My own view is that a farm should paint the marker before it buys a better camera, because the better camera without the marker is a more precise way of being wrong.

The livestock monitoring use case is about the same animals in a pen, where holding an identity across the frame is the hard part. A flume sidesteps that by design: it passes one fish at a time, and a fish that has left the frame is not coming back.

Fish are boxed and sorted into classes before anyone measures a millimetre

The first version of the grader camera does what the grader does: three classes, small, medium and large, decided from the length of the box against the calibrated scale. You type "fish" once, Lexi puts a box on every fish in every frame, and a person checks the boxes before anything trains. The size class is assigned downstream from the box length and the thresholds the farm chose, so the model never has to learn what "medium" means, which is good, because the thresholds change with the market.

The box is a rectangle and a fish in a flume is rarely straight along it. A fish crossing at an angle has a box longer and taller than the fish, and its box length overstates it. A fish with its tail flexed reads short. For a three-class sort with wide bands that error is inside the band most of the time, and the grader at the end of the channel catches the ones on the edge.

The agriculture work behind our numbers, 80k+ image annotations in production, was built on frames like these: animals and produce on the farm's own cameras, boxed and checked by a person, at the angles and the water and the light the farm actually has.

Keypoints give true length when the tolerance demands it

A farm selling into a market that pays by the gram at the size boundary needs a length, not a class. The box cannot give it. Two keypoints can: the tip of the snout and the fork of the tail, placed on every fish, with the distance between them in pixels converted through the marker's scale.

The labeling rule is what makes this work, and it is the whole of the labeling work. You type the two points once and Lexi proposes them on every fish, and a person checks their placement. The snout point goes on the tip of the upper jaw. The tail point goes at the fork, not the tip of either lobe. Every label follows that, on a fish lying straight and on a fish curled, because a model trained on points that wander learns to wander. The person checking the labels looks at nothing else.

With two points the angle of the fish is known too, and the length is measured along the fish rather than along the frame. A fish crossing at forty degrees measures the same as one crossing straight.

A bumped mount is the day every fish reads large

The failure to expect is the mount. The camera over the flume is on a bracket above moving water, and somebody leans on it during a wash-down, or the bracket is re-tensioned after a bolt works loose. The camera is now a few centimetres lower and the same fish covers more pixels. Every fish reads large, the medium tank fills faster than it should, and the grader with the wooden board notices by tank 2.

The drift catalog files this as a camera moved: someone nudged a lens and the model has been looking slightly past the thing ever since. On a sizing line the effect is not that the model misses the fish. It finds every one, and mis-sizes all of them by the same fraction, which is worse, because nothing about the frames looks wrong.

The marker absorbs most of it. If the scale is read from the painted line on every frame, a lower camera makes the line longer in pixels and the conversion corrects itself. What the marker cannot absorb is a tilt, since a tilted camera stretches the far end of the flume more than the near end. The fix for a tilt is the old one: re-verify the scale at both ends of the channel, and relabel a short window of frames from the new framing.

The grader keeps the board. When the camera and the board disagree on a fish, the board is what the farm believes, and the frame goes to a person.

Overlapping fish and splash are the frames that come back

LexData takes the grading model through its whole life. You type what to look for, Lexi puts a box and two keypoints on every fish in every frame, and a person checks each label before anything trains on it. The model then watches the camera over the flume, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime.

The frames that come back in the first weeks are predictable. Two fish overlapping in the flume despite the flow. A splash across the marker. Glare off the water at the hour the sun reaches the shed door. A fish belly-up with the fork hidden. Each is a correction, and when the corrections cross the project's threshold a new version trains on the farm's own footage, with the glare hour in it.

A fish that reads at a length no trout in that channel could be is held rather than sorted. It goes to the grader with the frame, and the grader, who has the board, decides.