In this lesson
Corrections are the highest-value labels you will ever get
A label drawn on a randomly chosen frame teaches the model something it may already know. A correction on a frame the model got wrong teaches it exactly what it does not know. The second is worth many times the first.
Which means the review queue is not overhead, it is your data collection pipeline. If your operators are fixing model output and those fixes are not flowing back into training, you are throwing away the best dataset you have.
The underlying effect - frames the model struggles with are worth more - has been measured at production scale. When NVIDIA A/B-tested it for autonomous driving, frames selected by model disagreement improved night-time pedestrian detection roughly three times more than the same volume of manually curated frames, and more than four times on cyclists. The frames the model struggles with are simply worth more.
Do not throw away the old data
The instinct when accuracy drops is to rebuild on fresh footage. This usually makes things worse: you fix the new condition and lose the old one, because the model no longer sees examples of the world it used to handle.
Keep the original set and add to it. If the new condition is rare in the combined set, weight it up rather than deleting history. The goal is a model that handles both winters, not one that swaps which winter it fails in.
The failure has a name in the literature - catastrophic forgetting. A network trained sequentially on new data overwrites what it knew, and the standard defense is exactly the one above: keep old examples in the mix every time you train.
The exception is genuine concept drift. If a definition changed, old labels are now wrong labels and must be re-cut rather than kept.
Hold out the future, not a random slice
A random train/test split leaks. Frames from the same hour, the same camera, the same pass end up on both sides, so your test score measures memorisation and reports it as accuracy.
Split by time and by site instead. Train on everything up to a date and test on what came after, or hold out entire cameras. The number will be lower and it will be true, and it is the only number that predicts what happens when you deploy.
Know what better means before you promote
Aggregate accuracy hides the trade you actually made. A new model can be a point better overall while being significantly worse on the rare class you built the system to catch.
Compare per class, and compare on the conditions that triggered the retrain in the first place. Then ship it alongside the current model before it replaces it: run both, compare on live traffic, and promote only when the new one wins on the axis you care about. A retrain that regresses quietly is worse than no retrain, because you have now spent the budget and lost the trust.
None of this works without one piece of bookkeeping: every model must be able to name the exact dataset that trained it. Snapshot the training set at every run - promote-or-revert is meaningless if reverting cannot reproduce the dataset the old model came from, and a spec change that re-cuts labels is only auditable if the before and after both still exist.