In this lesson
A ranking, not a probability
A detection tagged 0.9 has not told you there is a ninety percent chance it is right. The score is what remains after training losses and post-processing have shaped it: a number that orders this detection against that one, on data that looks like the training set. As an ordering it is genuinely useful. As a probability it is a misreading.
The distinction earns its keep the first time someone sets a business rule on it. “Auto-file anything above 0.8” treats the score as a chance of being correct. What 0.8 means depends on the class, the camera, and the season - the same score can be near-certain for one class and coin-flip for another.
Modern models are overconfident
This is measured, not folklore. As networks got deeper and more accurate, their scores got less honest - a landmark 2017 study found modern architectures systematically overstate their certainty, where the shallower networks of decades earlier were roughly trustworthy. Accuracy went up; calibration went down.
It gets worse exactly when you need it most. Under distribution shift - the drifted conditions the previous lesson maps - models do not reliably become hesitant. They are often confidently wrong on inputs unlike anything they trained on. A score calibrated against your validation set is not calibrated against next winter.
This is why the drift lesson watches the confidence distribution for movement rather than trusting any single score. Shape change is signal; the number alone is not.
A threshold is an operating point
Every threshold you set chooses a position on a curve: catch more and flag more falsely, or flag less and miss more. Neither side is free, so the choice belongs to the acceptance sentence from chapter one - how many false alarms a shift can absorb, and what a miss costs.
Base rates do the surprising work here. A detector that is right 95 times in 100 sounds finished - point it at a defect that occurs once in a thousand frames and most of what it flags will still be false alarms. Nothing about the model got worse; the arithmetic of rare events is just unforgiving. Set thresholds against the real frequency of the thing you are hunting, not against the score distribution alone.
What the score is actually for
Spend it as an ordering. Sort the review queue by confidence and human attention lands where the model is least sure, which is where it earns the most. Watch per-class score distributions over time and you have the cheapest early drift signal there is.
What the score is never for: skipping verification on the classes that matter. A high score on a safety-critical class is a strong hint, not a verdict - the expensive failures in this field are the confident ones.