Twenty questions to put in a computer vision RFP for a plant or a utility

The utility's RFP for substation monitoring runs to forty pages, and the vision section is one of them. It asks for detection accuracy, a list of supported cameras, and a delivery date. Three vendors reply with an accuracy figure above ninety, a list of cameras that includes the ones in the yard, and a date. The scoring sheet cannot tell them apart, because the section asked nothing that separates a demo from an operation.

An RFP for a vision system is usually written by someone who has never labeled a frame and read by someone who has never stood in the substation. The twenty questions below are the ones that would have separated the three replies. They are grouped by what they are really asking about.

Four questions about what happens to the frames the model doubts

A model on the transformer camera will be unsure of some frames: a bird on the bushing at dusk, frost on the insulator in January, a view it has weak evidence for. What the vendor does with those frames is the difference between a system that improves and one that decays.

• Which frames does the model return for a person to look at, and how does it decide?
• Who looks at them, our people or yours, and where does the work happen?
• What happens to a correction once a person has made it? Ask for the path from the reviewer's verdict to the next training run, in steps.
• What signal tells us the model is drifting, and where do we see it? The honest answer is the correction rate: frames coming back, corrections landing, the override rate rising. A vendor who answers with a dashboard of confidence charts has described a monitor that cannot see a spec change.

Four questions about how a new version reaches the cameras

The frames were corrected. Now the model has to change, and the transformer camera has to keep running while it does.

• What triggers a retrain, and can we set the threshold ourselves?
• Does the new version train on the original set plus the corrections, or on the corrections alone? The second answer means a model that forgets last February every time it learns this one.
• Is a new version compared with the current one per class before it replaces it, and on which frames?
• What is the downtime on rollout, per camera? The right answer is none, and a vendor whose answer involves a maintenance window has described a deployment, not an operation.

Each version should also be able to name the frames it trained on. Ask whether the previous version can be brought back, and what it takes.

Four questions about where the footage goes and who can see it

The utility's security team will ask these before anyone else does, usually by the first Monday of the evaluation, so put them in the tender rather than discovering them at contract stage.

• Can the model run on our servers, or on hardware beside the recorder, so that footage stays inside the fence? If it can, what leaves the site? The answer we give is the doubted frames and the alerts, and nothing else. The security page has the shape of that boundary.
• If we run it ourselves, who manages it, and what does the vendor need access to?
• Is sign-on through our identity provider, and is access role based, so that a substation technician sees the substation and not the fleet?
• Which security attestations does the vendor hold today, with the documents attached rather than promised?

That last question should be answered with paper or with a plain no. Either is scoreable. A roadmap is not.

Four questions about how an alert reaches the person on shift

A detection nobody sees is a row in a table. The RFP should ask how a detection becomes an action on a Sunday night when the control room is one person.

• How is an alert defined, and by whom? Ask for an example. A rule written as a sentence, with a severity and a cooldown, approved before it goes live, is what we would expect to see.
• Where can it be delivered? Slack, email and a webhook into the utility's own systems cover most control rooms; anything that stops at a vendor portal will not be read.
• Does the alert carry the frame with the detection drawn on it, so the person paged can judge from a phone?
• With hardware on site, does the alert fire locally if the link to the cloud is down?

On the energy sites we run, the alert path pointed at grid assets is what produces the 21,000+ hazard detections a month behind our energy work, and every one of those arrived somewhere a person was already looking.

Four questions about the second substation and the end of the contract

The first site is a project. The tender should be scored on the fifth.

• What does the second site cost, in labeled frames and in money, once the first is live? A window of frames from the new cameras folded in before go-live is the honest answer; a fresh statement of work is a vendor who sells projects.
• Can we export the dataset and the model at any time? Ask for the formats. The dataset as COCO and the model as PT, ONNX or TorchScript is what we hand over, described by the device it will run on rather than the architecture.
• What runs, and what do we keep, if we stop paying?
• Who labels frames in year three, and does the commercial model reward our own people doing it?

That last one is the question that exposes a vendor's incentives, and it belongs near the top of the scoring sheet rather than the bottom.

The accuracy question is the one to strike

My view, which will not make me popular with procurement, is that the accuracy question should be struck from the vision section entirely. Any figure a vendor supplies was measured on frames the utility has not seen, under conditions the yard does not have, and it cannot be checked until the model has run on the transformer camera for a season. The twenty questions above can all be answered before a contract is signed and scored against paper.

If a figure has to be in the tender, ask for it per defect class, on frames from the utility's own cameras, after a pilot window, with the correction rate over that window beside it. That number is worth having. The one on page thirty-one is not.

LexData takes the substation model through its whole life. You type what to look for, Lexi puts a box on every frame, and a person checks each label before anything trains on it. The model then watches the cameras the yard already has, in the cloud, on your servers, or on a runner beside the recorder. Frames it is unsure of come back to a person, the corrections retrain it, and the new version replaces the old one with no downtime. The twenty questions are that sentence taken apart, and the platform page and the deployment doc are where our own answers to them are written down.