A pilot of a planning or prediction tool ends with a slide. The slide says the model was 87 percent accurate, or cut forecast error by 23 percent, or predicted 9 of the 11 late shipments. Everyone agrees the model works. Nothing changes, because nothing in the pilot was designed to change. The pilot was scored on the model when it should have been scored on the planner.
The only metric that pays is decisions changed: how many times did a person do something different from what they would have done without the tool, and what did the different choice produce.
Accuracy without a decision is a research result
Take an illustrative ETA model. Before the pilot, 68 percent of arrivals are predicted within 30 minutes. After, 84 percent. Now ask what the dispatch desk does with an ETA. The rule on the desk is: call the driver when the shipment is 2 hours late. The rule does not consult the ETA. The model can be 99 percent accurate and the number of decisions it changes is 0, because the decision was never wired to the prediction.
Build the decision inventory first
Before the pilot starts, list the decisions. Not the processes, the decisions: assign this load to that vehicle, accept or decline this booking, hold this shipment or release it, reprice this lane, escalate this delay or wait. For each one record who makes it, how many times a week, with what information in front of them, and what the current rule or habit is.
A typical desk turns out to make a few hundred such decisions a week, and the pilot is relevant to perhaps 3 of the types. That is fine. It makes the target explicit: of those 3 decision types, how many instances did the tool touch, how many did it change, and what happened on the changed ones versus the unchanged ones.
Four numbers, in order
Decisions touched: the tool produced an output while the decision was live and the person saw it.
Decisions changed: the person did something different from the counterfactual. This requires recording the counterfactual, which is the uncomfortable part. The simplest method is a 1 line entry before the output is shown: what would you do.
Outcome delta on changed decisions: cost, service, or margin on the changed set against the unchanged set and against the counterfactual. This is the number the business case is built on.
Override rate: how often the person saw the output and did what they would have done anyway. A high override rate with high accuracy is the most informative result a pilot can produce, because it says the problem is trust, timing, or the interface, not the mathematics.
What this changes about the pilot
Scoring on decisions changed alters the pilot's design before it alters its result. The tool has to be in front of the person at the moment of decision, not in a report the next morning. The output has to be in the person's units: a recommended action, not a probability. The pilot has to run long enough for a few hundred relevant decisions to occur, which on a weekly planning desk may be a quarter, not 3 weeks.
It also protects against a quiet failure. A model tuned for accuracy learns to be confident on the easy cases, the ones the planner already gets right. Decisions changed are concentrated in the hard cases, where accuracy is lowest and the value is highest. Optimizing accuracy can push the model away from the only cases that matter.
A pilot that changed 40 decisions out of 400 and improved the changed set by 6 percent has a business case. A pilot that was 92 percent accurate and changed 3 decisions has a publication. Only one deserves a budget.