Accuracy is not the metric your client cares about

Accuracy alone tells you nothing

A prospective client once asked us to guarantee a machine learning model would hit 95% accuracy before they would sign off on a project. It was a reasonable-sounding request and almost the wrong question entirely. Accuracy, on its own, tells you almost nothing about whether a model is worth deploying — it depends entirely on what the other 5% costs, and what the alternative to the model actually is.

Consider two models, both at 90% accuracy. One classifies incoming invoices by vendor so they route to the right approver — a wrong classification costs someone thirty seconds to manually reroute an email. The other flags transactions for fraud review — a wrong classification (in either direction) costs either a false accusation against a real customer or a missed fraud loss that can run into thousands of dollars. The same accuracy number represents a trivial inconvenience in one case and a business-critical failure mode in the other.

Translating the model into the client's terms

So before we write a line of model code, we translate the machine learning problem into the unit the client already thinks in: hours of manual work removed, dollars of error avoided, or risk exposure reduced, each against a clearly stated baseline. For a document processing engagement, that baseline is usually the fully manual process — how long does it currently take a person to do this, and how often do they get it wrong today, because human baseline error rates are rarely as good as clients assume. For a recommendation or scoring model, the baseline is often the existing heuristic or rule-based system the client is trying to replace. The model does not need to be perfect; it needs to beat the baseline by enough margin, on the metric the client actually feels.

This reframing changes what we optimize during model development, too. Once the client's real cost function is explicit, precision and recall trade-offs stop being abstract ML terminology and become business decisions: is a false positive on fraud review more expensive than a false negative, and by how much? That ratio, not a single accuracy number, is what should set the classification threshold. We build this cost matrix with the client during scoping, before training, so the eventual model is tuned for the failure mode they can least afford.

What we hand back

It also changes how we present results. Instead of a confusion matrix and an F1 score in a slide deck, we hand back a dollar-and-hours estimate: this model saves approximately this many analyst-hours per month at current volume, and reduces this category of error by this percentage relative to the manual baseline. That is the number that survives contact with a CFO, and it is the number that actually predicts whether the model gets adopted and kept running, versus quietly abandoned three months after launch.

None of this means accuracy does not matter — it is usually still the metric we are directly optimizing under the hood. It means accuracy is an engineering detail, not the deliverable. The deliverable is the business outcome the accuracy number is supposed to produce, stated in terms the client can put in front of their own leadership.

If you are scoping a machine learning project and are not yet sure what the right success metric is, that conversation is worth having before any model gets trained. Talk to us and we will help you find the number that actually matters for your case.