You are paying for an AI tool, or thinking about it. Is it worth the money? It is a simple question that a surprising number of businesses cannot answer, because they are measuring the wrong things: how many people logged in, how many drafts it produced, how fast it feels.
This guide gives you three measures that work for a small business: a baseline, the full cost of each finished task, and how often people overrule the AI. None of them needs special software. If you have not signed yet and want to agree measures with a consultant first, see how will your AI consultant measure success?
Key takeaways
- Write down how the task works today before you change anything. That is your baseline.
- Count the full cost of each task done properly, not the price of the tool.
- Include the time people spend checking and fixing AI output.
- Track how often people edit, reject or ignore what the AI produces.
- Measure against things you already care about: replies sent, orders processed, complaints.
Why usage figures do not tell you much
It is tempting to judge an AI tool by how much it is used. But usage is activity, not value. A tool can be opened every day and still save nothing, if people are reading its output and then doing the job themselves anyway. Equally, ten licences with three regular users is money going out of the door.
What you want to know is whether the work that matters is getting done better, faster or cheaper than before, once everything is counted.
Start with a baseline
You cannot show improvement without a starting point. Before you switch on an AI tool, spend a week or two noting how the task works now:
- How long it takes, roughly, per item (per invoice, per enquiry, per quote).
- How often it goes wrong or needs redoing.
- What it costs in staff time.
- Anything customers notice: waiting times, complaints, follow-up calls.
Rough is fine. A tally on paper is better than nothing. Without it, any later discussion about whether AI helped turns into opinion.
It is also worth asking how you would solve the problem without AI, and roughly what that would cost. If a cheaper, plainer fix gets you most of the way, the AI option has to beat that, not just beat doing nothing.
Measure the full cost of each finished task
This is the most useful single number. Pick the unit your business actually delivers for this task, such as an enquiry answered, an invoice processed or a quote sent. Then work out what each one costs, all in, and count only the ones that were done properly.
| Include | Why it is easy to miss |
|---|---|
| Licences and subscriptions | The obvious one |
| Usage charges | Some tools charge per use; bills can grow with volume |
| Set-up and integration | Often a one-off cost that is forgotten later |
| Time spent checking AI output | Rarely appears on any supplier's slide |
| Time spent fixing mistakes and redoing work | Including work customers send back |
| Training and support | Ongoing, not just at launch |
| Running the old process alongside the new one | Common, and doubles the cost |
"Done properly" matters. An enquiry answered wrongly, which comes back as a complaint, is not a finished task. Neither is a quote someone had to rewrite from scratch.
Compare this cost per finished task with your baseline. If it is lower, and quality has held, the tool is paying off. If it is the same or higher, find out why before you renew.
Watch how often people overrule it
How often do people accept what the AI produces as it is? How often do they edit it, reject it, or ignore it? This tells you a lot.
- If people overrule it often, it is adding a step rather than removing one. The same person is making the same decision, with extra reading first.
- If people almost never overrule it, check that they are still really looking. People under time pressure can start approving whatever looks plausible. A very low override rate can mean the tool is excellent, or that nobody is checking properly.
- If output is accurate but nobody acts on it, it is delivering nothing.
You do not need a dashboard for this. For a few weeks, ask the people using the tool to tally "used as is", "edited" and "rejected" for each item. Pair it with a spot-check of a handful of items each week.
Measure what customers and staff notice
Internal figures can all hit target while the people on both ends are unhappy. A tool can answer every enquiry quickly and still leave customers ringing back. So measure outcomes your customers would recognise, not only your own activity figures, and ask customers and staff directly. Their answer is part of the result.
A simple way to compare before and after
If you want a fairer test, try alternating. Use the AI for the task one week and the old way the next, with the same people, and compare time, errors and rework. Alternating cancels out busy and quiet weeks, and stops you mistaking the extra attention any new tool gets for a lasting improvement.
Your monthly check
- Is the cost per finished task lower than the baseline?
- Has quality held up: errors, rework, complaints?
- How much time is going on checking and fixing AI output?
- Are people accepting, editing or rejecting most of what it produces?
- Are we still running the old process as well? If so, why?
- Is everyone we pay a licence for actually using it?
- Are usage charges growing faster than the value?
- Would anyone object if we switched it off tomorrow?
That last question is a good gut check. If people would fight to keep a tool, it is probably earning its place. If nobody would notice it had gone, it probably is not.
What if you did not measure at the start?
Plenty of owners do not, so do not worry. Start now. Note how the task runs with the tool today, then try a fortnight without it, or with it on half the work, and compare. You will have a rough answer within a month, which is far better than none. For the next tool, build measurement in from day one: our guide to why AI pilots stall shows how to set a baseline and a stop point before you begin.