Before you sign with an AI consultant, you need to agree one thing above all: how you will both know whether it worked. Without that, the end of the project becomes a debate about impressions. The good news is that settling it takes a short conversation, not a spreadsheet. This guide gives you the questions to ask before you sign and explains what a good answer sounds like. If the tool is already running and you want to know whether it is earning its keep, see how to tell whether AI is paying off.
Key takeaways
- Agree what success means, in your own business terms, before any work starts.
- Measure how things are today, so there is something to compare against.
- Make sure any savings count the time your staff spend checking the AI's work.
- Set a review date and decide in advance what result means carry on, change or stop.
Why agree success before the work starts?
Because afterwards it is too late to be fair. Once money has been spent, both sides have a reason to see the result kindly. A consultant can point to things that improved. You may want to believe the spend was worth it. Neither is dishonest, but neither is evidence.
Agreeing the measure first protects both of you. You get a clear way to judge what you paid for. A good consultant gets a clear way to show they delivered. If a consultant is reluctant to commit to any measure at all, ask why.
What should success be measured in?
In your terms, not theirs. AI projects come with their own vocabulary: accuracy scores, usage numbers, the number of questions answered. These can be useful to the people building the tool. They do not tell you whether your business is better off.
| Measures that sound good | Measures that tell you something |
|---|---|
| How accurate the model is in testing | How many jobs it handled correctly on your real work |
| How many staff have logged in | How much time the task takes now, compared with before |
| How many questions the chatbot answered | How many customer queries were resolved without a follow-up |
| How much content it produced | How much of that content was used without heavy rewriting |
| How fast it responds | How quickly a customer or invoice gets dealt with, start to finish |
Ask your consultant to restate what they promise in the right-hand column. For example: "hours saved each week on processing supplier invoices" or "fewer quotes sent out with mistakes". A good consultant will welcome the question, because it gives them a target they can actually hit.
Keep it to one or two measures. Ten measures make it easy to find something that improved and hard to say whether the project succeeded.
What is a baseline, and who measures it?
A baseline is a record of how things are now, before anything changes: how long the task takes, how often it goes wrong and what putting it right costs. It is the "before" in your before-and-after comparison. How to record one is covered in how to tell whether AI is paying off.
What matters at the contract stage is who records it and when. Ideally your team does it, with the consultant agreeing the method, before the project starts. If the consultant measures both the "before" and the "after", that is not necessarily a problem, but you should see how they did it. Also ask what the result would be without AI: if a simpler fix would get you most of the way, the AI has to beat that, not just beat doing nothing.
Do the savings include checking time?
They should, and this is the question most worth asking. AI tools produce work that people then need to check, correct or finish. That checking takes time. If a proposal promises hours saved without counting it, the real saving will be smaller.
Ask: "Will the time savings you are forecasting be net of the time our staff spend reviewing the output?" Then, during the project, measure it. If people are spending as long checking as they used to spend doing, the tool has added a step rather than removed one.
How will I know if people actually use it?
A working tool that nobody uses delivers nothing. So alongside your main measure, ask the consultant how they will track whether the tool has become part of everyday work.
Simple signs are enough for a small business. How often do staff accept the tool's output as it is, edit it, or ignore it? Do people use it without being reminded? If you took it away, would anyone complain? Ask the consultant how they will help with training and adoption, not only with the build.
What should I ask before I sign?
Take these into your conversation with any consultant you are considering.
- What one or two measures will show this has worked, in our business terms?
- How will we record how things are today, and who will do it?
- What would the result be without AI, and how much better do you expect this to be?
- Do your forecast savings include the time our staff spend checking the output?
- How will we know whether people are actually using it?
- When will we review the results, and what will we decide then?
- What result would make you recommend we stop?
- Will the measures and review date be written into the agreement?
The second-to-last question is the most revealing. A consultant who can tell you what failure would look like has thought about your problem. One who cannot may not have thought it through yet.
Download: agreeing how success is measured (Word)
What happens at the review date?
You compare the result against the measure and the baseline, and make one of three decisions: carry on and expand, change course, or stop. Agree before you start which result leads to which decision. That way the review is a check, not a negotiation.
If the result is mixed, look at why. Poor results can trace back to messy data or a process that needed fixing first, rather than the AI itself. That is worth knowing, even if you stop.
And if you are already partway through a project with no agreed measures, it is not too late. Agree them now, record where things stand today, and set a review date. You will lose the true "before", but you will have a fair basis from here on.
For the wider set of questions to ask, see questions to ask an AI consultant, and for putting your measures in writing, see how to write an AI project brief.