You have sat through an AI demo and it looked brilliant. The question now is whether it will do the same job on your work, with your files, on a busy Tuesday. A demo cannot tell you that on its own. This guide shows you how to turn a demo into a fair test, using your own data and a few pointed questions, before you commit any money.
Key takeaways
- A demo shows what a tool can do on the best inputs. It does not show how it copes with yours.
- Ask to see it run on a sample of your real, messy data, including the awkward cases.
- Ask how it behaves when it gets things wrong, not just when it gets them right.
- Speak to a customer who uses it day to day, not only to the person selling it.
Why can't I trust a good demo?
Because a demo is built to succeed. The person showing it has chosen the examples, cleaned the data and practised the run. That is not dishonest. It is what a demo is for. But it means you are watching the best case.
Your business is not the best case. Your customer emails have typos, your spreadsheets have gaps, and plenty of your jobs are "a bit different from usual". The distance between how a tool demos and how it runs on everyday data is where disappointment tends to start.
So treat a good demo as proof that something is possible. Then find out whether it is possible for you.
What should I ask to see instead?
Ask to see the tool working on your own material. A reasonable request is: "Before we go further, can you run this on a sample of our real data and show us the results, including the ones it got wrong?"
A confident provider will usually agree, and may ask for a small fee to cover the setup. That is fair. Pay attention to three things:
- How long setup takes. If getting it running on your sample takes weeks, that tells you something about how much work the real thing will need.
- What they report. You want to see the misses and the near misses, not a highlights reel.
- What they need from you. If the tool needs far more data than you have, or data in a shape you do not keep, you have learned that cheaply.
If a provider will not test on anything but their own examples, ask why. There may be a good reason, but you should hear it.
How do I prepare a test on my own data?
You do not need to be technical. You need a small, honest sample of the work you want help with, and a way to tell good results from bad ones.
- Pick one task, for example sorting incoming enquiries or pulling details from invoices.
- Gather a sample of real examples: the routine ones and the awkward ones.
- Include the exceptions that trip your staff up: unusual formats, missing details, odd requests.
- Write down what the right answer is for each example, ideally with the person who does the task now.
- Note how long the task takes today, so you have something to compare against.
- Before you share customer records with anyone outside the business, check your obligations with a data protection adviser. Removing names and contact details from the sample may be enough for a first test.
- Give the same sample to every provider you are comparing, so the results are fair.
The answer sheet matters most. Without it, you are judging results on how plausible they sound, and AI tools are very good at sounding plausible. With it, you can count how many they got right and look closely at the ones they missed.
What should I ask about when it goes wrong?
Every AI tool makes mistakes. The useful question is what happens next. Ask:
| Question | A reassuring answer sounds like |
|---|---|
| What happens when an input is incomplete or unusual? | "It flags it for a person rather than guessing", with an example. |
| How would we know it had made a mistake? | A clear description of the check, and who does it. |
| Can we undo something it has done? | Yes, with an explanation of how. |
| Tell me about a recent time it got something wrong. | A specific, slightly unglamorous story of an error, how it was caught and what changed. |
| Who checks the output before it reaches our customers? | A named role on your side or theirs, never "it doesn't need checking". |
Be wary of anyone who says the output needs no human review. Be equally wary of promises of perfect accuracy. No honest provider can guarantee that, and the ones who handle errors well will tell you so.
How do I check a reference properly?
Ask for a customer who has been using the tool in their day-to-day work, ideally a business with similar data and a similar size to yours. Then call them yourself.
Case studies written by the provider are a starting point, not evidence. A conversation with a real customer tells you far more. Useful questions for that call:
- What did it take to get it working in your business, and how long?
- Did the results hold up once the pilot ended and the novelty wore off?
- How much time do your staff spend checking or correcting what it produces?
- Have you kept paying for it, or expanded it? If not, why not?
- What do you wish you had known before you started?
A customer who renewed or expanded is a stronger signal than one who launched with a press release. And if the provider cannot find anyone willing to talk to you, take that into account.
Is it really AI, and does that matter?
Sometimes what is sold as AI is ordinary automation or a thin layer over a public chatbot. That is not automatically bad: a simple rule that works is better than clever AI that does not. The problem is paying a premium for a label.
Two questions cut through it. First: "What would change if you took the AI out?" Second: "Could we get most of this from tools we already pay for?" A good provider will answer both plainly, and will sometimes tell you that part of your problem does not need AI at all.
What if I have already bought on the strength of a demo?
It happens to careful people too, and it is fixable. Run the same test now. Take a sample of your real work, write down the right answers, and measure how the tool does against how long the task took before.
If the results are good, you have evidence to build on. If they are poor, you have something concrete to take back to the provider, which is far more useful than a vague sense of disappointment. Ask them what would close the gap, and agree a date to check again.
For more on what to ask before you sign, see questions to ask an AI consultant and red flags when hiring an AI consultant.