You tried an AI tool. The demo was impressive, a few people liked it, and three months later nobody quite uses it, nobody has decided to stop paying for it, and nobody can say whether it worked. If that sounds familiar, you are in good company. This guide is about running a pilot so it ends in a clear decision, and moving it into everyday work. Choosing the task in the first place is covered in what a good first AI project looks like.
The short answer: pilots rarely stall because the technology failed. They stall because nobody agreed what success looked like, the trial ran on tidy examples instead of real work, and the people who would have to use the tool every day were brought in too late.
Key takeaways
- Decide before you start what result would make you carry on, and what would make you stop.
- Measure how the task works today, so you have something to compare against.
- Test on your real, messy work, with the people who will actually use the tool.
- Give every pilot a named owner and an end date.
- Plan for training and new habits, not just the software.
Why do so many AI trials go nowhere?
Experienced advisers see the same handful of causes again and again.
There was no finish line: If you start a trial "to see how it goes", any result can be read as promising. Success criteria agreed after the event can always be talked round. A pilot with no agreed stopping point never fails, but it never succeeds either. It drifts.
It ran on a demo, not on your work: Suppliers show tools on clean, well-chosen examples. Your emails, invoices and customer records are messier. Pilots that shine on tidy data often fall over when they meet missing fields, odd formats and the exceptions your team handles without thinking.
Nobody owned it: When a trial belongs to "the team", nobody chases it, nobody decides, and the licence quietly renews.
The people side was skipped: A tool that works but that staff do not trust, or do not see the point of, will be worked around. People keep doing the job the old way and add the AI as an extra step, which is worse than either.
Too many things at once: Trying three tools in parallel makes it impossible to tell what worked. For a small business, one pilot at a time is usually right; two is the practical limit.
What does a pilot that reaches real work look like?
Think of a pilot as a short, honest experiment with a decision at the end. The table compares the two patterns.
| A pilot that stalls | A pilot that reaches real work | |
|---|---|---|
| Goal | "Try AI for customer emails" | "Cut the time to answer routine order queries without more complaints" |
| Starting point | Not measured | Today's time, cost and error rate written down |
| Data | Supplier's examples | A slice of your own recent work, warts and all |
| Users | Enthusiasts only | The people who do the task, including a sceptic or two |
| Length | Open-ended | A fixed few weeks, with a review date in the diary |
| Owner | "The team" | One named person |
| Ending | Fades out | A decision: carry on, change, or stop |
How do you set the finish line?
Before you switch anything on, write down three things on one page.
- The task. One specific, repetitive job, such as answering stock queries, sorting supplier invoices or drafting quote follow-ups. If you have not chosen one yet, see what a good first AI project looks like.
- How it works today. Roughly how long it takes, how often it goes wrong and what it costs in staff time. A week of rough notes is enough. Without this you will never know whether AI made a difference.
- What "worth keeping" means. The improvement that would justify the cost and the change, and the point at which you would stop. For example: "If replies are faster and no less accurate after four weeks, we keep it. If staff are spending as long checking the drafts as they used to spend writing them, we stop."
That last point matters more than it looks. The time people spend checking and correcting AI output is real cost. A pilot that only counts the time saved, and not the time spent reviewing, will look better than it is. Our guide to telling whether AI is paying off covers this in more detail.
Who should be in the pilot?
A small group is fine: a handful of people who do the task every day, plus a champion who is respected by colleagues and will keep it moving. Resist the urge to pick only the enthusiasts. Pilot groups are usually keener and more confident than the rest of the team, so a trial run only by fans will overstate how well the wider roll-out will go. Include at least one person who is unconvinced, and listen to them.
Tell the group why you are trying the tool, what you hope it will do, and what it will not change. People fill silence with worry, and a worried team quietly avoids new tools.
What happens after a successful pilot?
This is where many trials stall a second time. A good result in a few weeks, with the supplier or consultant on hand and everyone paying attention, is not the same as a result that holds up in an ordinary month.
Treat moving from pilot to everyday use as its own decision. Before you commit, ask:
- Did we beat the starting point we wrote down, after counting time spent checking?
- Does it still work on ordinary days, without the supplier helping?
- Do we know the full running cost: licences, usage charges, set-up, staff time?
- Who will own it day to day, and who fixes it when it goes wrong?
- What training will the rest of the team need, and when will they get it?
- Which old steps will we stop doing, so people are not running two processes?
- When will we check again that it is still paying its way?
If you cannot answer most of these, the pilot is not finished. That is fine. Extend it with a new end date, or stop and keep what you learned.
Plan the people side from the start
A common pattern is a pilot where the tool works but stalls just before going live, because nobody planned how the team would change the way it works. The fix is to treat training and new habits as part of the pilot, not something bolted on when trouble appears. Set aside time for the team to learn the tool on their own work, write down the new way of doing the task, and say clearly which old steps are going away. Our guide to bringing your team with you goes further.
What if your last pilot already stalled?
Do not write it off, and do not blame yourself. Go back and fill in the one-page brief above as if you were starting again: the task, today's baseline, and what success would have meant. Often the answer is obvious once it is written down. The tool may have been fine for a narrower job, or the real block may have been training or trust. Either way, you now know far more than you did, and your next pilot will be better for it.
If you want outside help to run a pilot, a good adviser will ask about your baseline and your stopping point before they talk about tools. Our guide to writing an AI project brief shows you how to set that out.