Key points
- The question at this stage is simple: did the change really make things better? Answering it means using the baseline, the number you measured before anything changed.
- Measure the baseline number again, the same way, over a similar stretch of time. If hours went on a tally sheet before, they go on a tally sheet again. Switching method partway through leaves nothing fair to compare.
- Next, count every cost, because saving time in one place can add work in another. Say an AI tool writes a reply in a minute, yet each needs five minutes of checking. The true saving is less than it seems.
- The costs to include are staff time reviewing and fixing the tool's work, what the AI models and tools cost to run, upkeep and support, and hours given to training and to problems.
- Then look for side effects. Ask the people who use the tool and the customers it affects whether quality has shifted, whether new errors have crept in, and whether some cases now fare worse. Figures on their own can mask trouble that people notice at once.
- If the gain falls short of hopes, admit it and find the cause. That cause may be the tool, the information, the process, or the way the tool is used.
- A small, genuine gain you understand beats a large one you cannot account for. And if the project failed, the baseline and the way the pilot was designed should reveal that fast and at little cost, which also counts as a good result.
- So: measure the same way, count every cost, ask the people affected, and be honest about the result. All of this rests on the baseline you set at the start.
Transcript
The question at this stage is simple: did the change really make things better? Answering it means using the baseline, the number you measured before anything changed. Measure the baseline number again, the same way, over a similar stretch of time.
If hours went on a tally sheet before, they go on a tally sheet again. Switching method partway through leaves nothing fair to compare. Next, count every cost, because saving time in one place can add work in another. Say an AI tool writes a reply in a minute, yet each needs five minutes of checking.
The true saving is less than it seems. The costs to include are staff time reviewing and fixing the tool's work, what the AI models and tools cost to run, upkeep and support, and hours given to training and to problems. Then look for side effects.
Ask the people who use the tool and the customers it affects whether quality has shifted, whether new errors have crept in, and whether some cases now fare worse. Figures on their own can mask trouble that people notice at once. If the gain falls short of hopes, admit it and find the cause.
That cause may be the tool, the information, the process, or the way the tool is used. A small, genuine gain you understand beats a large one you cannot account for. And if the project failed, the baseline and the way the pilot was designed should reveal that fast and at little cost, which also counts as a good result.
So: measure the same way, count every cost, ask the people affected, and be honest about the result. All of this rests on the baseline you set at the start.