How to tell if an AI project worked

An AI project worked if a task got faster, cheaper or better without getting worse somewhere else, and people are still using the new way three months later. You can only show that if you measured the task before you started. None of it needs a data team: a spreadsheet, a stopwatch and someone who knows the work will do.

Start with a baseline

Before anything is switched on, measure the task as it runs today. Pick one task and define where it starts and stops, for example from “enquiry arrives” to “reply sent”. Then, over two to four normal weeks, record:

• how many you handle each week;
• how long each one takes, start to finish;
• how often something goes wrong (a correction, a complaint, a redo);
• what it costs, which is mostly staff time.

Use real examples, including the awkward ones. Then, before the pilot starts, write down what would count as success, for example “at least a fifth faster, with no more errors”. Agreeing that in advance stops anyone moving the goalposts later. If the records you need are scattered across files, look at your spreadsheets first.

Time per task: use a clock, not a feeling

Asking people how much time they saved is tempting, and unreliable. In the UK government’s Microsoft 365 Copilot experiment, users were asked to estimate their average daily time saving, and the average answer was 26 minutes. The report itself says the savings are self-reported and should be read alongside other evidence.

A 2025 study by the research group METR shows why that caveat matters. Sixteen experienced software developers expected AI tools to speed them up by 24%. Measured, they took 19% longer. Afterwards, they still believed they had been 20% faster. The authors are clear that it covered one kind of work and doesn’t show AI is unhelpful in general, but the lesson for measuring holds: what people feel and what the clock says can point in opposite directions.

So time a sample of tasks before and during the pilot, the same way each time, and include the time spent checking and correcting the AI’s work. It’s the easiest part to forget.

Error rates: a person checks a sample

A faster process that makes more mistakes hasn’t worked. Each week, ask someone who knows the work to check a random sample of the AI-assisted output: say twenty items, or one in ten, whichever is manageable. Random matters. If people choose which examples to show you, you’ll see the good ones.

Use a simple scale:

• fine as it is;
• minor edit;
• major fix;
• wrong.

Run the same check on work from before the pilot, so you’re comparing like with like. Note what kind of error it was, too (missing information, wrong tone, a made-up detail), because the pattern tells you what to fix. Why AI gets things wrong covers the usual causes.

Cost per task

Add up everything the new way costs in a month and divide by the number of tasks done. Include:

• licences or usage charges;
• staff time spent checking and correcting;
• set-up and training, spread over the months you expect to use it;
• the time someone spends looking after it.

Compare that with the cost per task before, which is mostly staff time. A tool can save time and still cost more per task than it’s worth, especially at low volumes.

Adoption: are people using it?

A tool nobody uses saves nothing. Each week, look at how many of the people who could use it actually did, and what share of eligible tasks went through it. Watch whether that holds up after the first month, once the novelty wears off.

Ask the team two quick questions each week: did you use it, and if not, why not? The reason may well be fixable: it’s slow, it doesn’t know about a common case, or it lives in yet another window. Some fixes are technical. Others are training, or a short, clear AI policy so people know what’s allowed.

Ask how people feel about it as well, but keep that separate from the time figures. In the government experiment, 82% said they would not want to go back to the way they worked before Copilot. That’s a good sign for adoption. It isn’t a measurement of time saved.

A simple measurement table

What to measure	How to measure it	How often	Watch out for
Volume	Count tasks from your system, or keep a tally	Weekly	Busy and quiet weeks skewing the picture
Time per task	Time 20 or so real tasks, start to finish, including checking	Before the pilot, then every two weeks	Counting only the AI’s part
Error rate	A person checks a random sample against a simple scale	Weekly	Checking only the examples people choose to show you
Cost per task	Monthly costs divided by tasks done	Monthly	Leaving out checking time and set-up
Adoption	Usage figures, plus the two questions above	Weekly	Early enthusiasm that fades after a month
Side effects	Complaints, rework and escalations	Monthly	Problems that quietly move to another team

When the results are mixed

Expect them to be. An average can hide a tool that’s excellent on routine cases and poor on unusual ones, so break the results down by type of case. Then choose:

• Narrow it. Keep AI for the cases where it works and send the rest to a person.
• Fix the inputs. Clearer instructions, a template or access to the right documents can sort out the commonest errors.
• Run it longer if the numbers are too small to tell, using the same measures.
• Stop. A clear “no” after four weeks costs less than a vague “maybe” that runs for a year. Stopping is a result too.

Whatever you decide, write it up on one page: what you tested, the numbers and the decision. It makes the next project much easier to judge. If you’d like help setting up the measures before a pilot starts, see how we work: the baseline is agreed before anything is built.