I've sat in a lot of rooms where the AI update is a list of pilots. A copilot for merchants. A chatbot that answers store policy questions. A forecasting model running in one category. Everyone is fairly proud of the list, and it's usually longer than it was last quarter.
Then someone asks what those pilots have added to margin, and nobody has a good answer.
In my experience the technology usually does what it was meant to do. The problem is that we're counting the wrong thing.
Why pilots multiply
Part of this is just human. Executives like show and tell. The teams reporting to them feel the pressure to show progress, and a pilot is the easiest thing to show. It fits into a steering committee slot, it demos well, and it gives leadership something to tell shareholders about how the company is using AI.
None of that is a problem in itself. What matters is what the pilot is actually set up to prove.
What a pilot should prove
A good pilot or proof of concept should prove at least one of three things, and ideally more than one. It can prove the technology, meaning the model or the agent can actually do the task. It can prove capability, which is about whether the organisation can absorb it: will the team adopt it, is the data clean enough to trust, and does our AI policy hold up once a real agent starts touching real decisions? And it can prove value, meaning something moves in the P&L when it works.
Almost every pilot I've seen was set up to prove the technology. The other two rarely get explored at all. That's how a pilot can succeed completely and still tell you almost nothing about whether it's worth scaling.
Where the value actually sits
Value shows up one level higher than the pilot, in the workflow and the decision at the centre of it.
Take promotions in retail. A model that predicts promotional lift is a pilot. The workflow is everything around it: how the calendar gets built, how supplier funding is negotiated, who approves the mechanic, what happens in store, and whether anyone looks at the result afterwards. In a lot of retailers that last step barely happens. Promotions run again because they ran last year.
If you put a better forecast into that workflow, very little changes. If you change the workflow itself, so that every promotion is reviewed within a week of it ending, funding decisions start to shift, and that's where the margin improvement comes from.
A compliance workbench at a medtech client
The best example I have of getting this right wasn't in retail at all. It was with a global medtech client.
We agreed early on that we weren't building a pilot. We were building a compliance workbench, something the compliance team would use every day. It was an agentic solution that surfaced anomalies in compliance data, so the team spent less time hunting for where the problems might be and more time investigating the cases that mattered.
Most of the effort didn't go into the technology. It went into taking the compliance team through the journey and meeting them where they were: how they actually worked, what they trusted, and what they'd need to see before relying on it. IT and the business built it together from the start.
What helped most was the chief compliance officer, who didn't care much about the technology and cared a great deal about the outcomes, so the outcomes were what we measured.
Once the outcomes were there, the CCO took the workbench to the board. By then it wasn't a pilot any more. It was in production, the team used it every day, and the business could see the results.
The questions I'd ask instead
If you're an executive looking at a list of pilots, these are the questions I'd put to the team.
Where will this fail in production? I mean with real data, real users and real exceptions, not in the demo.
Have we built it with a production mindset, or are we testing it because it's exciting to try something new?
Have we thought about it end to end? That means the data, the integrations, the policy, the people who'll use it, and whoever owns it once it's live.
And then metrics, metrics, metrics. What's the expected value? What's the baseline today? How will we know it has worked? A demo that works doesn't answer any of those.
In practice, this means swapping the long list of pilots for a much shorter list of workflows, each with a baseline, a target and one executive who owns the number. Anything that can't be tied to a workflow someone owns is really research. That can be worth funding, but I'd call it research and budget for it that way. And expect a few of those workflows, once you size them honestly, to come back smaller or slower than the pilot made them look. Stopping those early frees up money for the ones that will pay.
The compliance workbench made it to the board because the compliance team's numbers moved, and in that meeting nobody asked how many pilots we had running.
Written by Akshay Bardia, with AI help for research, drafting and editing. The views and experience are his.
