Measure Impact, Not Activity for a Successful Data and AI Program

Across organisations, there is no shortage of activity in data and AI. Use cases are identified, pilots are developed, and progress is regularly reported. The programs that go on to scale are not the ones doing more of this. They are the ones that, at some point, stop asking what has been delivered and start asking what has actually changed.

That question, what has changed, is a harder one to answer than it sounds, and most programs never get around to building the visibility that would let them answer it with any confidence.

Why activity gets measured and impact does not

Activity is easy to report. Use cases in the pipeline, models developed, data processed. These numbers move in a clear direction and give the impression of momentum, which is exactly why so many programs default to them.

Impact is harder to pin down. A pilot can be technically successful without changing how the business operates. A model can be deployed without being used. A use case can generate a genuinely good insight without ever influencing a single decision. The programs that scale are the ones willing to sit with that discomfort early, building the discipline to track adoption and value even when the answer is not flattering, rather than retreating to the activity metrics that always look good.

Activity Reporting Versus Impact Visibility: What We See in Practice

The pattern shows up clearly in our diagnostic work, and it tends to split organisations into two groups.

In one engagement, a well-funded data and AI program had built a substantial pipeline of use cases, with regular updates going to senior leadership highlighting progress across multiple initiatives. When we asked how many had scaled into business-as-usual operations, the answer was far less confident than the pipeline reporting had suggested. The organisation had built a strong activity story without ever building the mechanism to check whether it was true.

In a second, the organisation reported heavily on delivery milestones: models built, dashboards released, pipelines established. What it lacked was any clear view of how those outputs were actually being used, or whether they were shaping decisions in any meaningful way. Adoption had been assumed rather than measured.

A third organisation took a different path. It held regular forums where teams presented their use cases and outcomes, with leadership present but deliberately constructive rather than critical. Successes were recognised. So were failures, some of which were named plainly as mistakes where the original assumptions had been wrong from the start. Others were treated as genuine learning, with the insight carried into the next round of work. In one case, an initiative that had struggled in one part of the business was picked up elsewhere, adapted, and went on to deliver real value.

The difference was not that this organisation got everything right. It was that it had built a structure and culture for finding out what was working, which is precisely what let it keep getting better.

Impact has to be designed for

The organisations that scale treat impact as something to be designed, in the same way the delivery itself was designed.

That starts with clarity at the outset: what value is expected from each initiative, how it will be measured, who is accountable for it, and how it will be reported. It continues with real discipline in tracking adoption, not whether a model has been deployed, but whether it is being used, by real people, in real workflows, on a normal Tuesday rather than during a leadership demo. It requires a genuine appetite for learning from outcomes in both directions, treating a result that did not work as useful information rather than something to quietly shelve.

None of this happens by accident. It happens because someone decided, early, that activity alone would not be the measure of success.

How Q22 approaches this: the Execution and Impact Review

Understanding the difference between activity and impact is the easy part. The harder work is building a structured view of whether data and AI are actually being used, delivering value, and improving over time. Q22's Execution and Impact Review is built to do exactly that, testing the programme across three connected dimensions.

Adoption in practice: where data and AI outputs are genuinely being used within decisions and workflows, day to day, not in theory. Which teams are using them, how consistently, and in what contexts.

Value realisation: how that usage translates into outcomes that matter, quantitative measures like revenue, cost, and risk, alongside qualitative signals such as improved decision-making and greater organisational confidence.

Learning and adjustment: how the organisation captures insight from both its successes and its failures, and whether that insight actually shapes what comes next.

Together, these three dimensions give a clear picture of whether a program is on track to scale, or simply staying busy.

Where to start

The starting point is not more use cases or more investment.

It is clarity: how is value being defined for each initiative, who is accountable for delivering it, how is adoption actually being tracked, and what mechanism exists to capture and act on what the organisation is learning.

Answering those questions early gives a Data & AI leader something more useful than a status update. It gives them a credible narrative, one that connects activity to outcomes, holds quantitative return alongside qualitative progress, and shows how the organisation is genuinely getting better over time.

That narrative is what separates programmes that stay busy from programs that scale.