How to Measure the Value of AI Work Without Skipping the Middle
How to Measure the Value of AI Work Without Skipping the Middle
Say your product lead has to decide whether the next release includes single sign-on. A few larger customers have asked for it, and some of their deals are stuck in security review. Before the planning meeting, the team uses AI to prepare a brief: what competitors offer, what those customers have actually said and what building it would push back.
A month later someone asks whether the AI work was worth it. You know what it cost. You can point to the release. What's hard to show is everything in between: whether the brief was any good, whether anyone used it and whether it changed what the team decided.
That middle is where AI-value claims usually fall apart. The fastest way to invent AI ROI is to record the cost, point at a business result and call the distance between them value.
How should you measure AI value?
Keep four linked records for each piece of work that matters:
- What the work consumed: money, people's time and the process you're comparing it with.
- What it produced: the brief itself, and whether it passed review.
- What someone did with it: the decision or action it informed.
- What happened afterward: the result you said you'd check, over the time you said you'd check it.
We think of these as a proof chain, because each record only means something linked to the others. Set your baseline before the work starts, and claim ROI or cause only when the records support it.
Knowing what the single sign-on brief cost tells you how much went into it, and that's all. The accepted brief tells you the work got done, and the decision record tells you someone used it. Only the last record shows whether anything changed, and even that doesn't show the brief was the reason.
Why usage numbers don't answer the question
Prompts, tokens, active users, pages generated and tasks completed are useful numbers. They tell you about adoption and cost. They can't tell you whether the brief was trusted, whether it changed the release decision or whether that decision turned out well.
AI makes this easy to blur because output is plentiful and easy to see. A team could produce ten competitor briefs in the time it used to take to write one and still not know whether any of them was accurate, used or worth acting on.
In evaluation terms, those ten briefs are outputs. The UK government's guidance on evaluating AI interventions, part of HM Treasury's Magenta Book, sets out a theory of change: outputs ("what is delivered or produced") are expected to lead to outcomes and, later, impacts. Each step comes with the assumptions behind it and how strong the evidence is. For the release brief, the output is the document. The outcome is what the team did differently because of it.
The usage numbers still belong in the chain. Prompts and tokens go with what the work consumed, and pages generated and tasks completed go with what it produced, next to how many passed review. What the product lead decided is a separate record.
The four records, through one brief
1. What the work consumed
Start by defining the unit. "AI research" is too broad to measure. "A cited brief on single sign-on, prepared for the release-planning decision" has a purpose, an owner and a point where someone reviews it.
Then record what that brief took: the AI cost, the time someone spent setting up the request, the reviewer's time and how long it took to go from request to finished brief. Pull the same numbers for the last few briefs the team wrote by hand. That's your baseline, and it's much harder to reconstruct after the fact.
If the product lead rewrote half the brief, that time goes here. It's part of what the work cost.
2. What the work produced
Save the brief the product lead accepted, along with its sources and the result of any fact check. The question is whether it did the job you gave it. Are the competitor comparisons sourced? Do the customer quotes come from the customers stuck in security review?
Keep the drafts the product lead sent back, too. If only the accepted version counts, the record makes the work look better than it was.
3. What someone did with it
Write down what the product lead decided, when, and which version of the brief was in the room. Maybe the team put single sign-on into the next release and moved a reporting change to the one after.
This is the record that usually goes missing. A brilliant brief nobody used may have taught the team something, but it hasn't done the job you're trying to measure.
4. What happened afterward
Before the release ships, choose what you'll check and when. For this decision: did the plan hold, did it need rework, and did the deals stuck in security review close within a quarter of the release?
Write down the side effects as well. The brief might have sped up the decision while adding review time, or moved work from the product lead to an engineer who had to check the technical claims.
Perhaps the brief met the bar and the team decided faster. Perhaps it was accepted, but the planning meeting slipped for an unrelated reason, so it changed nothing. Or the team decided faster and then had to rework the plan. All three are worth knowing. Only one makes a good slide.
Contributed is not the same as caused
Now suppose that in the same quarter, sales started offering a discount on annual plans and the company changed who handles enterprise accounts. The stuck deals close. The decision record shows the brief shaped what went into the release, and those deals were waiting on single sign-on, so the team can fairly say the AI work contributed. It can't say the brief caused those deals to close, because the discount and the new account owners could explain the same result.
To separate them, you need a comparison. If you have similar deals that were never waiting on single sign-on, put the stuck deals next to them over the same quarter. That's the stronger route. The UK guidance lists the groundwork: a baseline set before rollout, a precise description of business as usual and a clear account of the comparison group.
Often there's nothing similar to compare against. The guidance describes three main kinds of method (experimental, quasi-experimental and theory-based), and without a comparison the last is the realistic route. Theory-based methods such as contribution analysis ask how far the work contributed to the outcome rather than putting a precise number on it.
Whichever route you take, the answer is about this brief. The guidance warns that impact "may vary substantially between different tasks, different contexts and for different groups," so the single sign-on result may not tell you much about the next brief, on pricing.
All of that is a higher bar than "the number went up after we started using AI." It should be.
Match your words to what the records show:
| Word | What the records have to show |
|---|---|
| Produced | The brief exists and passed its review. |
| Enabled | A decision record shows the brief informed the decision. |
| Contributed | Several sources support a plausible role in the result. |
| Caused | A comparison supports a causal estimate against a credible alternative. |
"Contributed" is the word most teams can honestly use, and when the records back it, it's a strong claim.
When you can put a dollar figure on it
Calculate ROI only when both sides of the ratio hold up. For the single sign-on brief, the four records already give you most of the cost side: the AI cost, the setup, the review and the product lead's rewrite. The benefit side is harder. It needs a value, a time window, the human work added or removed and a comparison. A brief that arrived three days sooner has no automatic dollar value. Its value depends on what the product lead did with those days, and whether that time went to other useful work.
That's why published productivity studies show what's possible rather than handing you a business case. The NBER working paper Generative AI at Work studied 5,179 customer support agents as an AI assistant was rolled out to them in stages. Measured as issues resolved per hour, productivity rose 14% on average, including a 34% improvement for novice and low-skilled workers, with minimal impact on experienced and highly skilled ones. That's a careful study of one deployment with one defined measure, and even there the effect depended on who was using the tool. It isn't a multiplier for your release brief.
Keeping the record without drowning in it
You don't need records for every AI request. Keep them for each kind of work that feeds a decision someone will be held to, give each record an owner and look at where the chain breaks. If the decision or the spending is large, the risk is high or someone wants to say the AI caused the result, a formal evaluation of the kind the UK guidance describes is worth the effort.
For a release brief, a small ledger is enough:
| Record | Question | Minimum evidence |
|---|---|---|
| Consumed | What did this work take? | Defined unit, cost, people's time, baseline |
| Produced | What acceptable work exists? | Saved brief, review result |
| Used | What changed because someone used it? | Owner, decision, date, brief version |
| Afterward | What happened next? | Chosen measure, time window, baseline, side effects |
Where it breaks tells you what to fix. If briefs keep failing review, look at the request, the sources the work can reach, the model or the checks. If accepted briefs never show up in a decision, the problem is probably fit or trust, not the AI. If decisions happen but the expected results don't follow, go back to why you thought that decision would produce that result.
NIST's AI Risk Management Framework says measurement is there to provide "a traceable basis to inform management decisions." A ledger with the gaps left out can't do that. Say nobody wrote down which version of the single sign-on brief was in the planning meeting. Record the gap, and next time the product lead notes the version before the meeting starts.
Where Supanova fits
Say the single sign-on brief ran as a Supanova project. The product lead writes the project brief, describing the decision coming up and which customers asked for single sign-on, and the atoms do the work, from the competitor research to the written brief. Finished work comes back for the product lead's review, and that review is the second record: the product lead reads the brief and decides whether it's good enough to take into the planning meeting.
The decision happens in the planning meeting and the result after the release, so the team adds those two records to its own ledger. They're the ones that show whether the brief was worth what it cost.
Before the next AI-assisted brief goes into a meeting, write down which decision it's for and what you'll check afterward. Then make the strongest claim the records allow, and no stronger.
Sources
- HM Treasury and Evaluation Task Force, Guidance on the Impact Evaluation of AI Interventions, Magenta Book collection.
- Erik Brynjolfsson, Danielle Li and Lindsey R. Raymond, Generative AI at Work, NBER Working Paper 31161.
- NIST, AI Risk Management Framework Core.