AIReturn

The CFO & the AI budget

From Adoption to Impact: Why AI Usage Dashboards Are Vanity Metrics

AI adoption metrics are vanity metrics. Seats, prompts, and hours-saved measure activity, not impact. Here's the impact metric to use instead.

Rodrigo Paredes BassiPublished 8 min read

TL;DR

  • AI adoption metrics are vanity metrics. Seats activated, prompts per week, and hours saved measure whether people touched AI — not whether the work got better or the spend paid off. Adoption is an input; impact is an outcome.
  • The gap is measurable and moving the wrong way. Adoption rose while value fell: 41% of workers received AI "workslop," costing ~1h 56m to fix each instance and an estimated ~$186 per employee per month (BetterUp Labs + Stanford, HBR, 2025).
  • The dashboards most teams rely on are vendor-locked. A Microsoft-style usage dashboard sees Copilot but is blind to ChatGPT, Claude, Cursor, and in-house agents — so it can't measure impact across the stack an employee actually uses.
  • The fix is one swap: for each vanity metric, report the impact metric instead — output-per-dollar-net-of-rework, per team, vendor-agnostic — against that team's own baseline.
  • 78% of finance executives still can't fully tie AI spend to outcomes (CloudZero, 2026). Usage dashboards are a large part of why: they answer "is AI being used?" and leave the fourth question — did the work actually improve? — unanswered.

Why are AI adoption dashboards vanity metrics?

AI adoption dashboards are vanity metrics because they measure activity, not impact. A rising seat count, a higher prompt volume, or a self-reported "3 hours saved" tells you people engaged with a tool. None of it tells you whether the output shipped, sold, or resolved anything — or whether a human had to redo it afterward. Activity looks like progress on a chart while the P&L stays flat. This is the workforce-analytics version of a pattern that repeats across every AI-measurement category: the dashboard reports the easy number and skips the hard one. For the finance-side counterpart, see how a CFO governs the AI budget per team.

Definitions: three terms that separate adoption from impact

Plant these once; the whole argument rests on them. AI adoption metrics are measures of engagement with AI tools — seats activated, monthly active users, prompts per week, features used, and self-reported time saved. They describe uptake. They are inputs to ROI, not ROI itself. Output-per-dollar-net-of-rework is the business output a team ships per dollar of AI spend, after subtracting the cost of redoing flawed AI work. It is the impact unit: cost in the denominator, redo cost netted out of the numerator, measured against the team's own pre-AI baseline. Vendor-agnostic measurement is measuring AI's impact across every tool employees actually use — Copilot, ChatGPT, Claude, Cursor, in-house agents — in one view, rather than a single-vendor dashboard that is blind to the rest of the stack.

The fourth question your dashboard doesn't answer

Most AI measurement answers three questions, each owned by a mature category. What did it cost? — FinOps. Does it work technically? — observability and evals. Is it compliant? — governance. Usage dashboards sit on top of these and add a fourth-looking metric — adoption — that feels like value but isn't. The question none of them answers continuously is the one a CFO actually funds against: did AI actually improve the work — by team, tied to business outcomes, net of rework — and what do I fix? Adoption is not that answer. Two teams can show identical seat counts and prompt volumes while one ships clean work and the other generates output that gets rewritten twice before it's usable. On a usage dashboard, they look the same.

Why adoption can rise while value falls

The uncomfortable finding of 2025–2026 is that adoption and value have decoupled. Usage went up; useful output did not follow. Workers reported receiving AI "workslop" — output that looks polished but lacks the substance to advance the task — at a rate of 41% in a single month. Each instance took an average of ~1h 56m to resolve, at an estimated ~$186 per employee per month in lost time (BetterUp Labs + Stanford Social Media Lab, HBR, 2025). (The ~$186 is self-reported time valued by respondents, not instrumented — treat it as directional scale, not a booked cost.) Read that against an adoption chart and the contradiction is stark: more people using AI, more output generated, and a growing share of that output arriving as rework for someone else to fix. Only 28% of workers in the same study said AI improved decision quality. Volume rose; quality did not. A dashboard that counts prompts will show this as success. Impact measurement — how rework quietly erases AI ROI — shows it as cost.

The single-vendor blind spot

There is a second, structural problem with the dominant dashboards: they are vendor-locked. Microsoft-style usage analytics — the Viva and Copilot family — report on Microsoft's own AI. Workforce-analytics tools such as Worklytics report on the surfaces they instrument. None of them see the full stack. An employee who runs Copilot for email, ChatGPT for research, Claude for drafting, and Cursor for code shows up as one line — Copilot usage — in a Microsoft dashboard, and the other three tools are invisible. That makes vendor-locked adoption metrics doubly misleading. They over-credit the one vendor they can see, and they under-count the AI spend the company is actually accountable for. You cannot measure impact across a stack you can only partially observe. Vendor-agnostic measurement is the fix: one view across every AI tool, so the impact number reflects the work, not one supplier's slice of it.

The swap: vanity metric → impact metric

The correction is not a better usage dashboard. It is a different metric on the same row. For each vanity metric, report the impact metric instead — measured per team, against that team's own baseline, across every vendor.

Vanity metric (what dashboards report)What it actually tells youImpact metric (report this instead)
Seats activated / licenses assignedHow many people can use AIOutput-per-dollar-net-of-rework, per team
Monthly active users / adoption rateHow many people did use AIChange in useful output vs. the team's own pre-AI baseline
Prompts per week / messages sentVolume of activityAccepted, non-reworked outputs per dollar of AI spend
Hours saved (self-reported)A survey estimate of freed timeOutcomes shipped, sold, or resolved that survived rework
Features used / "AI-assisted actions"Breadth of tool engagementAI rework rate: share of AI output that had to be redone
Single-vendor usage (e.g., one dashboard)One supplier's slice of the stackVendor-agnostic impact across every AI tool in use
Read the right-hand column top to bottom and it resolves to one line: output-per-dollar-net-of-rework, per team, vendor-agnostic, versus baseline. That is the metric a CFO can reconcile. Everything on the left is an input to it. For the board-facing version of this argument, see how to prove AI ROI to your CFO and board.

Why this matters now

The demand for impact over adoption is no longer optional. 78% of finance executives say they can't fully tie AI spend to business outcomes, and boards are increasingly gating further AI funding on proof of return (CloudZero, 2026). Independently, only 28% of AI use cases fully meet their ROI expectations (Gartner, 2026). Usage dashboards are a direct contributor to that gap: they gave organizations something to report that felt like return and wasn't, which is also part of why most AI pilots never reach the P&L. Adoption was the right question in 2024, when the job was to get people to try AI at all. In 2026 the question is whether the work improved. A dashboard that still leads with seats and prompts is answering last year's question — and quietly costing the CAIO credibility with the CFO who has to defend the spend. The related trap, why hours saved is not ROI, is the same error in a different metric: an input dressed up as a return.

FAQ

Q: What is the difference between AI adoption and AI impact? A: Adoption measures engagement — seats, active users, prompts, hours saved. Impact measures whether the work got better and the spend paid off: output per dollar, net of rework, tied to an outcome. Adoption is an input; impact is the return. Two teams can show identical adoption while one ships clean work and the other generates output that gets redone. Only impact tells them apart. Q: Are AI usage dashboards useless? A: Not useless — just insufficient. Usage data (seats, prompts, active users) is a legitimate input for spotting who has access and where AI is being tried. It becomes a vanity metric when it's reported as value. The fix is to pair every usage number with its impact metric — output-per-dollar-net-of-rework, per team, against that team's own baseline — so activity is context, not the conclusion. Q: Why can't Microsoft Copilot analytics measure my AI ROI? A: Because it's vendor-locked. Copilot and Viva analytics report on Microsoft's own AI and are blind to ChatGPT, Claude, Cursor, and in-house agents your teams also use. That over-credits one supplier and under-counts the AI spend you're accountable for. Measuring ROI requires a vendor-agnostic view across every AI tool in the stack, tied to output net of rework — not one vendor's usage feed. Q: What should we measure instead of AI adoption? A: Measure output-per-dollar-net-of-rework, per team, against each team's own pre-AI baseline, across every AI vendor. Concretely: accepted, non-reworked outputs per dollar of AI spend, plus an AI rework rate (the share of AI output that had to be redone). That answers the fourth question — did the work actually improve — which 78% of finance execs say they still can't answer (CloudZero, 2026). Q: Is "hours saved" a vanity metric? A: Yes, when it's reported as ROI. Self-reported hours saved is a survey estimate of a freed input, not a delivered outcome — and the freed time is often spent fixing AI output. In one study, 41% of workers received AI "workslop" costing ~1h 56m each to resolve (BetterUp Labs + Stanford, 2025). Saved time becomes return only when it converts to an outcome and survives rework.

Sources

See the return on your own numbers.

Get a clear, per-team return-on-AI picture in weeks — output net of rework, reconciled to your invoice.