AIReturn

The CFO & the AI budget

Agent ROI in the Age of Agent-Washing: Measuring What Agents Deliver

Agent ROI is output per dollar, net of rework — for agents and copilots in one view. How to measure agent-fleet return amid agent-washing (2026).

Rodrigo Paredes BassiPublished 10 min read

TL;DR

  • Agent ROI is the business output an agent fleet delivers per dollar of AI spend, net of the human rework it creates — measured for agents and human copilots in one view. Not task completions, not "autonomy," not a demo.
  • Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027 — cost, unclear value, and weak risk controls — and counts only ~130 genuine agentic vendors among the thousands claiming the label. The gap between the two is agent-washing.
  • Meanwhile the spend is real: IDC projects worldwide AI spending to reach $1.3T by 2029, driven by agentic AI, with a 10x rise in the number and complexity of enterprise agents. The bill scales whether or not the return does.
  • Only 28% of AI use cases in infrastructure and operations fully meet their ROI expectations (Gartner, Apr 2026). Measuring an agent fleet demands more than the usual dashboard: output net of rework, cost by model and skill, and the one axis nobody else instruments — the agent harness itself.
  • The differentiator is AHOE: score each agent's maturity L1→L4, find the bottleneck across six axes, rank the next moves by Impact×Effort, and track the improvement back on the same Cost × Rework matrix — the loop from diagnosis to measured gain.

What is agent ROI, and why is it hard to measure?

Agent ROI is the business output an agent fleet produces per dollar of AI spend, after subtracting the cost of redoing the work it got wrong. It is the same output-per-dollar-net-of-rework unit AIReturn applies to any AI spend — but agents make the denominator noisier and the rework harder to see. An agent chains many model calls, tool calls, and retries to reach an outcome. Each step costs money and can introduce error. So an agent that "completes the task" can still be net-negative once a human re-checks, re-instructs, or re-does its output. Task completion is a vanity metric; output net of rework is the P&L metric. That is why most agent programs can't answer the question their board is now asking. The spend is legible; the value is not.

Definition block

  • Agent-washing (named by Gartner) — Rebranding conventional software — chatbots, RPA, rules engines, a single scripted LLM call — as "agentic AI" without genuine autonomous, goal-directed behavior. Gartner estimates ~130 vendors are genuinely agentic out of thousands claiming the label (Jun 2025). The marketing outran the capability.
  • The 4th question — The one nobody answers continuously: did AI actually improve the work — by team, tied to business outcomes, net of rework — and what do I fix? For an agent fleet it becomes: is this agent delivering output worth more than it costs to run and to correct — and which move raises that number next?
  • Output-per-dollar-net-of-rework — AIReturn's core ROI unit: the business output a team (or agent fleet) delivers per dollar of AI spend, after subtracting the cost of redoing flawed AI work. Cost is the denominator; rework-adjusted output is the numerator.
  • AHOE (Agent Harness Optimization Engine) — AIReturn's agent-fleet depth. After diagnosis, it benchmarks each agent and improves the harness — prompt, model routing, tools, context — scoring a maturity vector L1→L4, locating the bottleneck across six axes, and ranking the next moves by Impact×Effort, then tracking the resulting improvement back on the Cost × Rework matrix.

Why are 40%+ of agentic projects being canceled?

Because the label was sold faster than the return could be proven. Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls (Jun 2025). The named culprit is agent-washing: buyers deployed "agents" that were dressed-up chatbots, then couldn't show a return the CFO would fund. The spend keeps climbing regardless. IDC projects worldwide AI spending to reach $1.3T by 2029, driven by agentic AI, and expects a 10x increase in the number and complexity of third-party and custom agents inside five years (Sep 2025). More agents, more model calls, more surface area for silent rework — and, for now, thin proof of return: only 28% of AI use cases in infrastructure and operations fully meet their ROI expectations, while 20% fail outright (Gartner, Apr 2026). The lesson for a Chief AI Officer is not "agents don't work." It is that a fleet you can't measure is a fleet you can't defend. Cancellation is what happens to spend without a number attached to it. Proving the number — the Chief AI Officer's mandate — is now the job.

How to tell a real agent from agent-washing

Four questions separate genuine agentic behavior from a repackaged script:

  • Does it decide, or only respond? A real agent selects among actions to reach a goal; a washed one runs a fixed script triggered by a prompt.
  • Does it plan across steps? Genuine agents sequence and revise multi-step plans; chatbots answer one turn at a time.
  • Does it use tools and act on the result? Agents call tools, read the outcome, and adapt; RPA executes a predetermined path.
  • Can you measure its output net of rework? If the vendor can only show completions and latency — never the human cost to correct the output — you are buying a demo, not a return.

The metrics that actually measure an agent fleet

Measuring agents needs three layers most tools skip. The first two are table stakes. The third is the moat.

LayerWhat it measuresWhy it matters for agents
Output net of reworkAccepted, non-reworked agent output vs. the team's own prior baselineAn agent that "finishes" but triggers re-checks and re-dos is net-negative — completions hide this
Cost by model & skillGranular AI cost attributed per skill, per model, per product via OTEL — not an aggregate token billA chatty agent routing every step to the priciest model can erase its own return; you can't fix what you can't attribute
Harness state (AHOE)Agent maturity L1→L4, the bottleneck across six axes, next moves ranked by Impact×EffortAgents are tunable — the harness (prompt, routing, tools, context) is where most wasted spend and most rework actually live
Note the cost row. AIReturn attributes AI cost by skill, by model, and by product — including several products at once — via OTEL. It is not a lump-sum token bill. For an agent that fans out across dozens of model calls, per-step cost attribution is the difference between "this fleet costs $X" and "this skill on this model is where the money and the rework go."

The differentiator: optimizing the harness, not just scoring it

Most tools that touch agents stop at observability — traces, evals, latency. A green eval is not a good outcome, and a groundedness score does not tell a CFO whether the work shipped or got rewritten. AIReturn's AHOE goes past the score to the fix. The loop is diagnosis → prescription → measured improvement:

  1. Diagnose each agent on the Cost × Rework matrix — high/low AI cost against high/low rework — for a per-agent verdict: scale, keep, fix, or cut.
  2. Score maturity L1→L4 — from a brittle single-prompt agent (L1) to a routed, tool-using, context-managed, self-correcting fleet (L4).
  3. Find the bottleneck across six harness axes — prompt, model routing, tools, context, control flow, and error handling — so you know which lever is capping the return.
  4. Rank the next moves by Impact×Effort — a prioritized list, not a research project: the change that lifts output-net-of-rework most, for the least work, first.
  5. Track the improvement back on the same matrix over time — so a harness change shows up as a measured shift in cost and rework, not a claim in a slide. This is the loop competitors leave open. FinOps tools price the agent. Observability tools evaluate it. Neither improves the harness and then proves the improvement moved the P&L number. That closed loop — and the fact that agents and human copilots sit on one view — is the CFO's guide to the AI budget made operational for the agent era.

Common mistakes when measuring agent ROI

  • Counting completions as value. A completed task that gets reworked is a cost, not a win. Measure output net of rework, always.
  • Reading the aggregate token bill as "cost." Without per-skill, per-model attribution you can't see which step is bleeding margin — or fix it.
  • Trusting green evals as ROI. Eval scores measure the model; the CFO is asking about the work. They are not the same axis.
  • Measuring agents in isolation from copilots. Your people run copilots and your fleet runs agents against the same outcomes. One view or you double-count the effort and miss the rework.
  • Treating the harness as fixed. Most agent waste is tunable. If you never optimize prompt, routing, tools, and context, you are paying full price for L1 maturity forever.

Agents and copilots, in one view

A Chief AI Officer does not own "the agents" and "the copilots" as separate budgets — they own the return on AI across the work. AIReturn measures human copilots (Copilot, ChatGPT, Claude, Cursor) and autonomous agents in parallel, in the same Cost × Rework matrix, against each team's own history. That matters because the two trade off. Shifting a workflow from a human-in-the-loop copilot to an autonomous agent can cut cost and raise rework — or the reverse. You can only see the net effect if both live on one axis. This is where the agent-fleet depth is most built out today — in engineering and technical work, where rework is concrete (churn, reverts, reopens, review rounds) — and it expands into other functions as each one's output and rework definitions mature.

"The question isn't whether an agent completed the task. It's whether the output was worth more than the cost to run it and the cost to fix it — and which change to the harness raises that number next. That's the number a CFO funds." — AIReturn Research

FAQ

Q: What is AI agent ROI, and how is it different from regular AI ROI? A: AI agent ROI is the business output an agent fleet delivers per dollar of AI spend, net of the human rework it creates. It uses the same output-per-dollar-net-of-rework unit as any AI ROI — but agents chain many paid model and tool calls, so cost and rework are harder to trace. Task completions are a vanity metric; output net of rework is the P&L metric. Q: What is agent-washing? A: Agent-washing, a term named by Gartner, is rebranding conventional software — chatbots, RPA, rules engines, a single scripted LLM call — as "agentic AI" without genuine autonomous, goal-directed behavior. Gartner estimates only ~130 vendors are truly agentic out of thousands claiming the label (Jun 2025). It is a leading reason projects get funded, underdeliver, and are canceled. Q: Why are so many agentic AI projects being canceled? A: Gartner predicts more than 40% of agentic AI projects will be canceled by the end of 2027, citing rising costs, unclear business value, and weak risk controls (Jun 2025). Much of it traces to agent-washing: teams deployed "agents" that were repackaged scripts, then couldn't prove a return. A fleet you can't measure is a fleet you can't defend. Q: How do you measure the ROI of an AI agent fleet? A: In three layers. Measure output net of rework against each team's own baseline; attribute cost granularly by skill, model, and product via OTEL — not as a lump token bill; then diagnose and optimize the harness itself. AIReturn's AHOE scores maturity L1→L4, finds the bottleneck across six axes, ranks next moves by Impact×Effort, and tracks the gain on a Cost × Rework matrix. Q: Can you measure AI agents and human copilots together? A: Yes — and you should, because they trade off. Moving a workflow from a copilot to an autonomous agent can cut cost while raising rework, or the reverse. AIReturn plots agents and copilots (Copilot, ChatGPT, Claude, Cursor, in-house agents) on one Cost × Rework matrix, against each team's own history, so a Chief AI Officer sees the net effect on output-per-dollar-net-of-rework, not two disconnected dashboards.

Sources

See the return on your own numbers.

Get a clear, per-team return-on-AI picture in weeks — output net of rework, reconciled to your invoice.