Why You Can't Compare AI ROI Across Teams (Baseline, Not Benchmark)

TL;DR

• You can't compare AI productivity across teams, because their output isn't the same unit. A merged pull request, a resolved ticket, and a closed deal don't convert to one another — so ranking teams by "AI ROI" is a category error, not a scoreboard.
• The only valid comparison is each team versus its own pre-AI baseline over time. The question is never "which team got more from AI," but "did this team's output-net-of-rework improve since AI, relative to where it was before?"
• Cross-team scoreboards don't just mislead — they create perverse incentives: teams optimize the visible metric (volume) and quietly generate the invisible cost (rework), which is exactly what a benchmark can't see.
• Measurement is team-level, never individual. Individual AI uplift (directionally 20–40% for coding, per Faros/DX) doesn't roll up to team output, and individual scoreboards cross a privacy line without producing a better number.
• Only 28% of AI use cases fully meet their ROI expectations (Gartner, Apr 2026). A defensible baseline — not a borrowed benchmark — is what separates the teams that can prove return from the ones still guessing.

Can you compare AI ROI across teams?

No. You cannot compare AI ROI across teams, because each function's output is denominated in a different unit, and units that don't convert can't be ranked. Engineering ships merged changes; support closes cases; sales advances deals. Putting those on one axis and calling the taller bar "more AI ROI" is a category error. The only valid comparison is each team measured against its own pre-AI baseline over time. That has a sharp consequence: there is no company-wide "AI ROI" number worth reporting, and no cross-team benchmark worth chasing. This post explains why the comparison fails, why the scoreboard version backfires, why measurement sits at the team level not the individual, and how to set a baseline you can defend to a CFO. It is the measurement rule underneath how AI ROI differs team by team.

The definitions this rule runs on

Three terms carry the argument. Per-team baseline is a function's own output-net-of-rework measured before AI — the only valid comparison point for that function's AI ROI, never a cross-team or cross-company average. The reference is always this team, now, versus this team, before AI. Output-per-dollar-net-of-rework is the core ROI unit: the useful output a team ships per dollar of AI spend, after subtracting the cost of redoing flawed AI work. In full: (business outcome delivered − rework cost) ÷ (AI cost + human cost to get it right), per team, continuously, across every AI vendor. Team-level measurement means the return is measured and reported at the team level, not the individual — because rework, cost, and outcome only reconcile at the team level, and individual measurement crosses a privacy line without improving the number.

Why a PR isn't a ticket isn't a deal

Cross-team comparison fails at the first step: the outcomes don't share units. Here is what each function actually ships, and why the numbers refuse to line up.

The only valid comparison: each team vs. its own baseline

If teams can't be compared to each other, what's left is comparing each team to itself. The valid unit of AI ROI is change over the team's own pre-AI baseline — improvement, not level. "This engineering team ships 11% more durable software per AI dollar than it did before AI" is a defensible claim. "Engineering has higher AI ROI than support" is not a claim at all; it's a unit error. A baseline is honest for a reason a benchmark can't match: it holds the units constant and absorbs what makes functions structurally different, isolating the one variable you care about — the introduction of AI, against an otherwise steady reference. A support team's baseline already reflects support's inherent volume, so you're not penalizing it for not looking like engineering. Deloitte's 2026 survey shows why the "level" comparison misleads even on clean data — 66% of organizations report efficiency gains but only 20% report revenue gains (n=3,235); a cross-team benchmark blends those two realities into a middle number that describes no actual team, while a per-team baseline keeps them separate.

Why cross-team scoreboards mislead — and create perverse incentives

A cross-team AI leaderboard doesn't just produce a meaningless number; it changes behavior in the wrong direction. Once a team knows it's ranked against other teams on an AI metric, it optimizes for the visible metric — almost always volume, which is precisely what AI inflates for free and rework quietly erases. A team climbs the scoreboard by shipping more pull requests or generating more proposals, while the reverts, reopens, and stage regressions that follow land downstream, on someone else's week, and never touch the ranking. The scoreboard rewards the inflation and hides the cost: more motion, ranked as more value — the opposite of what a CFO is trying to fund. The perverse incentives compound in three specific ways:

• It rewards the wrong output. Ranking on volume tells teams to maximize the number AI already inflates, not the durable outcome net of rework.
• It punishes honest functions. A team that slows down to review AI output before it ships looks worse on a velocity leaderboard than a team that ships fast and reworks later — exactly backwards.
• It manufactures a false winner. Because the units don't convert, whichever team ships the highest-volume, easiest-to-count work item "wins" the ranking regardless of actual return. The scoreboard crowns a category error. The perception gap makes this worse: the teams generating the most rework often feel the most productive. In METR's early-2025 trial, 16 experienced developers were measured 19% slower using AI while estimating they were 20% faster (METR, Jul 2025; later relabeled "historical," so read it as one snapshot). A scoreboard built on self-reported speed ranks the feeling; a baseline built on measured output-net-of-rework measures the outcome.

Why measurement is team-level, not individual

The correct unit is the team, not the person — a methodological rule and a privacy principle at once. Individual AI uplift is real but doesn't roll up: industry estimates put individual coding gains at roughly 20–40% (Faros/DX, 2026, directional, vendor-adjacent), yet a faster individual can generate rework that lands on a teammate, or lift their own numbers while the team's accepted-outcome rate falls. Measured per person, the redo cost disappears into someone else's week; measured per team against its own baseline, it stays on the ledger. The privacy principle is just as load-bearing for a financial-grade tool. Measuring AI's return does not require ranking individual developers or scoring keystrokes to the CFO — the decision it feeds is fund this team, remediate that one, which resolves at the team level. Even where individual signals inform the underlying model, the return is reported at the team level, so the redo cost that only reconciles in aggregate stays on the ledger rather than vanishing into one person's week. AIReturn reports at the team level for exactly this reason: the economics and the ethics point the same way.

How to set a defensible baseline

A baseline is only useful if you can defend it to a skeptical CFO. Setting one is concrete work, done per function — the same discipline as our step-by-step framework for measuring AI ROI. Five steps make it decision-grade rather than anecdotal:

1. Pick the team's real outcome work item. Name the unit of useful output that team ships — merged change, resolved case, advanced deal — not an activity proxy like tokens or seats. The outcome, not the motion, is what the baseline measures.
2. Capture a clean pre-AI window. Measure output-net-of-rework over a representative stretch before the team adopted AI, long enough to smooth seasonality (a full quarter is a reasonable floor). This is the reference the AI period is read against.
3. Define the function's rework signal. Specify what "getting redone" looks like in that team's own tools — reverts in engineering, reopens in support, stage regression in sales — and normalize it to an index. The baseline includes the rework the team always had, so AI's effect shows as the delta.
4. Attribute AI cost granularly, as the denominator. Cost the AI usage by skill, model, and product — not a single aggregate token bill — so the denominator is precise enough to defend. OpenTelemetry (OTEL) captures usage cost at the level of the work, not the invoice.
5. Compare only to that same team, and re-date it. Read the AI-period number against the team's own baseline, never another team's, and refresh continuously — rework drifts as models, prompts, and workflows change. A quarterly snapshot misses it; a continuous read catches it. One honest limit belongs here. Isolating AI-caused rework from the baseline rework a team always had is the hardest open question in this work. It's concrete in engineering — churn, reverts, reopens, and review rounds are already logged, which is why AI ROI in engineering, where the baseline is most instrumented is the deepest case — but for deals, support cases, and product initiatives the rework proxy is still being defined. The method is strongest in engineering today and expands function by function as those proxies mature; anyone claiming precise, universally validated cross-function rework attribution in 2026 is overselling.

Why the baseline is the measure that gets funded

The benchmark version is what has stalled AI budgets: only 28% of AI use cases fully meet their ROI expectations (Gartner, 782 I&O leaders, Apr 2026), and a cross-team scoreboard is just a faster way to that same unfundable number. A baseline survives contact with a CFO where a benchmark can't. "We rank third on the AI leaderboard" invites the obvious question — third at what, measured how? — while "this team's output-net-of-rework is up 11% versus its own pre-AI baseline, on a denominator we attribute by model and product" is a claim the board can review, challenge, and fund. That's the difference between a scoreboard and an accounting. To see the baseline rendered as a deliverable, look at how AIReturn measures each team against its own baseline.

FAQ

Can you compare AI productivity across teams? No. Output isn't comparable across functions — a merged pull request, a resolved ticket, and a closed deal aren't the same unit, so there's no exchange rate to rank them by. The only valid comparison is each team against its own pre-AI baseline over time. You compare a team's improvement, function by function, never its output level against another team's. What is an AI ROI baseline? An AI ROI baseline is a team's own output-net-of-rework measured before it adopted AI, used as the sole reference point for that team's AI return. It's never a cross-team or cross-company average. Because functions ship different units and carry different inherent rework, the baseline holds those differences constant and isolates the one variable that changed: the introduction of AI. Why is comparing AI ROI across departments a mistake? Because it commits a category error and creates perverse incentives. The units don't convert — ranking functions against each other measures something that doesn't exist — and once teams know they're ranked, they optimize the visible metric (volume, which AI inflates) while the invisible cost (rework) lands downstream. Deloitte found 66% of organizations see efficiency gains but only 20% see revenue gains (2026); a blended cross-team number hides which teams actually earn their spend. Should AI productivity be measured per person or per team? Per team, against that team's own baseline. Individual uplift (directionally 20–40% for coding, per Faros/DX) doesn't roll up — a faster individual can generate rework that lands on a teammate, and per-person measurement lets the redo cost vanish into someone else's week. Team level is where cost, output, and rework reconcile, and it gives a CFO the decision without crossing into individual surveillance. Is there a good AI productivity benchmark to hit? No fixed benchmark exists, because there's no universal unit to benchmark against — the target is improvement over your own baseline, function by function. Directionally, independent surveys put the share of AI use cases meeting their ROI target at roughly 28% (Gartner, 2026), which is a reminder that most spend still needs proof, not evidence of a number to chase. Measure each team against itself, continuously, and re-date as models and workflows change.

Sources

• Deloitte — State of AI in the Enterprise 2026 (66% report efficiency gains vs. 20% reporting revenue gains; n=3,235), 2026. https://www.deloitte.com/us/en/what-we-do/capabilities/applied-artificial-intelligence/content/state-of-ai-in-the-enterprise.html
• Gartner — AI Projects in Infrastructure and Operations Stall Ahead of Meaningful ROI Returns (28% of use cases meet ROI, 782 I&O leaders), April 2026. https://www.gartner.com/en/newsroom/press-releases/2026-04-07-gartner-says-artificial-intelligence-projects-in-infrastructure-and-operations-stall-ahead-of-meaningful-roi-returns
• METR — "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity," July 2025 (19% slower / 20% faster; result relabeled historical, method redesigned February 2026). https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
• Faros / DX — individual AI coding uplift ~20–40% (directional, vendor-adjacent; individual, not team-level), 2026. https://www.faros.ai/blog/lab-vs-reality-ai-productivity-study-findings

Related

• Up to the pillar: how AI ROI differs team by team.
• The function where the baseline is most instrumented: AI ROI in engineering, where the baseline is most instrumented.
• The unit it measures: output minus rework as the working definition of AI productivity.
• Run it: our step-by-step framework for measuring AI ROI.
• See the output: how AIReturn measures each team against its own baseline.