AIReturn

AI ROI by team

AI ROI in Software Engineering: Beyond 'Developers Feel Faster'

GitHub Copilot ROI isn't acceptance rate or 'feels faster.' It's shipped software net of rework — churn, reverts, reviews — vs. your team's baseline.

Rodrigo Paredes BassiPublished 10 min read

TL;DR

  • GitHub Copilot ROI is the shipped, working software your team delivers per dollar of AI spend, net of rework — not acceptance rate, adoption, or "feels faster." Suggestion-acceptance measures whether developers take the code, not whether the code ships and holds.
  • In METR's early-2025 randomized trial, experienced developers using AI tools were measured 19% slower while estimating they were 20% faster — a 39-point perception gap. Felt speed is not a reliable ROI input.
  • Vendor telemetry across 10,000+ developers found AI raised throughput (21% more tasks completed, 98% more PRs merged) but also 154% larger PRs, 91% longer reviews, and 9% more bugs per developer — with no delivery gain at the organizational level (Faros AI, 2025, directional).
  • AI rework is where the speed-up leaks back out: code churn, reverts, extra PR review rounds, reopened issues, and change-failure rate. Velocity measured gross of rework overstates return.
  • Engineering-intelligence tools measure velocity. AIReturn measures velocity net of rework, tied to shipped software, against your team's own baseline — and extends the same read across support, sales, product, and ops.

What "GitHub Copilot ROI" actually means

GitHub Copilot ROI is the value of the working software your team ships per dollar of AI spend, after subtracting the cost of redoing AI output that didn't hold. It is not the suggestion-acceptance rate, not seat adoption, and not a survey of how much faster developers feel. Those are inputs — activity signals that can all rise while the amount of shipped, durable software stays flat. This is the most-measured AI function in most companies, and still the most mismeasured. Engineering has the richest telemetry of any team — commits, PRs, reviews, CI, incidents — so the temptation is to grab the easiest number (acceptance rate, PRs per week) and call it return. The problem is that the easy numbers count motion, not delivery. Return is output-per-dollar-net-of-rework: shipped software, minus the cost of fixing what came back, over what the AI cost to produce it.

Why acceptance rate and "feels faster" are not ROI

Two of the most common Copilot-ROI proxies fail for the same reason: they measure the moment of generation, not the outcome downstream. Acceptance rate tells you a developer accepted a suggestion. It says nothing about whether that code passed review unchanged, survived to production, or got reverted a week later. A high acceptance rate on code that generates three extra review rounds and a rollback is a cost, not a return. "Feels faster" is the more seductive trap, because it is often sincere — and often wrong. In a randomized controlled trial published by METR in July 2025, 16 experienced open-source developers worked 246 tasks across large, familiar repositories. Developers using early-2025 AI tools were measured 19% slower, yet estimated they had been 20% faster — a 39-point gap between felt and measured performance. One caveat, stated plainly: METR later relabeled that result "historical" and redesigned its methodology in February 2026, so treat the -19% as a single early-2025 snapshot, not a timeless law that "AI makes developers slower." The durable finding is not the direction of the number. It is the size of the perception gap. When a team says "20% faster" and the stopwatch disagrees by 39 points, self-reported time-saved cannot anchor an ROI model. Lead with the outcome; treat the feeling as a hypothesis to test.

The velocity trap: more output, same delivery

Engineering-intelligence platforms answer "are we moving faster?" — and increasingly the honest answer is "yes, and it isn't reaching customers." The sharpest illustration comes from a vendor study that deliberately contrasted itself with METR. Faros AI analyzed telemetry from 10,000+ developers across 1,255 teams in real work settings, over as long as two years. High-AI-adoption teams shipped more: 9% more tasks touched per day, 47% more pull requests, 21% higher task-completion rates, and 98% more merged PRs. On any velocity dashboard, that is a win. But the same data showed the cost of that volume: 154% larger PRs, 91% longer code-review times, and 9% more bugs per developer as adoption grew — and, critically, no correlation between AI adoption and faster delivery at the organizational level. More PRs, more merges, more bugs, more review load — same shipped software. (Directional: these are one vendor's telemetry, not audited, and reported as correlation, not proven cause. Use them to frame the mechanism, not as a settled dollar.) That is the velocity trap in one line: more PRs can mean more rework. A velocity metric read gross of rework counts the extra PRs as progress and never subtracts the extra reviews, the larger diffs, and the bugs they carry. AIReturn reads velocity net of that — the throughput of the team paying for AI, minus the friction it generated getting to a working outcome.

AI rework in engineering: the signals that matter

AI rework is the redo cost of AI output that looked done but wasn't — why rework quietly erases AI ROI is the cross-functional version of this story. In engineering, rework is not a vibe; it is already logged in the systems your team runs. These are the signals that separate shipped software from motion.

Rework signalWhat it measuresWhy AI inflates it
Code churnLines rewritten/deleted soon after they're mergedAI generates verbose, first-draft code that gets reworked
Reverts / rollbacksMerged changes later backed outPlausible code that fails in production
PR review roundsBack-and-forth cycles before a PR mergesLarger, AI-generated diffs need more review passes
Reopened issuesTickets marked done that come backWork that looked complete but wasn't
Change-failure rateShare of deploys causing a failure/hotfixHigher when generation outpaces verification
These are read as a trend against the team's own pre-AI history, normalized to an index — not as a cross-team scoreboard.
The measurement rule is strict: compare each signal only against that team's own baseline, never against another team. A platform team and a front-end team don't share churn norms, and measuring each team against its own baseline, not a cross-team benchmark is what keeps the read honest. The question is never "is this team's churn high?" — it is "did this team's churn, reverts, and review rounds move since AI, relative to where they were before?"

Tie it to an outcome: shipped, working software

Velocity dashboards stop at motion. ROI has to land on an outcome — and in engineering the outcome is shipped, working software: changes that reach production and stay there without a rollback or a reopened issue. That reframes the whole calculation. The numerator is not "PRs opened" — it is durable, delivered change. The denominator is not just the Copilot bill — it is the AI cost plus the human cost of getting the work to a state that ships. Put together:

AI ROI = (business outcome delivered − rework cost) ÷ (AI cost + human cost to get it right), measured per team, continuously, across every AI vendor. Output-per-dollar-net-of-rework is the unit. For engineering, "output" is working software delivered; "rework" is the churn/reverts/review-rounds/reopens tax above; "dollar" is granular AI cost — attributed by skill, model, and product, not a single aggregate token bill. Read together against the baseline, they answer "is Copilot worth it" with a number instead of a feeling. You can see a sample per-team AI ROI report for how that verdict renders.

Where AIReturn sits vs. engineering-intelligence tools

Engineering-intelligence platforms (the DORA/velocity cluster) do real work: they surface cycle time, deployment frequency, and PR throughput, and the better ones now flag AI-era side effects like larger diffs and slower reviews. But two structural gaps remain. First, they measure velocity; AIReturn measures velocity net of rework, tied to shipped software and priced against granular AI cost. A faster cycle time on code that gets reverted is not return; subtracting the rework and dividing by the actual AI spend is what turns a velocity chart into an ROI number. Second, and sharper: your engineers are not your only AI users. Sales, support, legal, product, and ops all run AI, and none of it shows up in a DORA dashboard. AIReturn runs the same output-net-of-rework read across every function and every vendor — Copilot, Cursor, Claude, ChatGPT, in-house agents — so how AI ROI differs team by team becomes one comparable view, and the CFO can decide how to allocate the AI budget per team instead of defending one org-wide "AI is working" claim. An honest limit: isolating AI-caused rework from a team's baseline rework is the hardest open question in this work. Engineering is where it is most concrete — churn, reverts, reopens, and review rounds are already instrumented — but the exact lag between an AI-produced change and the rework it triggers is not fully pinned. We would rather name that boundary than sell a precise dollar we can't yet defend.

Common mistakes measuring engineering AI ROI

  • Reporting acceptance rate as ROI. It measures whether code was taken, not whether it shipped and held.
  • Trusting "feels faster." The 39-point perception gap (METR, early 2025) is why felt speed is a hypothesis, not a result.
  • Counting PRs gross of rework. More PRs with 154% larger diffs and 91% longer reviews (Faros, directional) can be net-negative.
  • Ignoring change-failure rate. Throughput that raises rollbacks and hotfixes is buying motion, not delivery.
  • Benchmarking against other teams. Compare each team to its own pre-AI baseline; churn norms aren't portable.
  • Measuring once. Rework moves as models, prompts, and workflows change — a quarterly snapshot misses it; a continuous read catches it.

FAQ

Is GitHub Copilot worth it for our engineering team? It depends on shipped software net of rework, not on adoption or acceptance rate. Copilot is worth it when the working software your team delivers per dollar rises versus its own pre-AI baseline, after subtracting code churn, reverts, extra review rounds, and reopened issues. If throughput climbs but change-failure rate and review load climb with it, the tool may be net-neutral or negative — measure before you conclude. How do you measure AI developer productivity beyond acceptance rate? Measure delivered, durable software and subtract rework. Track code churn, reverts/rollbacks, PR review rounds, reopened issues, and change-failure rate, normalized against the team's own history. Acceptance rate tells you code was accepted; these signals tell you whether it shipped and held. In METR's early-2025 trial, developers felt 20% faster while measured 19% slower — proof that felt speed is not a productivity measure. Does GitHub Copilot actually make developers faster? Sometimes, on some tasks — but "faster developer" is not the same as "faster delivery." One vendor study of 10,000+ developers found AI raised task completion by 21% and merged PRs by 98%, yet with 154% larger PRs, 91% longer reviews, 9% more bugs, and no faster delivery at the organizational level (Faros AI, 2025, directional). Individual speed can rise while shipped software stays flat. What is AI rework in software engineering? AI rework is the redo cost of AI-generated code that looked done but wasn't — the effort to fix, re-review, or revert it. In engineering it shows up as code churn, reverts/rollbacks, extra PR review rounds, reopened issues, and a higher change-failure rate. It is the term you subtract from velocity to get real return; measured gross of rework, throughput overstates ROI. Why can't our engineering-intelligence dashboard show AI ROI? Because it measures velocity, not velocity net of rework tied to shipped software and priced against AI cost. A faster cycle time on code that gets reverted isn't return. It also only sees engineering — your sales, support, and ops AI usage never appears. Proving ROI needs output net of rework, versus each team's baseline, across every function and vendor — which a single-lane DORA view doesn't do.

Sources

See the return on your own numbers.

Get a clear, per-team return-on-AI picture in weeks — output net of rework, reconciled to your invoice.