The cost unit is accepted change

A tool can produce a patch quickly while shifting effort to context preparation, review, correction, and integration. METR's July 2025 randomized study found a 19% completion-time slowdown for experienced open-source maintainers using early-2025 AI tools on their own repositories. That specific result is a useful warning against equating generation speed with developer speed. It is not a universal estimate: the task mix, tools, and participants are defined. DORA's 2025 report discusses AI-assisted software development in a broader organizational setting, which is observational rather than the same randomized task experiment.

Build a complete ledger

For a representative assignment, record human preparation time, agent runtime, direct usage cost if measurable, review time, rework, tests, and any later repair. Distinguish elapsed calendar time from labor time: parallel agent work may reduce one while increasing the other. Count abandoned outputs and duplicated attempts. A patch that never merges is not zero cost. A patch that merges quickly and causes a later incident is not a clean win. This ledger is an editorial measurement proposal, not a claim that either cited research used exactly these accounting categories.

Review capacity can become the bottleneck

If generated changes arrive faster than maintainers can inspect them, the queue grows. Reviewers then see larger diffs, more context switches, and weaker accountability for assumptions. Bound assignments so a reviewer can understand each change and its evidence in one pass. Require tests that exercise the acceptance criteria, but remember tests can encode the wrong requirement. For risky paths, add an owner who knows the affected behavior. A successful build is a useful signal; it is not a complete audit of data flow, security, or long-term maintenance.

Choose a comparison that matters

Compare a small sample of similar work under ordinary team conditions, with and without agent help when feasible. Define success as accepted behavior over a follow-up window, and publish uncertainty rather than a single percentage stripped of context. METR's February 2026 update explains why its newer productivity experiment faced selection problems; that reinforces the need to report who chose which workflow and why. Teams may still rationally use AI for learning, accessibility, or tasks they would otherwise defer. Value is wider than immediate speed, but the chosen value measure should be stated before claiming improvement.

What to carry into the work

  • Time preparation, review, rework, and follow-up.
  • Count abandoned attempts and queue time.
  • Use comparable task families and state selection limits.
  • Report accepted outcomes, not raw patch volume.
Evidence boundary: this is a sourced editorial guide, not a hands-on product evaluation. Recommendations are our engineering analysis. Product documentation describes intended behavior, not independent proof of reliability.

Sources & dates

  1. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity ↗METR · 10 Jul 2025 · Checked 16 Sept 2026
  2. State of AI-assisted Software Development 2025 ↗DORA · Undated source · Checked 16 Sept 2026
  3. We are Changing our Developer Productivity Experiment Design ↗METR · 24 Feb 2026 · Checked 16 Sept 2026

Unknown source dates stay undated. Preparation is not publication; no historical byline or interview is implied.