Epistemic Question: How do we measure the true productivity impact of AI coding agents?

[weisberg]

Epistemic Question

Core Question: How do we measure the true productivity impact of AI coding agents, distinguishing between perceived speed and actual delivered value?

Why This Matters

Agile Agentic Analytics documents the "verification bottleneck and illusion of speed" failure mode: controlled evidence shows AI tools can slow experienced developers in familiar codebases because time shifts into prompting, waiting, and verification—yet developers may still believe they are faster.

This creates a measurement paradox: traditional velocity metrics become less meaningful when agents can spike output, but what should replace them?

Related Framework

The Agile Agentic Analytics page recommends a balanced KPI set:

• DORA metrics (deployment frequency, lead time, change failure rate)
• Agent productivity metrics (cycle time, PR merge rate, thrash rate)
• Quality metrics (escaped defects, flaky tests, maintainability)
• Hallucination/plausibility-failure rate
• Cost and latency (tokens, CI minutes)

But the question remains: How do we know if we're measuring the right things? How do we prevent gaming these metrics?

Open Questions to Explore

1. Counterfactual measurement: How do we establish a reliable baseline for "what would have happened without agents"?
2. Value vs. volume: Should we measure lines of code generated, or business value delivered? If the latter, how?
3. Learning curve effects: How long until teams reach "steady state" productivity, and how do we account for the transition period?
4. Maintenance burden: How do we capture the hidden cost of reviewing/debugging AI-generated code that may increase work for senior developers?
5. Long-term code health: Do current velocity gains come at the expense of technical debt accumulation?

Potential Research Directions

• Survey existing empirical studies on AI coding assistant productivity (academic + industry)
• Design A/B test framework for team-level productivity measurement
• Develop "productivity toxicity" indicators (high velocity but declining quality)
• Create rubric for "maintenance burden ratio" (time spent fixing AI outputs vs. manual coding baseline)
• Explore causal inference methods for attribution (what % of productivity change is genuinely due to AI?)

Success Criteria for Answering This Question

We will know we've made progress when we can:

1. Define 3-5 leading indicators that predict sustainable productivity (not just short-term throughput)
2. Establish measurement protocols that work across different team skill levels and codebases
3. Distinguish between "tool-amplified productivity" and "tool-induced toil"
4. Provide decision rules: "When should we increase agent autonomy?" vs. "When should we pull back?"

Cross-References

• Vibe Analytics: Principle 2 - Specification Over Execution
• Agile Agentic Analytics: Metrics and KPIs
• Risks and Mitigation: Quality Decay

[m13v]

been thinking about this a lot since we run 5+ agents in parallel on our codebase daily. pure lines-of-code or PR count metrics are misleading because agents generate verbose code. what actually correlates with real productivity for us is time-to-working-feature and number of human review cycles before merge. also tracking cost per feature is underrated - sometimes an agent burns through tokens on something a human would've done faster.

[m13v]

we track per-query costs and token usage in our agent provider layer if you want to see how we instrument it: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatProvider.swift

and the tool execution side where we measure how many tool calls each task takes (another useful proxy for complexity): https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatToolExecutor.swift

[voidborne-d]

Q5 is the one I keep bouncing off of — the cost doesn't surface until someone has to touch the code again, so it's invisible to merge-time measurement.

A few things that have actually worked from running multiple agents in parallel:

Debt-delta per merge as a time series. Run a static analyzer before and after each PR, record the composite-score delta bucketed by authorship (human / agent / mixed). You don't need the absolute number to be meaningful — the rate of change is what reveals "productivity toxicity." If agent PRs land at +0 median delta while human PRs net -0.3 over a quarter, the agent is indistinguishable from a contributor who never refactors. That's the leading indicator.

Counterfactual replacement for Q1. Save the agent's diff, then regenerate a minimal-diff version of the same behavior change ("same result, smallest edit"). Measure the size ratio. First-pass agent output is usually 2-4× the minimal equivalent; that gap is what turns into technical debt. It's also a metric the model can't see during generation, so it's hard to game.

Review cycles — decompose by cause. @m13v's point on review count is right, but the subtypes diverge: cycles on style/obvious issues are a lagging indicator teams learn to skip (so they drop even as debt climbs). Cycles on structural issues (duplication, dead branches, missing error paths, god files) keep rising until someone fixes the generation prompt itself. Separating those two is what makes review-cycle metrics actually predictive.

On instrumentation: the categories that matter for AI-authored code specifically are duplication patterns (AI rewrites instead of refactoring), dead code (v5 kept, v1-v4 still there), missing error paths (happy-path bias), god files, and dependency bloat. Any analyzer covering those can serve — I maintain sober-coding scored along those axes, but the principle works with any composite you trust.

On Q4 (maintenance burden ratio) concretely: I'd measure it as (reviewer time) + 2×(time anyone spends touching that file in the next 90 days attributable to the AI change). The 2× weight accounts for follow-on debugging done by someone who didn't write it — cognitive switching cost is real but hidden in flat time accounting.