AI Coding Metrics

Measures coding assistants and coding agents without collapsing results into vanity metrics or one blended score.

The critical distinction is mode: assistants help inline or in chat; agents execute multi-step work and need task-level measurement. Do not measure them as if they were the same thing.

When to Use This Skill

Defaults

Workflow

1. Define the decision.
2. Pick the program mode: assistant, agent, or mixed.
3. Build the minimum viable scorecard.
4. Choose the study design.
5. Produce one deliverable.

ASCII Flow

AI coding metrics request
  -> decision to support: buy, renew, improve, prove, or diagnose
  -> split mode: assistant, agent, or mixed
  -> select scorecard families: adoption, delivery, quality, economics, experience
  -> choose study design and baseline window
  -> collect team-level and task-level evidence
  -> report confidence, sample size, and confounds
  -> deliver ROI model, dashboard, experiment plan, or executive report

Quick Reference

Decision to Deliverable Map

Program Modes

Metric Families

Use the smallest scorecard that can answer the decision:

Study Design Defaults

Minimum baseline: 8 weeks of pre-intervention data. Two-week baselines produce noisy causal inference — week-to-week variance in PR throughput, review lag, and defect escape routinely exceeds the signal size of AI tooling effects.

Cohort and denominator contract

Freeze the measurement population before reading outcomes. Record the eligible population, assignment rule, actual exposure, observation window, and accepted outcome for each metric. Report eligible, assigned, exposed, and observed counts side by side; never silently replace the assigned cohort with active users, completed tasks, or merged PRs. That survivor-only denominator makes adoption and success look better precisely when setup failures, abandoned agent runs, or unmerged changes are the problem.

For incomplete observations, name the reason (not_started, abandoned, still_open, telemetry_missing, or excluded_by_rule) and keep it in the funnel. Treat still-open work as right-censored rather than failed until the outcome window closes. A report may be directional with imperfect telemetry, but it must state which denominator supports each percentage and how missing cases could change the decision.

Measurement Checklist

Use before publishing any AI coding report:

• Baseline established (≥8 weeks before intervention)
• Assistant and agent funnels tracked separately
• Every speed metric paired with at least one quality metric
• Sample size, confidence level, and study design stated
• Confounds documented (team changes, release pressure, policy changes)
• Vendor evidence labeled as vendor evidence
• Usage measured after stabilization (not week-1 novelty period)
• Review burden and rework cost included in ROI model
• Edit-capable agents measured across evolving-spec checkpoints, including late-checkpoint cost and quality slopes
• Aggregated at team level (no manager-visible individual dashboards)

Current Evidence Posture (as of 2026-08-21)

Anti-Gaming Checklist

Reject a scorecard or report if any of the following apply:

• Single blended AI productivity score mixing usage, speed, sentiment, and quality
• Seat activation or prompt volume cited as delivery impact
• Cross-team comparison without controlling for stack, task mix, staffing, or release pressure
• Measurement period is <8 weeks or includes week-1 novelty window
• Vendor benchmark cited as production ROI evidence
• Review burden excluded from ROI model
• Individual-level AI usage visible to managers
• Directional before/after movement stated as causal without controlled design
• SlopCodeBench averages or trajectory signals used as organizational targets or causal ROI evidence

Navigation

References

• references/adoption-metrics.md — assistant and agent adoption funnels, metric definitions, stall patterns, privacy rules
• references/productivity-metrics.md — DORA and SPACE applied to AI workflows, delivery stack decomposition, confound management
• references/quality-metrics.md — defect, complexity, test, security, and technical debt metrics with targets and alert thresholds
• references/roi-framework.md — full cost model (including review burden), benefit model, scenario planning, executive report structure
• references/developer-experience-metrics.md — satisfaction surveys, cognitive load, friction indicators, trust calibration, DX anti-patterns
• references/agent-execution-metrics.md — agent funnel, core metrics, reviewer burden, scorecards for pilot / scaling / executive decisions
• references/benchmarking-methodology.md — A/B, before/after, crossover, shadow designs; statistical rigor; confound management
• references/theory-of-constraints-applied.md — bottleneck identification before instrumenting, throughput accounting for ROI, DBR for review-queue protection, CRT for stalled rollouts, evaporating cloud for adoption-vs-quality tensions
• references/evidence-update.md — load when citing current research: METR RCT (2025 baseline), METR 2026 update (selection-bias caveat), DORA 2025 AI report, DX Core 4, Faros 2026 telemetry

Assets and data

• assets/metric-dashboard-template.md
• assets/adoption-survey-template.md
• assets/roi-calculator-template.md
• assets/executive-report-template.md
• assets/experiment-design-template.md
• data/sources.json
• data/sample-ai-metrics.json

Scripts

• scripts/roi_calculator.py
• scripts/README.md

Cross-References

• ../dev-context-engineering/SKILL.md
• ../ai-agents/SKILL.md
• ../qa-observability/SKILL.md
• ../product-management/SKILL.md

Fact-Checking

• Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
• Verify current research claims, benchmark status, and vendor telemetry specifics before final advice.
• Prefer peer-reviewed, official, and first-party telemetry docs over social or vendor marketing claims.
• If live verification is unavailable, mark current-evidence claims as unverified.

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.