Story ideas: AI tools and coding

Central conclusion Studies disagree on whether developers overestimate how much AI tools speed them up

supported approved supported (approved): 1 supporting (1 independent, strongest 0.8), 1 contradicting, 1 qualifying.

Contradicted by Copilot RCT (Peng): In the Copilot RCT both groups estimated a ~35% productivity gain, an underestimate of the measured 55.8% speedup.

• Supports METR 2025 RCT right 80%, direct lines 17-24Developers forecast a 24% speedup and afterward estimated 20%, but were actually slowed 19%. Experts also overestimated, predicting 38-39% faster.

• Contradicts Copilot RCT (Peng) wrong 45%, direct (reviewed) lines 184-188In the Copilot RCT both groups estimated a ~35% productivity gain, an underestimate of the measured 55.8% speedup.

• Qualifies METR 2026 update partly 35%, partial lines 63-63METR Feb 2026: some developers self-report very high speedups, which METR notes can be 'quite unreliable' per its earlier study; it also says the true speedup may be higher among those who selected out, so no new measured overestimate.Applicability: the verdict note limits its own scope ("self-report")

• Other Anthropic internal study unknown, partial lines 36-36Anthropic's +50% figures are self-reported; the post cites METR's overestimation finding and admits perception, social-desirability and recall biases, so it cannot settle the question.Applicability: the verdict note limits its own scope ("self-reported")

Finding 1 (evidence) Developers overestimate how much AI tools speed them up

disputed qualify review disputed (qualify): 1 supporting (1 independent, strongest 0.8), 1 contradicting, 1 qualifying.

Contradicted by Copilot RCT (Peng): In the Copilot RCT both groups estimated a ~35% productivity gain, an underestimate of the measured 55.8% speedup.

Suggestion: Present the disagreement: name the studies on each side instead of stating one side as fact.

• Supports METR 2025 RCT right 80%, direct lines 17-24Developers forecast a 24% speedup and afterward estimated 20%, but were actually slowed 19%. Experts also overestimated, predicting 38-39% faster.

• Contradicts Copilot RCT (Peng) wrong 45%, direct (reviewed) lines 184-188In the Copilot RCT both groups estimated a ~35% productivity gain, an underestimate of the measured 55.8% speedup.

• Qualifies METR 2026 update partly 35%, partial lines 63-63METR Feb 2026: some developers self-report very high speedups, which METR notes can be 'quite unreliable' per its earlier study; it also says the true speedup may be higher among those who selected out, so no new measured overestimate.Applicability: the verdict note limits its own scope ("self-report")

• Other Anthropic internal study unknown, partial lines 36-36Anthropic's +50% figures are self-reported; the post cites METR's overestimation finding and admits perception, social-desirability and recall biases, so it cannot settle the question.Applicability: the verdict note limits its own scope ("self-reported")

Central conclusion AI tools slowed experienced open-source developers working on their own familiar codebases (METR 2025 RCT)

supported approved supported (approved): 2 supporting (2 independent, strongest 0.85).

• Supports METR 2025 RCT right 85%, direct lines 11-27METR RCT: 16 experienced developers, 246 tasks in their own mature repos; allowing early-2025 AI tools increased completion time by 19%.
• Supports METR 2026 update right 70%, direct lines 58-62METR Feb 2026 restates the early-2025 result (tasks took 19% longer with AI, CI +2% to +39%) but adds that the 10 returning developers are now estimated ~18% faster with AI (CI -38% to +9%), so the slowdown was specific to early 2025.

Finding 1 (evidence) AI tools slowed experienced open-source developers working on their own familiar codebases (METR 2025 RCT)

supported approved supported (approved): 2 supporting (2 independent, strongest 0.85).

• Supports METR 2025 RCT right 85%, direct lines 11-27METR RCT: 16 experienced developers, 246 tasks in their own mature repos; allowing early-2025 AI tools increased completion time by 19%.
• Supports METR 2026 update right 70%, direct lines 58-62METR Feb 2026 restates the early-2025 result (tasks took 19% longer with AI, CI +2% to +39%) but adds that the 10 returning developers are now estimated ~18% faster with AI (CI -38% to +9%), so the slowdown was specific to early 2025.

Finding 2 (evidence) In real enterprise codebases AI yields modest average productivity gains, far below headline claims

qualified qualify review qualified (qualify): 2 supporting (2 independent, strongest 0.7), 2 qualifying, 1 context only.

METR 2025 RCT: METR 2025: in large, mature repos (~10 years old, >1.1M LOC) AI helped less than in greenfield RCTs, and on average slowed developers by 19%; this goes beyond 'modest gains', and the repos are open-source rather than enterprise. [partial: the verdict note limits its own scope ("rather than")]

Shan et al. field study + RCT: Field DID at a 5,000-person IT services firm (201 devs, 27 weeks, 2023-24 Copilot/ChatGPT/Gemini): AI users completed 1.08 more user stories per week (+18%), a real gain but modest next to 55% or 10x headlines; usage was self-reported and partly self-selected. [partial: the verdict note limits its own scope ("self-reported")]

Context only, Google migrations report (partly): bespoke migration pipelines with fine-tuned models and engineers' estimates; not average developer work in an enterprise codebase.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Supports Google RCT right 70%, direct lines 519-528Google RCT, 96 full-time engineers on an enterprise-grade C++ task in the monorepo: AI cut time on task by ~21% in the best-fit model (95% CI crosses zero, p=0.086), well below Peng et al.'s 56% and in line with Cui et al.'s 26%.
• Supports Microsoft/Accenture field RCTs right 70%, direct lines 774-784Field RCTs with 4,867 developers at Microsoft, Accenture and a Fortune-100 firm: Copilot raised weekly completed PRs by 26% (SE 10.3%), substantially smaller than the lab's 56% speedup, which the authors attribute to real work being less assistable.

• Qualifies METR 2025 RCT partly 50%, partial lines 995-1001METR 2025: in large, mature repos (~10 years old, >1.1M LOC) AI helped less than in greenfield RCTs, and on average slowed developers by 19%; this goes beyond 'modest gains', and the repos are open-source rather than enterprise.Applicability: the verdict note limits its own scope ("rather than")
• Qualifies Shan et al. field study + RCT partly 50%, partial lines 489-493Field DID at a 5,000-person IT services firm (201 devs, 27 weeks, 2023-24 Copilot/ChatGPT/Gemini): AI users completed 1.08 more user stories per week (+18%), a real gain but modest next to 55% or 10x headlines; usage was self-reported and partly self-selected.Applicability: the verdict note limits its own scope ("self-reported")

• Context Google migrations report partly 30%, contextual (reviewed) lines 245-251Google reports large end-to-end savings (about 50% on int64, about 89% on small Joda clusters) in a 500M+ LOC codebase, but only for bespoke migration pipelines with fine-tuned models, not average developer work, and the figures are engineers' estimates.Applicability: bespoke migration pipelines with fine-tuned models and engineers' estimates; not average developer work in an enterprise codebase

Finding 3 (evidence) The evidence does not support "Engineers who adopt agentic coding are about ten times as productive as peers who don't"

qualified qualify review qualified (qualify): no direct support, 2 qualifying.

Anthropic internal study: Anthropic: users self-report an average +50% gain at 59% usage, plus about 67% more merged PRs per engineer after Claude Code adoption, an order of magnitude below 10x. [partial: the verdict note limits its own scope ("self-report")]

Shan et al. field study + RCT: The field gain from AI assistants was about 18% more user stories (24-31% under alternative estimators), far from 10x. The tools were 2023-24 assistants, not agentic coding. [partial: measures 2023-24 assistants, not agentic coding; it bounds the plausible gain rather than testing the 10x claim]

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Anthropic internal study wrong 40%, partial lines 35-35Anthropic: users self-report an average +50% gain at 59% usage, plus about 67% more merged PRs per engineer after Claude Code adoption, an order of magnitude below 10x.Applicability: the verdict note limits its own scope ("self-report")
• Qualifies Shan et al. field study + RCT wrong 30%, partial (reviewed) lines 489-493The field gain from AI assistants was about 18% more user stories (24-31% under alternative estimators), far from 10x. The tools were 2023-24 assistants, not agentic coding.Applicability: measures 2023-24 assistants, not agentic coding; it bounds the plausible gain rather than testing the 10x claim

Finding 4 (evidence) The measured AI slowdown is mainly an artifact of older models and would disappear with newer ones

qualified qualify review qualified (qualify): no direct support, 1 qualifying.

METR 2026 update: METR Feb 2026: METR believes developers are likely more sped up in early 2026. Returning developers show an 18% time reduction (CI -38% to +9%) and new recruits a 4% reduction (CI -15% to +9%), but selection effects make this 'very weak evidence', and models are not separated from other changes. [partial: METR's own belief with selection effects it calls very weak evidence; models are not separated from other changes]

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies METR 2026 update partly 40%, partial (reviewed) lines 61-62METR Feb 2026: METR believes developers are likely more sped up in early 2026. Returning developers show an 18% time reduction (CI -38% to +9%) and new recruits a 4% reduction (CI -15% to +9%), but selection effects make this 'very weak evidence', and models are not separated from other changes.Applicability: METR's own belief with selection effects it calls very weak evidence; models are not separated from other changes

• Other METR 2025 RCT unknown, direct lines 587-599The paper tests only Feb-June 2025 tools. It says more reliable or better-elicited systems could speed up developers, and that early agents already make progress, but it doesn't test newer models.

Central conclusion AI-assisted code is no harder for later developers to maintain than unassisted code

qualified qualify review qualified (qualify): 1 supporting (1 independent, strongest 0.7), 2 qualifying, 1 context only.

Shan et al. field study + RCT: The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers. [partial: the verdict note limits its own scope ("self-reports")]

Vibe-coding literature review: 11 of 47 sources raise maintainability concerns: conversational code that lacks cohesive structure and documentation, structural complexity from rapid iteration, and long-term debt. These are reported concerns rather than measured comparisons with unassisted code. [partial: the verdict note limits its own scope ("rather than")]

Context only, Code-ownership experiment (Martin-Lopez) (right): a related-work citation of Borg et al.; the study's own data cover only the original author, not later maintainers.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Supports Echoes of AI maintainability RCT right 70%, direct lines 2519-2524Preregistered RCT, 75 mostly professional developers manually evolving Java code written with or without AI: no reliable difference in completion time or perceived productivity, and the study is only powered to rule out large effects.

• Qualifies Shan et al. field study + RCT wrong 30%, partial lines 1219-1229The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers.Applicability: the verdict note limits its own scope ("self-reports")
• Qualifies Vibe-coding literature review wrong 20%, partial lines 1434-145211 of 47 sources raise maintainability concerns: conversational code that lacks cohesive structure and documentation, structural complexity from rapid iteration, and long-term debt. These are reported concerns rather than measured comparisons with unassisted code.Applicability: the verdict note limits its own scope ("rather than")

• Context Code-ownership experiment (Martin-Lopez) right 30%, contextual (reviewed) lines 880-892The related work cites Borg et al.'s 151-participant controlled experiment, which found no consistent sign that AI-assisted code was harder for later developers to maintain, while warning of cognitive debt. This is secondary evidence; the study's own data cover only the original author.Applicability: a related-work citation of Borg et al.; the study's own data cover only the original author, not later maintainers

• Other Debt study (302k AI commits) unknown, direct lines 1031-1035The mining study of 302k AI-authored commits explicitly has no human-written baseline, so its counts of surviving debt cannot show that AI code is harder to maintain than human code.

Finding 1 (evidence) AI-assisted code is no harder for later developers to maintain than unassisted code

qualified qualify review qualified (qualify): 1 supporting (1 independent, strongest 0.7), 2 qualifying, 1 context only.

Shan et al. field study + RCT: The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers. [partial: the verdict note limits its own scope ("self-reports")]

Vibe-coding literature review: 11 of 47 sources raise maintainability concerns: conversational code that lacks cohesive structure and documentation, structural complexity from rapid iteration, and long-term debt. These are reported concerns rather than measured comparisons with unassisted code. [partial: the verdict note limits its own scope ("rather than")]

Context only, Code-ownership experiment (Martin-Lopez) (right): a related-work citation of Borg et al.; the study's own data cover only the original author, not later maintainers.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Supports Echoes of AI maintainability RCT right 70%, direct lines 2519-2524Preregistered RCT, 75 mostly professional developers manually evolving Java code written with or without AI: no reliable difference in completion time or perceived productivity, and the study is only powered to rule out large effects.

• Qualifies Shan et al. field study + RCT wrong 30%, partial lines 1219-1229The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers.Applicability: the verdict note limits its own scope ("self-reports")
• Qualifies Vibe-coding literature review wrong 20%, partial lines 1434-145211 of 47 sources raise maintainability concerns: conversational code that lacks cohesive structure and documentation, structural complexity from rapid iteration, and long-term debt. These are reported concerns rather than measured comparisons with unassisted code.Applicability: the verdict note limits its own scope ("rather than")

• Context Code-ownership experiment (Martin-Lopez) right 30%, contextual (reviewed) lines 880-892The related work cites Borg et al.'s 151-participant controlled experiment, which found no consistent sign that AI-assisted code was harder for later developers to maintain, while warning of cognitive debt. This is secondary evidence; the study's own data cover only the original author.Applicability: a related-work citation of Borg et al.; the study's own data cover only the original author, not later maintainers

• Other Debt study (302k AI commits) unknown, direct lines 1031-1035The mining study of 302k AI-authored commits explicitly has no human-written baseline, so its counts of surviving debt cannot show that AI code is harder to maintain than human code.

Finding 2 (evidence) AI-generated code frequently contains security vulnerabilities

qualified qualify review qualified (qualify): no direct support, 2 qualifying.

Debt study (302k AI commits): AI commits introduce more correctness and security issues than they fix, across all five tools. But about 7.5-14.8% of commits introduce any issue at all, and most issues are smells. [partial: counts issues introduced, most of them smells, in 7.5-14.8% of commits; 'frequently contains vulnerabilities' needs a vulnerability prevalence rate]

Vibe-coding literature review: Security risk is the most-cited open challenge (10 of 20 RQ8 sources); peer-reviewed and practitioner sources agree generated code may contain weaknesses when it is not rigorously verified, but no prevalence rate is synthesized.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Debt study (302k AI commits) partly 50%, partial (reviewed) lines 918-933AI commits introduce more correctness and security issues than they fix, across all five tools. But about 7.5-14.8% of commits introduce any issue at all, and most issues are smells.Applicability: counts issues introduced, most of them smells, in 7.5-14.8% of commits; 'frequently contains vulnerabilities' needs a vulnerability prevalence rate
• Qualifies Vibe-coding literature review partly 35%, direct lines 1793-1803Security risk is the most-cited open challenge (10 of 20 RQ8 sources); peer-reviewed and practitioner sources agree generated code may contain weaknesses when it is not rigorously verified, but no prevalence rate is synthesized.

Finding 3 (evidence) Developers should be able to explain AI-generated code without the AI's help before it is merged

qualified qualify review qualified (qualify): no direct support, 1 qualifying.

Code-ownership experiment (Martin-Lopez): Crossover experiment, 69 participants (mostly students): after AI-assisted tasks people answered fewer questions about their own just-written code correctly (median 87.5% vs 100%; AME -5.95 pp, OR 0.58). The authors argue this can stop contributors defending their choices in code review. That supports the reason behind the norm, but no merge gate was tested.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Code-ownership experiment (Martin-Lopez) partly 45%, direct lines 700-721Crossover experiment, 69 participants (mostly students): after AI-assisted tasks people answered fewer questions about their own just-written code correctly (median 87.5% vs 100%; AME -5.95 pp, OR 0.58). The authors argue this can stop contributors defending their choices in code review. That supports the reason behind the norm, but no merge gate was tested.

Central conclusion The evidence does not support "AI coding tools amplify experienced engineers far more than novices"

qualified qualify review qualified (qualify): 4 supporting (4 independent, strongest 0.5), 3 qualifying.

METR 2025 RCT: METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure. [partial: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites]

Google RCT: Google RCT: more senior engineers and those coding 5+ hours/day were faster with AI, but the AI x seniority (p=0.706) and AI x hours interactions were not significant, so 'far more' is unsupported.

Anthropic internal study: Anthropic: experience is needed to spot subtly dangerous Claude proposals, yet a senior engineer says the tools make juniors 'more productive and more bold', so the gain isn't clearly skewed to seniors.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Supports Microsoft/Accenture field RCTs wrong 50%, direct lines 758-770Microsoft experiment: short-tenure developers raised output 27-39% and juniors 21-40%, versus 8-13% for long-tenure and 7-16% for seniors (directionally consistent, mostly not significant), the opposite of AI amplifying experienced engineers more.
• Supports Shan et al. field study + RCT wrong 45%, direct lines 872-878Moderation analysis in the 253-developer RCT: developers with fewer years of experience got the speed benefit and quality was unaffected, so gains concentrated among less-skilled coders, not experienced ones (easy tasks, ChatGPT-3.5 chat, appendix result).
• Supports Copilot RCT (Peng) wrong 40%, direct lines 221-225Copilot RCT heterogeneity analysis: developers with less programming experience benefited more, which the authors highlight as promise for novices (small sample, noisy estimates).
• Supports Code-ownership experiment (Martin-Lopez) wrong 25%, direct lines 556-569Every group, BSc students included, gained significantly in completeness, and the authors say differences between groups were not related to experience. Professionals gained the most, but they had different tasks, and position is confounded with task and language.

• Qualifies METR 2025 RCT wrong 60%, partial (reviewed) lines 135-141METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure.Applicability: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites
• Qualifies Google RCT partly 30%, direct lines 490-505Google RCT: more senior engineers and those coding 5+ hours/day were faster with AI, but the AI x seniority (p=0.706) and AI x hours interactions were not significant, so 'far more' is unsupported.
• Qualifies Anthropic internal study partly 30%, direct lines 75-75Anthropic: experience is needed to spot subtly dangerous Claude proposals, yet a senior engineer says the tools make juniors 'more productive and more bold', so the gain isn't clearly skewed to seniors.

Finding 1 (evidence) The evidence does not support "AI coding tools amplify experienced engineers far more than novices"

qualified qualify review qualified (qualify): 4 supporting (4 independent, strongest 0.5), 3 qualifying.

METR 2025 RCT: METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure. [partial: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites]

Google RCT: Google RCT: more senior engineers and those coding 5+ hours/day were faster with AI, but the AI x seniority (p=0.706) and AI x hours interactions were not significant, so 'far more' is unsupported.

Anthropic internal study: Anthropic: experience is needed to spot subtly dangerous Claude proposals, yet a senior engineer says the tools make juniors 'more productive and more bold', so the gain isn't clearly skewed to seniors.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Supports Microsoft/Accenture field RCTs wrong 50%, direct lines 758-770Microsoft experiment: short-tenure developers raised output 27-39% and juniors 21-40%, versus 8-13% for long-tenure and 7-16% for seniors (directionally consistent, mostly not significant), the opposite of AI amplifying experienced engineers more.
• Supports Shan et al. field study + RCT wrong 45%, direct lines 872-878Moderation analysis in the 253-developer RCT: developers with fewer years of experience got the speed benefit and quality was unaffected, so gains concentrated among less-skilled coders, not experienced ones (easy tasks, ChatGPT-3.5 chat, appendix result).
• Supports Copilot RCT (Peng) wrong 40%, direct lines 221-225Copilot RCT heterogeneity analysis: developers with less programming experience benefited more, which the authors highlight as promise for novices (small sample, noisy estimates).
• Supports Code-ownership experiment (Martin-Lopez) wrong 25%, direct lines 556-569Every group, BSc students included, gained significantly in completeness, and the authors say differences between groups were not related to experience. Professionals gained the most, but they had different tasks, and position is confounded with task and language.

• Qualifies METR 2025 RCT wrong 60%, partial (reviewed) lines 135-141METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure.Applicability: METR's own trial had only experienced developers; the comparison with novices comes from prior work it cites
• Qualifies Google RCT partly 30%, direct lines 490-505Google RCT: more senior engineers and those coding 5+ hours/day were faster with AI, but the AI x seniority (p=0.706) and AI x hours interactions were not significant, so 'far more' is unsupported.
• Qualifies Anthropic internal study partly 30%, direct lines 75-75Anthropic: experience is needed to spot subtly dangerous Claude proposals, yet a senior engineer says the tools make juniors 'more productive and more bold', so the gain isn't clearly skewed to seniors.

Finding 2 (evidence) AI productivity gains concentrate among developers new to a codebase and on routine, well-trodden code

disputed qualify review disputed (qualify): 1 supporting (1 independent, strongest 0.6), 1 contradicting, 5 qualifying.

Contradicted by Google RCT: Google RCT on code unfamiliar to all participants found, if anything, more senior engineers gained more from AI (non-significant interaction), contrary to gains concentrating among less experienced developers.

Suggestion: Present the disagreement: name the studies on each side instead of stating one side as fact.

• Supports METR 2025 RCT right 60%, direct lines 960-984Slowdown is larger when developers have high prior task exposure. Developers report AI helps on unfamiliar issues and not where they're experts, and the paper expects speedups in unfamiliar or greenfield settings.

• Contradicts Google RCT wrong 25%, direct lines 490-505Google RCT on code unfamiliar to all participants found, if anything, more senior engineers gained more from AI (non-significant interaction), contrary to gains concentrating among less experienced developers.

• Qualifies Microsoft/Accenture field RCTs partly 45%, direct lines 758-770Microsoft experiment: gains were larger for short-tenure (newer to the company and its code) and junior developers (27-39% vs 8-13%), but differences were not significant and the routine-code half of the claim was not tested.
• Qualifies Copilot RCT (Peng) partly 40%, direct lines 174-177Copilot RCT: less experienced developers benefited more on a routine, well-trodden task (a basic HTTP server), consistent with the claim but from a small freelancer sample and without a codebase-familiarity test.
• Qualifies Anthropic internal study partly 40%, partial lines 70-70Anthropic: staff delegate where they have low context and complexity is low, and avoid tasks needing tacit codebase knowledge, which mirrors METR's slowdown factors; this concerns task choice rather than measured gains.Applicability: the verdict note limits its own scope ("rather than")
• Qualifies Shan et al. field study + RCT partly 35%, direct lines 872-878Less-experienced developers gained more speed in the RCT, consistent with the claim, but a follow-up on harder APPS tasks suggested quantity gains were larger on complex tasks (0.283 vs 0.245), which does not fit 'routine code only'.
• Qualifies Code-ownership experiment (Martin-Lopez) partly 30%, direct lines 240-247Large completeness gains (+43 points) came on small, popular-domain tasks chosen so that LLMs could help, many of them greenfield. This fits gains on routine code, but the study had no mature codebases to compare against.

Finding 3 (evidence) Getting good results from AI coding tools takes significant skill, so outcomes vary widely with user expertise

qualified qualify review qualified (qualify): no direct support, 7 qualifying.

Echoes of AI maintainability RCT: Bayesian analysis: the more AI-skilled the original developer, the larger the downstream CodeHealth benefit (+0.10, about half an SD, for habitual users). The authors conclude developers vary in their ability to use AI well, though Java proficiency mattered even more. [partial: the verdict note limits its own scope ("though")]

Google RCT: Google RCT, 96 engineers: senior and heavy-coding developers were tentatively faster with AI and natural-language-to-code needs prompting skill, but how often people already used AI tools had no significant effect on their AI speed-up, so the authors point to a 'low usability floor'.

Anthropic internal study: Anthropic survey (n=132): time-savings responses cluster at opposite ends, with some spending more time debugging AI code; authors attribute gains to learned 'strategic delegation skills' and task choice, supporting wide variation by user practice.

Shan et al. field study + RCT: Results depended on how developers used ChatGPT: direct acceptance raised output but not accuracy, while ideation, improvement and debugging raised accuracy, and debugging lowered output. Outcomes vary with usage style, but usage was self-chosen, so the evidence is correlational.

METR 2025 RCT: METR RCT: speedup did not improve over the first 50 hours of Cursor use, and the lone >50h developer's speedup (~25%) fell to ~0% once a second developer's misreported experience was corrected. The 'high skill ceiling' idea is plausible but underpowered.

Microsoft/Accenture field RCTs: Field RCTs, ~4,900 developers: Copilot gains varied with experience, but in the opposite direction from 'skill needed'. Short-tenure and junior developers gained 21-40% while senior and long-tenure developers gained 7-16% (noisy, not significant).

Vibe-coding literature review: Vibe-coding MLR (47 sources, mention counts): quality, maintainability and productivity outcomes are described as conditional on developer expertise, task complexity and verification practice, and outcome claims vary substantially by user expertise. [partial: mention counts across sources, not a measurement of outcome variance by expertise]

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Echoes of AI maintainability RCT right 50%, partial lines 2541-2551Bayesian analysis: the more AI-skilled the original developer, the larger the downstream CodeHealth benefit (+0.10, about half an SD, for habitual users). The authors conclude developers vary in their ability to use AI well, though Java proficiency mattered even more.Applicability: the verdict note limits its own scope ("though")
• Qualifies Google RCT partly 40%, direct lines 522-543Google RCT, 96 engineers: senior and heavy-coding developers were tentatively faster with AI and natural-language-to-code needs prompting skill, but how often people already used AI tools had no significant effect on their AI speed-up, so the authors point to a 'low usability floor'.
• Qualifies Anthropic internal study partly 40%, direct lines 36-40Anthropic survey (n=132): time-savings responses cluster at opposite ends, with some spending more time debugging AI code; authors attribute gains to learned 'strategic delegation skills' and task choice, supporting wide variation by user practice.
• Qualifies Shan et al. field study + RCT partly 40%, direct lines 1037-1064Results depended on how developers used ChatGPT: direct acceptance raised output but not accuracy, while ideation, improvement and debugging raised accuracy, and debugging lowered output. Outcomes vary with usage style, but usage was self-chosen, so the evidence is correlational.
• Qualifies METR 2025 RCT partly 35%, direct lines 1217-1234METR RCT: speedup did not improve over the first 50 hours of Cursor use, and the lone >50h developer's speedup (~25%) fell to ~0% once a second developer's misreported experience was corrected. The 'high skill ceiling' idea is plausible but underpowered.
• Qualifies Microsoft/Accenture field RCTs partly 35%, direct lines 758-770Field RCTs, ~4,900 developers: Copilot gains varied with experience, but in the opposite direction from 'skill needed'. Short-tenure and junior developers gained 21-40% while senior and long-tenure developers gained 7-16% (noisy, not significant).
• Qualifies Vibe-coding literature review right 30%, partial (reviewed) lines 1518-1526Vibe-coding MLR (47 sources, mention counts): quality, maintainability and productivity outcomes are described as conditional on developer expertise, task complexity and verification practice, and outcome claims vary substantially by user expertise.Applicability: mention counts across sources, not a measurement of outcome variance by expertise

• Other Code-ownership experiment (Martin-Lopez) unknown, direct lines 828-835Gains appeared in every experience group, but the authors state they cannot separate experience effects from task and language effects, so the study cannot say whether outcomes depend on user expertise.

Central conclusion AI assistance must be very reliable to save experts time, because checking and fixing its output is costly

qualified qualify review qualified (qualify): 1 supporting (1 independent, strongest 0.6), 2 qualifying.

Anthropic internal study: Anthropic: engineers skip Claude for short tasks or ones needing tacit team context ('I'm just going to go and do it myself') and find familiar work faster done by hand, consistent with checking costs eating the benefit.

Vibe-coding literature review: Sources often discuss defects alongside productivity gains, suggesting that faster development 'may introduce additional verification burdens', and the review calls gains conditional on verification; there is no time accounting.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Supports METR 2025 RCT right 60%, direct lines 1004-1012Developers accepted under 44% of Cursor generations and lost significant time reviewing, testing and modifying output before rejecting it. Low reliability is listed as a cause of the slowdown.

• Qualifies Anthropic internal study partly 35%, direct lines 67-69Anthropic: engineers skip Claude for short tasks or ones needing tacit team context ('I'm just going to go and do it myself') and find familiar work faster done by hand, consistent with checking costs eating the benefit.
• Qualifies Vibe-coding literature review partly 25%, direct lines 1454-1458Sources often discuss defects alongside productivity gains, suggesting that faster development 'may introduce additional verification burdens', and the review calls gains conditional on verification; there is no time accounting.

Finding 1 (evidence) AI assistance must be very reliable to save experts time, because checking and fixing its output is costly

qualified qualify review qualified (qualify): 1 supporting (1 independent, strongest 0.6), 2 qualifying.

Anthropic internal study: Anthropic: engineers skip Claude for short tasks or ones needing tacit team context ('I'm just going to go and do it myself') and find familiar work faster done by hand, consistent with checking costs eating the benefit.

Vibe-coding literature review: Sources often discuss defects alongside productivity gains, suggesting that faster development 'may introduce additional verification burdens', and the review calls gains conditional on verification; there is no time accounting.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Supports METR 2025 RCT right 60%, direct lines 1004-1012Developers accepted under 44% of Cursor generations and lost significant time reviewing, testing and modifying output before rejecting it. Low reliability is listed as a cause of the slowdown.

• Qualifies Anthropic internal study partly 35%, direct lines 67-69Anthropic: engineers skip Claude for short tasks or ones needing tacit team context ('I'm just going to go and do it myself') and find familiar work faster done by hand, consistent with checking costs eating the benefit.
• Qualifies Vibe-coding literature review partly 25%, direct lines 1454-1458Sources often discuss defects alongside productivity gains, suggesting that faster development 'may introduce additional verification burdens', and the review calls gains conditional on verification; there is no time accounting.

Finding 2 (evidence) As AI speeds up generation, human attention for review and understanding becomes the bottleneck

qualified qualify review qualified (qualify): 2 supporting (2 independent, strongest 0.55), 1 qualifying.

Code-ownership experiment (Martin-Lopez): AI users produced more, larger code but understood it less. Their median answering time was 3 min longer (+27%) and they switched back to the code more than twice as often, though the time effect was not robust. The authors suggest time saved may be offset later in the lifecycle. [partial: the verdict note limits its own scope ("though")]

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Supports Google migrations report right 55%, direct lines 417-423Google JUnit3->4 migration (5,359 files in 3 months): the bottleneck was how fast engineers could review, and change generation was deliberately throttled so reviewers were not overwhelmed.
• Supports Vibe-coding literature review right 40%, direct lines 1906-1914The review's central synthesis is a 'verification gap': a mismatch between how fast AI workflows generate code and developers' capacity to verify it. This is an interpretive synthesis of mention-level evidence.

• Qualifies Code-ownership experiment (Martin-Lopez) partly 40%, partial lines 736-767AI users produced more, larger code but understood it less. Their median answering time was 3 min longer (+27%) and they switched back to the code more than twice as often, though the time effect was not robust. The authors suggest time saved may be offset later in the lifecycle.Applicability: the verdict note limits its own scope ("though")

Finding 3 (evidence) Developers need solid programming knowledge to validate and correct AI-generated code

qualified qualify review qualified (qualify): 4 supporting (4 independent, strongest 0.5), 1 qualifying.

METR 2025 RCT: METR 2025: 75% of experienced developers read every line of AI code, 56% often make major cleanups, and 100% report needing to modify it; this shows correction is routinely needed, but the study does not test less-skilled users.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Supports Anthropic internal study right 50%, direct lines 93-94Anthropic: the 'paradox of supervision' (supervising Claude needs the coding skills overuse may erode), and a security engineer saw a 'dangerously smart' proposal that only experienced judgment could catch.
• Supports Vibe-coding literature review right 50%, direct lines 1929-1932The review concludes that skill in verification, constraint specification and review may matter as much as authoring skill, and that the central risk is a verification gap.
• Supports Google RCT right 40%, direct lines 484-493Google RCT authors suggest heavier coders gain more from AI because current tools require spending much time verifying and editing generated code (interaction beta=-0.29, not significant), so coding skill conditions the benefit.
• Supports Echoes of AI maintainability RCT right 40%, direct lines 2549-2556The Task 1 developer's Java proficiency influenced downstream outcomes more than AI use, and authors conclude human expertise still matters; qualitative quotes describe using experience to catch hacky AI suggestions.

• Qualifies METR 2025 RCT partly 45%, direct lines 1015-1021METR 2025: 75% of experienced developers read every line of AI code, 56% often make major cleanups, and 100% report needing to modify it; this shows correction is routinely needed, but the study does not test less-skilled users.

Central conclusion AI-generated code drifts from a codebase's established conventions

qualified qualify review qualified (qualify): no direct support, 3 qualifying.

METR 2025 RCT: METR 2025: developers report AI code leaves 'more to fix up, in terms of code conventions, code style' and needs major cleanup to meet repo standards (qualitative, 16 developers). [partial: qualitative reports from 16 developers, not a measured drift rate against a human baseline]

Debt study (302k AI commits): Correctness issues dominated by undefined variables (23,856) lead authors to say AI code 'may look locally correct, but still fail to stay consistent with the surrounding context'; this is consistency with surrounding code, not style conventions specifically.

Vibe-coding literature review: WL27 is quoted saying conversationally generated code 'often lacks cohesive structure, consistent patterns, and proper documentation'; this is a single peer-reviewed source, not a measured drift rate.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies METR 2025 RCT right 45%, partial (reviewed) lines 1015-1021METR 2025: developers report AI code leaves 'more to fix up, in terms of code conventions, code style' and needs major cleanup to meet repo standards (qualitative, 16 developers).Applicability: qualitative reports from 16 developers, not a measured drift rate against a human baseline
• Qualifies Debt study (302k AI commits) partly 30%, direct lines 672-674Correctness issues dominated by undefined variables (23,856) lead authors to say AI code 'may look locally correct, but still fail to stay consistent with the surrounding context'; this is consistency with surrounding code, not style conventions specifically.
• Qualifies Vibe-coding literature review partly 30%, direct lines 1434-1438WL27 is quoted saying conversationally generated code 'often lacks cohesive structure, consistent patterns, and proper documentation'; this is a single peer-reviewed source, not a measured drift rate.

Finding 1 (evidence) AI-generated code drifts from a codebase's established conventions

qualified qualify review qualified (qualify): no direct support, 3 qualifying.

METR 2025 RCT: METR 2025: developers report AI code leaves 'more to fix up, in terms of code conventions, code style' and needs major cleanup to meet repo standards (qualitative, 16 developers). [partial: qualitative reports from 16 developers, not a measured drift rate against a human baseline]

Debt study (302k AI commits): Correctness issues dominated by undefined variables (23,856) lead authors to say AI code 'may look locally correct, but still fail to stay consistent with the surrounding context'; this is consistency with surrounding code, not style conventions specifically.

Vibe-coding literature review: WL27 is quoted saying conversationally generated code 'often lacks cohesive structure, consistent patterns, and proper documentation'; this is a single peer-reviewed source, not a measured drift rate.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies METR 2025 RCT right 45%, partial (reviewed) lines 1015-1021METR 2025: developers report AI code leaves 'more to fix up, in terms of code conventions, code style' and needs major cleanup to meet repo standards (qualitative, 16 developers).Applicability: qualitative reports from 16 developers, not a measured drift rate against a human baseline
• Qualifies Debt study (302k AI commits) partly 30%, direct lines 672-674Correctness issues dominated by undefined variables (23,856) lead authors to say AI code 'may look locally correct, but still fail to stay consistent with the surrounding context'; this is consistency with surrounding code, not style conventions specifically.
• Qualifies Vibe-coding literature review partly 30%, direct lines 1434-1438WL27 is quoted saying conversationally generated code 'often lacks cohesive structure, consistent patterns, and proper documentation'; this is a single peer-reviewed source, not a measured drift rate.

Central conclusion Writing code was never the main bottleneck in software delivery, so faster code generation yields limited end-to-end gains

qualified qualify review qualified (qualify): no direct support, 2 qualifying.

Google migrations report: Google: code review and rollout still need humans and quickly become the bottleneck, though AI still cut end-to-end migration time by about 50% or more, so the gains were real even though capped. [partial: the verdict note limits its own scope ("though")]

Microsoft/Accenture field RCTs: Authors explain why field gains (26%) are smaller than lab gains: coding is only part of a developer's job and some tasks are less assistable, so time saved on coding only partly becomes extra output (interpretation, not directly measured). [partial: the verdict note limits its own scope ("not directly")]

"never" is stronger than the evidence (strongest support 0).

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Google migrations report partly 45%, partial lines 605-617Google: code review and rollout still need humans and quickly become the bottleneck, though AI still cut end-to-end migration time by about 50% or more, so the gains were real even though capped.Applicability: the verdict note limits its own scope ("though")
• Qualifies Microsoft/Accenture field RCTs partly 40%, partial lines 778-782Authors explain why field gains (26%) are smaller than lab gains: coding is only part of a developer's job and some tasks are less assistable, so time saved on coding only partly becomes extra output (interpretation, not directly measured).Applicability: the verdict note limits its own scope ("not directly")

Finding 1 (evidence) Writing code was never the main bottleneck in software delivery, so faster code generation yields limited end-to-end gains

qualified qualify review qualified (qualify): no direct support, 2 qualifying.

Google migrations report: Google: code review and rollout still need humans and quickly become the bottleneck, though AI still cut end-to-end migration time by about 50% or more, so the gains were real even though capped. [partial: the verdict note limits its own scope ("though")]

Microsoft/Accenture field RCTs: Authors explain why field gains (26%) are smaller than lab gains: coding is only part of a developer's job and some tasks are less assistable, so time saved on coding only partly becomes extra output (interpretation, not directly measured). [partial: the verdict note limits its own scope ("not directly")]

"never" is stronger than the evidence (strongest support 0).

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Google migrations report partly 45%, partial lines 605-617Google: code review and rollout still need humans and quickly become the bottleneck, though AI still cut end-to-end migration time by about 50% or more, so the gains were real even though capped.Applicability: the verdict note limits its own scope ("though")
• Qualifies Microsoft/Accenture field RCTs partly 40%, partial lines 778-782Authors explain why field gains (26%) are smaller than lab gains: coding is only part of a developer's job and some tasks are less assistable, so time saved on coding only partly becomes extra output (interpretation, not directly measured).Applicability: the verdict note limits its own scope ("not directly")

Central conclusion Specs-to-code workflows that ignore the generated code do not work; developers must keep understanding and shaping the code

qualified qualify review qualified (qualify): no direct support, 2 qualifying.

Vibe-coding literature review: The review recommends reviewing generated code before committing rather than integrating unread output, and warns that deferred understanding (cognitive debt) raises later maintenance costs; this is recommendation-level evidence. [partial: the verdict note limits its own scope ("rather than")]

Code-ownership experiment (Martin-Lopez): The authors warn that a loss of understanding may hurt review, debugging and maintenance. They also concede that future workflows handing those jobs to agents might make low-level comprehension matter less. So the paper backs the concern only speculatively.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Vibe-coding literature review partly 30%, partial lines 2076-2093The review recommends reviewing generated code before committing rather than integrating unread output, and warns that deferred understanding (cognitive debt) raises later maintenance costs; this is recommendation-level evidence.Applicability: the verdict note limits its own scope ("rather than")
• Qualifies Code-ownership experiment (Martin-Lopez) partly 30%, direct lines 917-940The authors warn that a loss of understanding may hurt review, debugging and maintenance. They also concede that future workflows handing those jobs to agents might make low-level comprehension matter less. So the paper backs the concern only speculatively.

Finding 1 (evidence) Specs-to-code workflows that ignore the generated code do not work; developers must keep understanding and shaping the code

qualified qualify review qualified (qualify): no direct support, 2 qualifying.

Vibe-coding literature review: The review recommends reviewing generated code before committing rather than integrating unread output, and warns that deferred understanding (cognitive debt) raises later maintenance costs; this is recommendation-level evidence. [partial: the verdict note limits its own scope ("rather than")]

Code-ownership experiment (Martin-Lopez): The authors warn that a loss of understanding may hurt review, debugging and maintenance. They also concede that future workflows handing those jobs to agents might make low-level comprehension matter less. So the paper backs the concern only speculatively.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Vibe-coding literature review partly 30%, partial lines 2076-2093The review recommends reviewing generated code before committing rather than integrating unread output, and warns that deferred understanding (cognitive debt) raises later maintenance costs; this is recommendation-level evidence.Applicability: the verdict note limits its own scope ("rather than")
• Qualifies Code-ownership experiment (Martin-Lopez) partly 30%, direct lines 917-940The authors warn that a loss of understanding may hurt review, debugging and maintenance. They also concede that future workflows handing those jobs to agents might make low-level comprehension matter less. So the paper backs the concern only speculatively.

Central conclusion Vibe-coded apps work for prototypes but break down when productionized and scaled

qualified qualify review qualified (qualify): no direct support, 1 qualifying.

Vibe-coding literature review: The 47-source review finds evidence strongest for prototyping and UI work and weakest for production. The break-down half is mostly unstudied, not shown. [partial: supports the prototype half; the production break-down half is mostly unstudied, not shown]

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Vibe-coding literature review partly 60%, partial (reviewed) lines 18-28The 47-source review finds evidence strongest for prototyping and UI work and weakest for production. The break-down half is mostly unstudied, not shown.Applicability: supports the prototype half; the production break-down half is mostly unstudied, not shown

Finding 1 (evidence) Vibe-coded apps work for prototypes but break down when productionized and scaled

qualified qualify review qualified (qualify): no direct support, 1 qualifying.

Vibe-coding literature review: The 47-source review finds evidence strongest for prototyping and UI work and weakest for production. The break-down half is mostly unstudied, not shown. [partial: supports the prototype half; the production break-down half is mostly unstudied, not shown]

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Vibe-coding literature review partly 60%, partial (reviewed) lines 18-28The 47-source review finds evidence strongest for prototyping and UI work and weakest for production. The break-down half is mostly unstudied, not shown.Applicability: supports the prototype half; the production break-down half is mostly unstudied, not shown

Central conclusion Goal or loop commands that re-prompt an agent until a goal is met let it work through a backlog unattended

qualified qualify review qualified (qualify): no direct support, 1 qualifying.

Airbnb migration: Airbnb's custom loop runner re-prompted the LLM with errors until a step passed or a retry cap was hit, and the first bulk run migrated 75% of 3.5K files in 4 hours; this is a bespoke pipeline, not a generic agent goal command. [partial: a bespoke retry pipeline for one migration, not a generic agent loop command on a backlog]

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Airbnb migration partly 45%, partial (reviewed) lines 19-22Airbnb's custom loop runner re-prompted the LLM with errors until a step passed or a retry cap was hit, and the first bulk run migrated 75% of 3.5K files in 4 hours; this is a bespoke pipeline, not a generic agent goal command.Applicability: a bespoke retry pipeline for one migration, not a generic agent loop command on a backlog

Finding 1 (evidence) Goal or loop commands that re-prompt an agent until a goal is met let it work through a backlog unattended

qualified qualify review qualified (qualify): no direct support, 1 qualifying.

Airbnb migration: Airbnb's custom loop runner re-prompted the LLM with errors until a step passed or a retry cap was hit, and the first bulk run migrated 75% of 3.5K files in 4 hours; this is a bespoke pipeline, not a generic agent goal command. [partial: a bespoke retry pipeline for one migration, not a generic agent loop command on a backlog]

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Airbnb migration partly 45%, partial (reviewed) lines 19-22Airbnb's custom loop runner re-prompted the LLM with errors until a step passed or a retry cap was hit, and the first bulk run migrated 75% of 3.5K files in 4 hours; this is a bespoke pipeline, not a generic agent goal command.Applicability: a bespoke retry pipeline for one migration, not a generic agent loop command on a backlog

Central conclusion In AI-assisted development the specification, not the code, should be the authoritative source of truth

qualified qualify review qualified (qualify): no direct support, 1 qualifying.

Vibe-coding literature review: The discussion says the developer's 'unit of work moves from the line of code toward the specification and the acceptance decision'; it also recommends reviewing code before committing, so it does not make the spec the sole source of truth.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Vibe-coding literature review partly 25%, direct lines 1924-1928The discussion says the developer's 'unit of work moves from the line of code toward the specification and the acceptance decision'; it also recommends reviewing code before committing, so it does not make the spec the sole source of truth.

Finding 1 (evidence) In AI-assisted development the specification, not the code, should be the authoritative source of truth

qualified qualify review qualified (qualify): no direct support, 1 qualifying.

Vibe-coding literature review: The discussion says the developer's 'unit of work moves from the line of code toward the specification and the acceptance decision'; it also recommends reviewing code before committing, so it does not make the spec the sole source of truth.

Suggestion: Add the qualification: the evidence supports a narrower or weaker statement.

• Qualifies Vibe-coding literature review partly 25%, direct lines 1924-1928The discussion says the developer's 'unit of work moves from the line of code toward the specification and the acceptance decision'; it also recommends reviewing code before committing, so it does not make the spec the sole source of truth.

Central conclusion 12 sources claim "AI code review tools catch real bugs that human reviewers miss"; no study in the corpus has checked it.

supported approved supported (approved): no direct support, 14 source statements.

Reports what the sources said, not whether it is true.

• Other Context Engineering & Coding Agents with Cursor sourceit actually caught a lot of bugs that we missed on code reviews
• Other Open Source security in spite of AI - Daniel Stenberg sourceI have fixed more than 100 bugs found by these tools when I started to use them
• Other AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) sourcehe found two bugs, critical bug line 223 where I put uh humidity instead of outside air quality
• Other Coding with LLMs in the summer of 2025 (an update) - <antirez> sourcemany were just removed immediately by Gemini / Claude code reviews.
• Other Control the ideas, not the code - <antirez> sourceFable and GPT 5.6 reviews to the sorted sets memory saving are going to spot ways more errors and subtle race conditions
• Other Claude Code Alternatives in 2026: Cursor, Qodo & GitHub Copilot Compared sourceIt says that there are open AI key exposed
• Other Using AI to write better code more slowly sourcethe latest public models from Anthropic and OpenAI are good enough to find plenty of bugs in an unscrutinized codebase.
• Other DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 sourcethe PRs that have been reviewed by agents caused far fewer issues in production
• Other Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo sourceI am 100% sure that AI can routinely find bugs before of users in complex C projects.
• Other Comparing all AI Pull Request Review tools to find the best one sourcethey are definitely here to stay. They were all useful to some extent
• Other Coding with AI episode 3: fixing a real world bug in Redis vector sets with LLMs sourceit immediately found my bug, which was awesome.
• Other Codex CLI: Everything You Need in 12 mins sourceCodex came back with actionable items, and also sorted by priorities
• Other My AI Coding Workflow That Actually Works sourceyou forgot about what happens if there's no audio
• Other Agentic Engineering in Action with Mitchell Hashimoto sourceoh, I actually forgot to update the code. Um, so it's found bugs that way.

Finding 1 (belief) 12 sources claim "AI code review tools catch real bugs that human reviewers miss"; no study in the corpus has checked it.

supported approved supported (approved): no direct support, 14 source statements.

Reports what the sources said, not whether it is true.

• Other Context Engineering & Coding Agents with Cursor sourceit actually caught a lot of bugs that we missed on code reviews
• Other Open Source security in spite of AI - Daniel Stenberg sourceI have fixed more than 100 bugs found by these tools when I started to use them
• Other AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) sourcehe found two bugs, critical bug line 223 where I put uh humidity instead of outside air quality
• Other Coding with LLMs in the summer of 2025 (an update) - <antirez> sourcemany were just removed immediately by Gemini / Claude code reviews.
• Other Control the ideas, not the code - <antirez> sourceFable and GPT 5.6 reviews to the sorted sets memory saving are going to spot ways more errors and subtle race conditions
• Other Claude Code Alternatives in 2026: Cursor, Qodo & GitHub Copilot Compared sourceIt says that there are open AI key exposed
• Other Using AI to write better code more slowly sourcethe latest public models from Anthropic and OpenAI are good enough to find plenty of bugs in an unscrutinized codebase.
• Other DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 sourcethe PRs that have been reviewed by agents caused far fewer issues in production
• Other Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo sourceI am 100% sure that AI can routinely find bugs before of users in complex C projects.
• Other Comparing all AI Pull Request Review tools to find the best one sourcethey are definitely here to stay. They were all useful to some extent
• Other Coding with AI episode 3: fixing a real world bug in Redis vector sets with LLMs sourceit immediately found my bug, which was awesome.
• Other Codex CLI: Everything You Need in 12 mins sourceCodex came back with actionable items, and also sorted by priorities
• Other My AI Coding Workflow That Actually Works sourceyou forgot about what happens if there's no audio
• Other Agentic Engineering in Action with Mitchell Hashimoto sourceoh, I actually forgot to update the code. Um, so it's found bugs that way.