AI tools and coding · 214 sources checked

Fast automated feedback loops (tests, type checks) set the ceiling on AI coding output quality

Leans right

Airbnb modelled its Enzyme->RTL migration as a state machine gated on Jest, ESLint and tsc, and retrying with the validation errors fed back fixed most simple-to-medium files within 10 attempts.

· denied 0 ·

Asserts (35)Denies (0)Verdicts (4)

Can Agentic AI Develop Embedded Systems with Model-Based Design? (MATLAB)“we were able to test early and often. We can catch errors at a point in the process where it's still easy to fix them.”
Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“the quality of your feedback loops influences how good your AI can code”
From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“we want to immediately give it as much feedback as possible so it can self-correct”
Building Blocks for Uber’s Software Factory— Uday Kiran Medisetty & Adam Huda, Uber (AI Engineer)“to build more advanced like endtoend features we really need to be able to validate uh the feature first”
Context Engineering & Coding Agents with Cursor (OpenAI)“the agent needs to be able to run the code, test it, and then verify it's actually working”
AWS re:Invent 2025 - Spec-driven development with Kiro (DEV314) (AWS Events)“the agent is able to read it ... ever since we've added this diagnostics tool, it's been really great”
Ralph Wiggum: Continuous Agent Loops in Claude Code (AI Makerspace)“tasks with automatic verification ... if we make it superverifiable”
Getting the most out of Codex (Academind)“you wanna give these tools ways of verifying their own work”
How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“it comes back to feedback loops ... tighten those feedback loops”
Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev) (The Pragmatic Engineer)“the faults of the model hallucination are avoided because the tool just runs itself and sees the results”
The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“constructing these like front-end feedback loops ... playwright ... storybook”
Context Engineering 101 - The Simple Strategy to 100x AI Coding (Cole Medin)“validation gates ... linting, and it'll create unit tests, and it'll run those unit tests and iterate”
Agentic coding changes everything | Thorsten Ball from Sourcegraph Amp (Changelog)“here's how you get feedback about your work. You know, here's the command you have to run to get llinter output or compiler errors”
What We Learned Deploying AI within Bloomberg’s Engineering Organization – Lei Zhang, Bloomberg (AI Engineer)“it would be really nice if there's some detistic verification capability”
Codex CLI: Everything You Need in 12 mins (lustoykov)“Codex opens the browser by itself and starts testing the application”
Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“Oh, if I have a better test harness for it to execute, it does a lot better.”
Why AI WON'T Replace Software Engineering... (Modern Software Engineering)“We need to know within minutes, not days, if AI-generated code broke something”
TDD, AI agents and coding with Kent Beck (The Pragmatic Engineer)“those tests can be run all the time to catch the genie accidentally accidentally breaking things.”
SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“I think it will have to involve these types of sensors to help the model correct”
FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“it's feedback loops to the system is very important.”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“writing code in a tool loop like where it compiles, runs, reads the error, does this. That's the thing that I care about”
My AI Skeptic Friends Are All Nuts (Fly)“Its strengths would have more to do with how you think about and structure builds and linting and test harnesses”
Test-driven development (TDD) with Claude Code (James Murdza)“take yourself out of the loop and to put tests, automated tests, there instead”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“we want it to take care of as much validation as possible”
A Year Of Vibes (Armin Ronacher)“The better the feedback loop to the machine, the better the results.”
Are Harnesses Worth the Cost? (martinfowler.com)“A behavioral harness turns written requirements into environmental feedback, bringing dropped details back into focus when a sensor goes red.”
TDD inside the agent loop - theater or actual value? (martinfowler.com)“That feedback should be automated wherever possible”
My AI Adoption Journey (Mitchell Hashimoto)“If you give an agent a way to verify its work, it more often than not fixes its own mistakes and prevents regressions.”
Learn the basics of Google Antigravity (Google Antigravity)“it can actually test the code that it just wrote all on its own”
The AI-Native Data Engineer featuring Fabric, dbt, VSCode devcontainers and copilot (Microsoft Reactor)“if you do not have test coverage and you're throwing LLMs at writing more code, at some point it's going to become sloppy”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“I think one critical observation about agents is they thrive on feedback, right?”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“so that we can trigger immediately a self-correction loop before we even look at the code”
DeepSeek R1 vs Kimi K2 vs Qwen3 Coder vs GLM 4.5 - Head-to-Head Testing! (Bijan Bowen)“All I did was paste in the actual error message. And for three of the four here they went ahead and just immediately fixed it”
Beyond the Hype: Real Talk on AI-Assisted Development • Jessica Kerr & Diana Montalion • GOTO 2025 (GOTO Conferences)“It's also great at HTML and CSS because it can run Playwright.”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“instruct it specifically to make uh a testing suite for you and then check that testing suite”
Very likely right AI guessAgent benchmarks and practice consistently show verifiable feedback (tests, compilers) is the main driver of agent success.
Leans right Google migrations report lines 316-327Google: every LLM edit goes through a build and unit-test loop with ML-powered repair of failures, and validation across several steps can 'recover' imperfect first attempts; this is the core of their migration pipeline.
Leans right Airbnb migration lines 14-22Airbnb modelled its Enzyme->RTL migration as a state machine gated on Jest, ESLint and tsc, and retrying with the validation errors fed back fixed most simple-to-medium files within 10 attempts.
Leans partly right Vibe-coding literature review lines 1213-122572% of 43 workflow sources put validation pipelines (tests, lint, runtime checks) at the core of the loop, and the review concludes quality 'seems to depend on the strength of the evaluation loop'; this is a frequency of mentions, and workflow comparisons are limited.

Developers need solid programming knowledge to validate and correct AI-generated code

Leans right

The review concludes that skill in verification, constraint specification and review may matter as much as authoring skill, and that the central risk is a verification gap.

· denied 0 ·

Asserts (22)Denies (0)Verdicts (6)

Can Agentic AI Develop Embedded Systems with Model-Based Design? (MATLAB)“this is something that Naren and I were able to verify was correct through our own expertise and understanding of the problem”
Aider Review — The Best AI Pair-Programmer in Your Terminal? (2025) (HMD Tech)“You still need technical experience to review patches”
AI Replaced 75% of Work for Software Engineers & Data Scientists - here's what's left (Sundas Khalid)“in order to validate what AI is spitting out, you actually need to know if the code ... is correct”
AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“you definitely need an experienced software engineer in the back uh to be able to to create good code”
Jensen Huang Said "Stop Learning to Code" — Then Hired 10,000 Developers (Olive Badger)“learn to code. Learn the fundamentals”
SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker (Hamel Husain)“any time an AI shows you a function library or an API you're not very familiar with you should stop”
Unpacking METR’s findings: Does AI slow developers down? (DX)“what good code looks like makes it easier for me to realize when a model is producing, you know, bad code, bloated code”
The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“I who know those fundamentals can begin nudging and cleaning out the pieces”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“how it can be risky to allow people new to teams to start using AI from day one without having the foundational knowledge”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“they still need the fundamentals because all the code that's being produced they need to be able to critique it properly”
Agentic Engineering in Action with Mitchell Hashimoto (Zed Industries)“still challenge yourself to understand how that works even if you're not going to write it”
AI Layoffs Don’t Add Up. Here’s the Data (HuMan Intelligence)“can you verify AI output? Can you apply domain expertise to catch when it's wrong?”
Control the ideas, not the code - <antirez> (antirez.com)“You have to understand how things work, what is the best design, how to reach a certain level of performance.”
Will AI replace Software Engineers ? Raw & Honest Opinion (Apporva arya)“we have to go through it and review it; we cannot follow everything blindly”
Does GitHub Copilot Improve Code Quality? Here (Jadarma)“if you can't write good code without an AI, then you shouldn't use one in the first place.”
The Law of Leaky Abstractions (Joel on Software)“the only way to deal with the leaks competently is to learn about how the abstractions work and what they are abstracting”
Agentic Coding is a Trap | Lars Faye (larsfaye.com)“only a skilled developer who's thinking critically, and comfortable operating at the architectural level, can spot issues in the thousands of lines of generated code”
An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“agents work best when you have approximate knowledge of many things with enough domain expertise to know what should and should not work”
A look at CloudFlare’s AI-coded OAuth library (Neil Madden)“What this interaction shows is how much knowledge you need to bring when you interact with an LLM.”
How much does AI impact development speed? (newsletter.getdx.com)“developers have to carefully check and fix the code the AI suggests”
The AI-Native Data Engineer featuring Fabric, dbt, VSCode devcontainers and copilot (Microsoft Reactor)“First, you as the data engineer have to become competent of what you're trying to do.”
Vibe code is legacy code (Val Town)“They're going to have to learn to use their human eyes to read the code”
Very likely right AI guessStudies show novices accept incorrect AI code more often; verifying output requires domain and code literacy.
Leans right Anthropic internal study lines 93-94Anthropic: the 'paradox of supervision' (supervising Claude needs the coding skills overuse may erode), and a security engineer saw a 'dangerously smart' proposal that only experienced judgment could catch.
Leans right Vibe-coding literature review lines 1929-1932The review concludes that skill in verification, constraint specification and review may matter as much as authoring skill, and that the central risk is a verification gap.
Leans partly right METR 2025 RCT lines 1015-1021METR 2025: 75% of experienced developers read every line of AI code, 56% often make major cleanups, and 100% report needing to modify it; this shows correction is routinely needed, but the study does not test less-skilled users.
Leans right Google RCT lines 484-493Google RCT authors suggest heavier coders gain more from AI because current tools require spending much time verifying and editing generated code (interaction beta=-0.29, not significant), so coding skill conditions the benefit.
Leans right Echoes of AI maintainability RCT lines 2549-2556The Task 1 developer's Java proficiency influenced downstream outcomes more than AI use, and authors conclude human expertise still matters; qualitative quotes describe using experience to catch hacky AI suggestions.

Developers overestimate how much AI tools speed them up

Very likely right

Developers forecast a 24% speedup and afterward estimated 20%, but were actually slowed 19%. Experts also overestimated, predicting 38-39% faster.

· ·

Asserts (21)Denies (1)Verdicts (5)

Stanford Study: AI Makes Developers 20% More Productive (But There's a Catch) (Josh Anderson)“people telling you estimating how their productivity was affected by AI is effectively a coin flip”
Measuring AI code assistants and agents with the AI Measurement Framework (DX)“developers uh self- reports of the fact that they were faster with AI than without it were were actually wrong in in many cases”
AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“be careful with using surveys ... to estimate how much more productive AI makes you”
SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker (Hamel Husain)“That's not what happened I haven't actually sped up”
Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev) (The Pragmatic Engineer)“estimations are maybe 3 to 5 hours, maybe 4 hours. I mean, okay, 4 hours saved is pretty good, but that's not 10x”
AI made Developers Slower (Telusko)“after the study they thought they it has improved their performance by 20%”
Unpacking METR’s findings: Does AI slow developers down? (DX)“I think everyone um estimates speed up because it's just it's so much fun to use AI.”
Does AI-driven productivity come at the cost of code quality? (Proxify)“for the first two months you might perceive that you're moving faster but in reality”
Why Agent Hype can fall short of reality – Joel Becker, METR (AI Engineer)“they think that it will have sped them up by something like 20%”
Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“they say that AI made them complete tasks in 20% less time.”
The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“because we produce a lot of output very fast, we are tricked into thinking that we're actually being more efficient”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“retradicting how much they were sped up in the past on the issues in this study predict that they went 20% faster”
The METR Paradox. Why AI Made Senior Devs 19% Slower (Macro Lens)“those same developers still believed AI had made them 20% faster”
Is AI making devs SLOWER? Yes, according to this study... (Native Notify)“they felt like they were finishing their work 20% faster”
How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR (AI Engineer)“developers will tell you that they were faster when they weren't.”
Where's the Shovelware? Why AI Coding Claims Don't Add Up (mikelovesrobots.substack.com)“developers were unreliable narrators of their own productivity”
METR's AI productivity study is really good (seangoedecke.com)“developers came away from the study thinking they were 20% faster, but were in fact 19% slower.”
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Simon Willison’s Weblog)“developers estimate that allowing AI reduced completion time by 20%.”
Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford (AI Engineer)“asking someone how productive they think they are is almost as good as flipping a coin”
Does AI Secretly Slow Developers Down? (The AI Daily Brief: Artificial Intelligence News)“even after the study concluded thought that AI had made them 20% more productive”
AI Made Devs 19% Slower (METR 2025 Study) (zencoderai)“After using it, they said they felt about 20% faster”
DENIES Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“people under report significantly the help that they get from AI because they don't want to lose their job.”
Very likely right METR 2025 RCT lines 17-24Developers forecast a 24% speedup and afterward estimated 20%, but were actually slowed 19%. Experts also overestimated, predicting 38-39% faster.
Probably right AI guessMETR found devs believed they were ~20% faster while measured slower; other studies also show self-reports outrunning measured gains.
Leans wrong Copilot RCT (Peng) lines 184-188In the Copilot RCT both groups estimated a ~35% productivity gain, an underestimate of the measured 55.8% speedup.
Leans partly right METR 2026 update lines 63-63METR Feb 2026: some developers self-report very high speedups, which METR notes can be 'quite unreliable' per its earlier study; it also says the true speedup may be higher among those who selected out, so no new measured overestimate.
Unknown Anthropic internal study lines 36-36Anthropic's +50% figures are self-reported; the post cites METR's overestimation finding and admits perception, social-desirability and recall biases, so it cannot settle the question.

Investing upfront in comprehensive written context (requirements, examples, references) yields much better AI coding results than ad-hoc prompting

Leans right

Airbnb: complex files only succeeded once prompts grew to 40-100K tokens with up to 50 related files, few-shot examples and migration guidelines, which proved 'highly effective'.

· ·

Asserts (22)Denies (1)Verdicts (3)

Can Agentic AI Develop Embedded Systems with Model-Based Design? (MATLAB)“if you don't think about exactly what you want in quantifiable and verifiable ways, then you don't know how to direct the agent”
Ralph Wiggum: Continuous Agent Loops in Claude Code (AI Makerspace)“if I want Claude to work for for 10 hours on something, I best have put 10 hours into planning”
Master 80% of Google Antigravity in 27 Minutes (Ali H. Salem)“It will perform much better if you plug it into a markdown file and let it use that”
The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“a lot of the power users were very writing very long prompts”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“planning before you prompt and it's providing that clear context”
Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro (AI Engineer)“you give us a prompt and we will take that and turn it into a set of clear requirements with acceptance criteria”
Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“spend a bit more time to be more concrete in your prompts which increases the probability of successful verification”
Context Engineering 101 - The Simple Strategy to 100x AI Coding (Cole Medin)“providing all information, examples, best practices, constraints to AI coding assistants upfront”
Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“Another thing that I do, which I understand many developers don't do, is to put a lot of context inside the AI.”
Ralph Wiggum killed programming (Ras Mic)“your plan matters a lot. How detailed, how precise ... will dictate the level of quality”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“the agent has to make a lot of assumptions ... That's why we want to first build the specification”
Inside minions: how Stripe uses autonomous one-shot AI coding agents (Stripe Developers)“we want the agent to be given enough context to accomplish this task in one shot”
Coding with LLMs in the summer of 2025 (an update) - <antirez> (antirez.com)“you need to provide extensive information to the LLM”
FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“So, I would first draft out a very detailed spec”
Spec Kit: Github's NEW tool That FINALLY Fixes AI Coding (Better Stack)“Specdriven development eliminates that guesswork. It gives AI clear structured guidance”
An SDD Focused IDE? (BMad Code)“learning how to use the AI agents to produce documentation and context to give to the developer agents”
Are Harnesses Worth the Cost? (martinfowler.com)“Invest in clear specifications and examples: The checks can enforce only the expectations humans encode.”
How far can we push AI autonomy in code generation? (martinfowler.com)“turned out to be the most effective strategy to get AI to generate the type of code we wanted.”
An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“there are always functional requirements that the agent won't magically apply because it cannot read minds and behaves as a literal genie”
What Is Spec-Driven Development - Kiro · Spec Kit · BMAD | Complete Visual Guide #2 (ByteLearn)“They all expect these three exact pieces of information before they generate a single line of code”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“A lot of people skip this entire thing ... and that's why they get poor results”
AI Made Devs 19% Slower (METR 2025 Study) (zencoderai)“before you prompt AI to build a function, you need to spec it out”
DENIES DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“in the agentic age, you should resist the temptation to be overly specific upfront. Be as vague as you can to manifest something”
Probably right AI guessClear requirements, examples and references reliably improve agent output; the caveat is that bloated context can hurt, so curation matters.
Leans right Airbnb migration lines 23-44Airbnb: complex files only succeeded once prompts grew to 40-100K tokens with up to 50 related files, few-shot examples and migration guidelines, which proved 'highly effective'.
Leans partly right Vibe-coding literature review lines 1064-1070Two practitioner grey sources say fuller context (requirements docs, examples, API docs, constraints) reduces ambiguity, and 40% of RQ2 sources report explicit context provisioning, but the review finds no comparison against ad-hoc prompting.

Writing code was never the main bottleneck in software delivery, so faster code generation yields limited end-to-end gains

Leans right

Studies of developer time show coding is a minority of work; DORA and Faros 2025 found individual coding gains often don't translate into delivery throughput.

· ·

Asserts (18)Denies (3)Verdicts (3)

2025 DORA report: Insights to impact with Google and Thoughtworks (Thoughtworks)“if our bottleneck is something like the code review process, simply generating more code isn't actually helping”
DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“As soon as you're having human teams work together on something, the bottleneck is rarely implementation. It's human bandwidth and communication.”
The AI Productivity Paradox: Why 10X Output Doesn't Mean 10X Business Outcome - Mik Kersten (Tech Lead Journal)“we saw that only 8% of the end-to-end time spent in delivering value through”
DORA Community Discussion - State of AI-assisted Software Development (DORA)“instead of using AI to say generate more code that just piles up and piles up due to a bottleneck like code review”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“people really only spend a small percentage of every day writing code. So just investing in one specific area, it's not going to”
AI in engineering: Q2 2026 benchmarks & research readout (DX)“so many of the savings those six hours get absorbed get eclipsed by what we know are other sources of time syncs”
Why AI WON'T Replace Software Engineering... (Modern Software Engineering)“The bottleneck in software was never the typing”
The 70% Problem - Agentic Engineering Glossary | AddyOsmani.com (AddyOsmani.com)“the hard part of software engineering was never typing the code”
No, AI is not Making Engineers 10x as Productive (colton.dev)“For that to happen each and every one of these bottlenecks has to also seen have 10x productivity gains.”
Real artists still ship (Jerod Santo)“Do I code faster than ever? Absolutely. But am I shipping more than ever? I can't say that I am”
AI Will Replace Software Engineers? Here’s What’s Actually Happening (Chris Schwenk | Tech Jobber)“The actual coding was never the real job”
Agentic Coding is a Trap | Lars Faye (larsfaye.com)“Contrary to the current narrative that is being espoused, we didn’t necessarily need to write code faster.”
Sustainable Augmented Development • Kent Beck • YOW! 2025 (GOTO Conferences)“they speed up coding by some measure by a factor of 10 and they get no more value out the end”
AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“but that's only 10% of my job”
Developer experience in the AI era (Develocity)“they're still not tackling a lot of the other problems that exist across the developer experience.”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“organizationally how fast we can create code has never been the bottleneck. It's been everything around it.”
Measuring the actual impact of AI coding | Abi Noda from DX (Changelog)“even with a pretty high acceleration of the coding part of the job. you you're kind of limited by these other factors”
AI Made Devs 19% Slower (METR 2025 Study) (zencoderai)“writing code is less than a third of a developer's day”
DENIES Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“people need to stop talking about code being the bottleneck and typing typing was never the bottleneck ... they're misdirecting the conversation”
DENIES </> htmx ~ Code is Cheap(er) (htmx.org)“While I appreciate the sentiment, I don’t completely agree with that: certainly coding was at least part of the problem.”
DENIES Where's the Shovelware? Why AI Coding Claims Don't Add Up (mikelovesrobots.substack.com)“That’s only true when you’re in a large corporation.”
Probably right AI guessStudies of developer time show coding is a minority of work; DORA and Faros 2025 found individual coding gains often don't translate into delivery throughput.
Leans partly right Google migrations report lines 605-617Google: code review and rollout still need humans and quickly become the bottleneck, though AI still cut end-to-end migration time by about 50% or more, so the gains were real even though capped.
Leans partly right Microsoft/Accenture field RCTs lines 778-782Authors explain why field gains (26%) are smaller than lab gains: coding is only part of a developer's job and some tasks are less assistable, so time saved on coding only partly becomes extra output (interpretation, not directly measured).

AI-assisted changes should be integrated in small, verifiable increments rather than large generated leaps

Leans right

Small batches with fast feedback are a strong DORA predictor of delivery performance; DORA 2024 linked AI to larger batches and lower stability.

· denied 0 ·

Asserts (21)Denies (0)Verdicts (1)

Can Agentic AI Develop Embedded Systems with Model-Based Design? (MATLAB)“this was a collaborative iterative process. We guided the agent step by step. We reviewed what it produced”
Stanford Study: AI Makes Developers 20% More Productive (But There's a Catch) (Josh Anderson)“if you can break your work down into smaller digestible chunks, it's easier for you to do that work”
How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“building things in terms of small slices with the humans reviewing it is still the way to bet”
SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker (Hamel Husain)“doing this a lot over the last year that it also makes the LLMs much more intelligent and works with us much better”
Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“I always go in small incremental chunks”
Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“you have to decompose tasks into the smallest possible conceptual move forward”
Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“I'm reviewing each chunk of the diff as it builds it up and then I'll review it at the end”
AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“we got to find ways to keep organize our work into these smaller steps and verify after very quickly after each step”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“it's about small reviewable diffs that um humans can review”
Why AI WON'T Replace Software Engineering... (Modern Software Engineering)“Good engineering is about small, reversible steps with fast feedback”
Using AI Code Generation to Migrate 20000 Tests (InfoQ)“the best u result that we got from our tools was when when the scope is so minimal uh the smaller the better.”
How To Solve It With Code—background on the course; a discussion with Jeremy and Hamel (Jeremy Howard)“when you build the code up slowly you end up much faster than the person who does it 200 at a time”
Spec Kit: Github's NEW tool That FINALLY Fixes AI Coding (Better Stack)“You can review each change before implementation instead of running bulky code dumps”
</> htmx ~ Code is Cheap(er) (htmx.org)“This is why I recommend incremental use of LLMs rather than allowing them to generate massive changelists”
Is AI making devs SLOWER? Yes, according to this study... (Native Notify)“cut your project into smaller pieces”
o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“the limitation on how much change I can let it make at once is how much code I'm willing to look at”
Agentic Coding is a Trap | Lars Faye (larsfaye.com)“I never generate more than I can review in a sitting.”
Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“stop and check every thousand lines or so”
How AI Is Reshaping Software Development Insights from the 2025 DORA Report | AppDevANGLE (SiliconANGLE theCUBE)“We want AI to generate that code in smaller segments so that we have an opportunity to review that code”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“think about chunking those into smaller poll requests”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“Bigger batch sizes are riskier. >> So we want to keep batch size small.”
Probably right AI guessSmall batches with fast feedback are a strong DORA predictor of delivery performance; DORA 2024 linked AI to larger batches and lower stability.

LLM output quality degrades as the context window fills, so tasks should be sized to stay in a small 'smart zone'

Leans right

Long-context benchmarks (e.g. NoLiMa, context-rot studies) show degradation as context grows; the exact threshold is model-dependent.

· denied 0 ·

Asserts (20)Denies (0)Verdicts (2)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“as you continually keep adding stuff to the same context window, it just gets dumber and dumber”
Context Engineering & Coding Agents with Cursor (OpenAI)“Models get worse at recalling information as the size of the context increases”
Stanford Study: AI Makes Developers 20% More Productive (But There's a Catch) (Josh Anderson)“It can handle more, but it quickly drops off at how effective it can be”
AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“first he will have his mind polluted by the first code and second it will cost three times more”
AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“by the time you were at 32,000 tokens, this model was already at like 50% of its performance”
The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“Sonnet 4 has a context window of 200K, but we see degradation typically around like 120 or 130K”
Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“it will fill the entire context window with that task ... and start making panicked decisions”
Unpacking METR’s findings: Does AI slow developers down? (DX)“models start to get overwhelmed with irrelevant information and they start getting confused.”
Building the Context Engine AI Agents Need (InfoQ)“if you go over 40% of the context window, you enter the dumb zone”
Context Engineering SF: Advanced Context Engineering for Agents - Dex Horthy (Chroma)“our goal all the time is to keep context utilization under 40%”
Why AI WON'T Replace Software Engineering... (Modern Software Engineering)“as that context window fills up, the AI will forget things and start to underperform”
How Agents Use Context Engineering (LangChain)“performance degrades with respect to context length”
vulnerability research just got easier (scarier?) (Low Level)“03 finds the Keraros off bug one out of 100 times with this larger number of input tokens.”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“Just because you can fit a million tokens into a large language model does not mean that you should”
Are Harnesses Worth the Cost? (martinfowler.com)“Juggling library choices, boilerplate setup, dependency injection, and asynchronous wiring across a growing context may contribute to such lapses”
The AI-Native Data Engineer featuring Fabric, dbt, VSCode devcontainers and copilot (Microsoft Reactor)“by the time it's done learning what you built, you've already filled up your context window”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“it comes with the trade off on like how well the models are able to keep attention on all of the many instructions”
Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford (AI Engineer)“as context length increases from 1,000 to 32,000 tokens performance decreases”
A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“how much can you throw into an LLM before it falls apart? Um not a lot it turns out”
"Do You Still Read the Code?" • zanlib (zanlib)“agents with polluted context windows drifting into the “dumb zone” will start making obvious local mistakes”
Very likely right AI guessLong-context benchmarks (e.g. NoLiMa, context-rot studies) show degradation as context grows; the exact threshold is model-dependent.
Leans partly right Vibe-coding literature review lines 1319-1327Vibe-coding MLR (47 sources): context-window/state-retention failures and context drift over long multi-turn sessions are reported, but only by 2 sources and labeled 'emerging'; no 'smart zone' sizing evidence.

AI-generated changes should pass a human review gate before merging

Probably right

The paper recommends careful review of AI code whatever the tool, along with static analysis, tests, security checks, and monitoring after merge.

· ·

Asserts (18)Denies (1)Verdicts (4)

AWS re:Invent 2025 - Spec-driven development with Kiro (DEV314) (AWS Events)“you have to read it and pay attention to what it's doing. Because otherwise it will do things that we don't expect”
Measuring AI code assistants and agents with the AI Measurement Framework (DX)“the key would be to have that code go through the same code reviews, the same gating quality processes as any other code”
Getting the most out of Codex (Academind)“You're responsible for your code. It's not the AI, it's you”
How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“treat it like a PR from a rather dodgy collaborator”
Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“you're responsible for the outcomes and you can't ever blame the AI for anything”
How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“we can't just put in a command and get something back and accept it we really need to evaluate it”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“you're holding that code to the same bar that you would hold a human to”
Inside minions: how Stripe uses autonomous one-shot AI coding agents (Stripe Developers)“read the code myself to make sure it's a sensible change”
My AI Skeptic Friends Are All Nuts (Fly)“If you build something with an LLM that people will depend on, read the code.”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“for any like serious production coding that I'm doing, I still am reviewing all of the code myself”
A Year Of Vibes (Armin Ronacher)“I still treat it like regular software engineering and review a lot.”
How Google is accelerating code migrations with AI (newsletter.getdx.com)“Only changes that pass all checks are submitted for review.”
The AI-Native Data Engineer featuring Fabric, dbt, VSCode devcontainers and copilot (Microsoft Reactor)“I'm going to be the one that writes the the commit. I'm going to be the one that pushes it.”
Learning to Code with AI and Steve Klabnik (Scott Hanselman)“I do review every PR that I merge. I think it's like professionally responsible, especially like for work stuff.”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“code review is maybe like the most important thing that a human can do in the context of software engineering”
576 - Using LLMs at Oxide | RFD | Oxide (Oxide Computer Company)“but that review should not be accepted as a human substitute.”
Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“do not send anyone code for review that you yourself have not read”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“I don't trust AI enough to just manually rubber stamp this stuff. So important to have that manual gate”
DENIES AI Dev 26 x SF: Andrew Ng: The Future of Software Engineering (DeepLearningAI)“if I got to review the code, then I become the bottleneck, and it also doesn't work”
Probably right AI guessGiven documented defect and security rates in AI code, human review before merge is standard practice (e.g. Amazon's two-person rule for AI-assisted changes); it's a sound norm.
Probably right Debt study (302k AI commits) lines 934-945The paper recommends careful review of AI code whatever the tool, along with static analysis, tests, security checks, and monitoring after merge.
Leans right Google migrations report lines 226-242In Google's practice humans review LLM-generated code like any other code, and in the int64 migration humans had to revert some incorrect or unnecessary model changes in most cases.
Leans partly right Vibe-coding literature review lines 1636-1662Human-in-the-loop review is recommended by 48% of 33 safeguard sources, but the review says few studies show whether such safeguards actually reduce risk, so they are 'recommended rather than proven'.

AI coding productivity gains appear only after a learning period of sustained use

Leans partly right

Learning curves are real (METR's one dev with >50h Cursor experience was faster), but evidence that ramp-up reliably flips results is thin.

· ·

Asserts (12)Denies (6)Verdicts (6)

Reconsidering AI w / Steve Klabnik (Zed Industries)“it is also true there's a learning curve”
How AI is changing Software Engineering: A Conversation with Gergely Orosz, @pragmaticengineer (AI Engineer)“it just takes a long time to get good at it and you need to keep doing it”
The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“I think it is a high ceiling skill”
AI made Developers Slower (Telusko)“it slows you down when you just started work for one day but overall it improves it”
Agentic coding changes everything | Thorsten Ball from Sourcegraph Amp (Changelog)“when have you ever become more productive with a tool after using it for just a single day? Like you have to put some effort”
Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“it takes much longer than an hour to get proficient with git, but you put in the effort and then you reap the rewards later.”
Agentic Engineering in Action with Mitchell Hashimoto (Zed Industries)“what tool have you ever adopted where you were immediately faster.”
My AI Adoption Journey (Mitchell Hashimoto)“I forced myself to reproduce all my manual commits with agentic ones.”
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Simon Willison’s Weblog)“getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect.”
Does AI Secretly Slow Developers Down? (The AI Daily Brief: Artificial Intelligence News)“these productivity gains are not free. They come with a learning curve”
AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“you'll get this tool and it'll make you productive on day one and it absolutely won't”
Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“you actually have to spend 200 hours with it. You have to spend 2,000 hours with it”
DENIES AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“you you'll get I think days”
DENIES Why Agent Hype can fall short of reality – Joel Becker, METR (AI Engineer)“maybe the developers really are not very capable at using AI ... I'm a little skeptical of that story”
DENIES How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR (AI Engineer)“how much is J curve explaining? I think it's not explaining that much.”
DENIES Where's the Shovelware? Why AI Coding Claims Don't Add Up (mikelovesrobots.substack.com)“There are no indicators that prompting is hard to learn.”
DENIES How much does AI impact development speed? (newsletter.getdx.com)“developers who had frequently used AI tools in the past didn’t complete tasks significantly faster”
DENIES METR's AI productivity study is really good (seangoedecke.com)“Developers didn’t get faster with AI over the course of the experiment”
Leans partly right AI guessLearning curves are real (METR's one dev with >50h Cursor experience was faster), but evidence that ramp-up reliably flips results is thin.
Leans partly right METR 2025 RCT lines 1205-1230There was no improvement across the first 50 hours of Cursor use, and dropping early tasks didn't change the slowdown. One developer with more than 50 hours did speed up, so a high skill ceiling is plausible but underpowered.
Leans wrong Shan et al. field study + RCT lines 1238-1242Gains appeared on first use: in the follow-up, after removing participants with prior GenAI coding experience, the main quantity and quality effects still held (brief, simple tasks).
Leans partly right Echoes of AI maintainability RCT lines 2908-2913Habitual AI users had a 55.9% posterior speedup vs 30.7% median across all AI users, suggesting proficiency matters, but the gains for non-habitual users were still positive and habitual status was self-selected.
Leans wrong Google RCT lines 522-532Google RCT found no significant evidence that more frequent prior AI tool use increased speed with AI (H5 rejected); authors suggest a low usability floor where benefits accrue quickly, though power was limited.
Unknown METR 2026 update lines 61-62METR Feb 2026: the same 10 developers went from 19% slower (early 2025) to ~18% faster (late 2025, wide CI), but the update cannot separate more experience from better tools and selection effects.

Classic software engineering fundamentals matter as much or more when working with AI agents

Leans right

DORA 2025 found AI amplifies existing engineering practices; strong fundamentals predict AI benefit.

· denied 0 ·

Asserts (17)Denies (0)Verdicts (1)

Can Agentic AI Develop Embedded Systems with Model-Based Design? (MATLAB)“why following proven engineering workflows matter even more when AI is involved”
Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“software engineering fundamentals ... also works super well with AI”
2025 DORA report: Insights to impact with Google and Thoughtworks (Thoughtworks)“AI's primary role in software development is that of an amplifier”
Nicole Forsgren: Leading high-performing engineering teams in the age of AI - The Pragmatic Summit (The Pragmatic Engineer)“what's good for humans is good for systems. If I have well-structured code, if I have well-structured documentation”
SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker (Hamel Husain)“There is no methodology that works with bad coding”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“AI is a multiplier. So, what helps the human can help the AI. So, invest in tests, CI, clear documentation”
The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“focus upon the fundamentals because those skills are never going to go away”
AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“there are 10 foundational principles and I don't think they change one jot”
AI-Generated Code Is Creating a New Kind of Technical Debt (Imran Gardezi)“AI makes disciplined teams unstoppable. AI makes undisciplined teams dangerous”
Why AI WON'T Replace Software Engineering... (Modern Software Engineering)“AI doesn't replace the need for software engineering, it exposes teams that were never really doing engineering”
SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“it turns out that AI needs a lot of the same things that we need to make safe changes to stuff”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“none of that is different just because you have an an AI”
The Law of Leaky Abstractions (Joel on Software)“All high level tools compared to the old K&R stuff, but I still have to know the K&R stuff or I’m toast.”
How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt (Margaret-Anne Storey)“developers need to slow down and use practices such as pair programming, refactoring, and test-driven development to address technical debt AND cognitive debt.”
AI Isn't Replacing Developers - Here's Why (Stefan Mischook)“Design patterns are hugely important in the AI age of software development”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“a lot of the things that have always been important have become some mix of more and differently important in the world of agentic programming.”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“even if you take the LLM out of this, this is just a good workflow for developing software”
Probably right AI guessDORA 2025 found AI amplifies existing engineering practices; strong fundamentals predict AI benefit.

Having the agent plan and research before implementing improves the code it produces

Leans right

Plan-then-execute consistently reduces agent rework in practice and in vendor evals, especially on multi-file tasks.

· denied 0 ·

Asserts (17)Denies (0)Verdicts (2)

Context Engineering & Coding Agents with Cursor (OpenAI)“significantly improves the quality of the code generated”
AWS re:Invent 2025 - Spec-driven development with Kiro (DEV314) (AWS Events)“planning upfront really helped the models focus and produce better output”
Getting the most out of Codex (Academind)“planning still matters ... I'm using that all the time”
Master 80% of Google Antigravity in 27 Minutes (Ali H. Salem)“I prefer to always have the human in the loop approach. So, I have the chance to adjust it before it builds”
Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“research ... to figure out how to implement this the right way not the wipe coded way”
Context Engineering 101 - The Simple Strategy to 100x AI Coding (Cole Medin)“it does quite a bit of research and architecting and planning”
Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“please spit out a step-by-step implementation plan including sample prompts and then I take that and then I hand it off to an agentic tool”
How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“because they were so systematic about it up front, they're much closer to something that looks like production code”
Context Engineering SF: Advanced Context Engineering for Agents - Dex Horthy (Chroma)“we have three phases research, plan and implement”
Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“Oh, if I do a separate planning step, it does so much better.”
Inside minions: how Stripe uses autonomous one-shot AI coding agents (Stripe Developers)“I'm just going to ask an agent first before we kick off the one shot, just to gather some context up front.”
Enterprise Vibe Coding: Building Secure, AI-Powered Apps in 30 Minutes - Parth Sharma (Developer Summit)“Mentor is first going to actually give you a blueprint that you can iterate on”
12 Months of Unity Dev with Claude Code (Mythmatic)“It just helps Claude pause at the end and give me an implementation plan just get final approval on”
TDD inside the agent loop - theater or actual value? (martinfowler.com)“the non-TDD and test-first runs always created the full design (architecture, data types, edge cases, contracts) before writing any code or tests”
My AI Adoption Journey (Mitchell Hashimoto)“For vague requests, split the work into separate planning vs. execution sessions.”
Learn the basics of Google Antigravity (Google Antigravity)“an implementation plan ... prior to actually going off and making changes”
Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“research, plan, then implement”
Probably right AI guessPlan-then-execute consistently reduces agent rework in practice and in vendor evals, especially on multi-file tasks.
Leans partly right Google migrations report lines 619-626Google migration team found LLM planning capabilities 'often not needed' and a complexity to avoid, favoring agentless localize-edit-validate steps, though this is only for well-defined migrations. (Narrow setting, so it limits the claim rather than refuting it.)

Heavy reliance on AI coding assistance erodes developers' own skills and learning

Leans partly right

Anthropic's 2026 RCT found AI-assisted learners scored lower on mastery of a new library, and other studies show reduced engagement; effects depend on how AI is used.

· ·

Asserts (16)Denies (1)Verdicts (7)

How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“when you stop paying attention to the code generated you stop learning and then you stop understanding”
SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker (Hamel Husain)“relying on tab completion and stuff it's damaging our ability to do high quality work over an extended period of time”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“Our first priority is really mitigating skill erosion”
ICM 2026 Public Lectures - Terence Tao (Simons Foundation)“it can be harmful to to to give students too much AI use early on.”
AI Coders Scored 17% Lower—Here's What They Did Wrong (DIY Smart Code)“Developers using AI scored 17% lower on skill tests”
Your Brain on ChatGPT: Accumulation of Cognitive Debt (June 2025) (AI Paper Slop)“reliance on AI for core thinking tasks might lead to those brain circuits being well less developed”
Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“it can end up retarding your development and preventing you from growing and actually getting stuff done.”
Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“I think if you use the tools wrong, you do think less”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“just to take over your decision-m process, then that part of your brain probably will not be trained”
Will AI replace Software Engineers ? Raw & Honest Opinion (Apporva arya)“obviously AI is making us dumb”
</> htmx ~ Yes, and... (htmx.org)“If a junior programmer does not learn to write code and simply generates it, they are robbing”
Agentic Coding is a Trap | Lars Faye (larsfaye.com)“The use of coding agents is actively diminishing the very skills needed to effectively manage the coding agents.”
My AI Adoption Journey (Mitchell Hashimoto)“not forming skills for the tasks you're delegating to the agent”
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Simon Willison’s Weblog)“One thing this might do is make them worse programmers when not using AI”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“Or I can't work on this code base without AI anymore, right? That's the original cognitive surrender cognitive debt kind of definition.”
"Do You Still Read the Code?" • zanlib (zanlib)“Skills deteriorate when they are not used, and it remains to be seen if merely reviewing AI-generated code, rather than typing it out, is enough”
DENIES An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“I am not worried that my programming skills are decaying due to agents”
Leans partly right AI guessAnthropic's 2026 RCT found AI-assisted learners scored lower on mastery of a new library, and other studies show reduced engagement; effects depend on how AI is used.
Leans partly right Vibe-coding literature review lines 1906-1913The review describes 'cognitive debt': deferring understanding during fast iteration raises later debugging and maintenance costs. It does not measure skill loss directly.
Leans partly right Anthropic internal study lines 86-101Anthropic: some engineers report skill atrophy and lost incidental learning, while others say their learning accelerated or the lost skills don't matter; these are perceptions, not measurements.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 951-963AI use lowered short-term comprehension of the participant's own code (-5.95 pp), and the authors caution that over-reliance may hinder students' deeper understanding. However, only immediate recall and explanation were measured, not lasting skill or learning.
Unknown METR 2025 RCT lines 1237-1242METR 2025 raises the possibility that skills without AI atrophy as developers use AI more, which could explain the >50h speedup, but says it is unclear and that more research is needed.
Unknown Echoes of AI maintainability RCT lines 2647-2657Raised as an open risk (over-reliance might erode skills); the study did not measure skill change.
Unknown Shan et al. field study + RCT lines 1396-1405The authors state that the study covers only short-term effects and does not address learning or skill erosion; they raise it only as a risk for future research.

Vibe-coded apps work for prototypes but break down when productionized and scaled

Probably partly right

The 47-source review finds evidence strongest for prototyping and UI work and weakest for production. The break-down half is mostly unstudied, not shown.

· denied 0 ·

Asserts (16)Denies (0)Verdicts (2)

Can Agentic AI Develop Embedded Systems with Model-Based Design? (MATLAB)“The problem, though, with this approach, at least for engineering problems, is that you're skipping the process that makes the result trustworthy.”
AI Replaced 75% of Work for Software Engineers & Data Scientists - here's what's left (Sundas Khalid)“great for like prototyping, but scaling it to like a larger um user base, like that's when it starts to fail”
How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“vibe coding ... fine for throwaway explorations”
SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker (Hamel Husain)“He has not seen any which have been built you know through vibe coding”
Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“prototyping, vip coding, totally very good to do that, but do it in a way that you know this is not your production code”
Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“the code was actually the easy part ... most of it actually was when I tried to make it real”
Context Engineering 101 - The Simple Strategy to 100x AI Coding (Cole Medin)“vibe coding builds prototypes that break when you try to scale”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“it's easy to do uh like the quick prototype with this type of tooling, but it's really hard to productionize something safely”
Why AI WON'T Replace Software Engineering... (Modern Software Engineering)“Enough to prototype the idea, but not even close to being production ready”
No, AI is not Making Engineers 10x as Productive (colton.dev)“you're simply going to hit a productivity wall once the codebase gets large enough”
How To Solve It With Code—background on the course; a discussion with Jeremy and Hamel (Jeremy Howard)“they've gone from zero to mark five and then the kids like oh thanks Dad that's nice but when I click this button”
</> htmx ~ Yes, and... (htmx.org)“The temptation to vibe your way through problems is very, very high, but you will need to fight against that temptation.”
AI Coding and Technical Debt (NT Productions)“you're also going to be dealing with all the other aspects like the release and the protection”
Nobody Agrees About Agentic Development (Convex)“the speed of getting to product-market fit goes up, but the result may be somewhat brittle or less durable”
How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“if you're building infrastructure for the internet you likely want to understand how that's being built”
Vibe code is legacy code (Val Town)“vibe coding is perfect for prototypes and throwaway projects: It's only legacy code if you have to maintain it!”
Probably right AI guessUnreviewed AI-built apps commonly ship with security, scaling and maintainability problems; prototypes are their sweet spot.
Probably partly right Vibe-coding literature review lines 18-28The 47-source review finds evidence strongest for prototyping and UI work and weakest for production. The break-down half is mostly unstudied, not shown.

AI tools slowed experienced open-source developers working on their own familiar codebases (METR 2025 RCT)

Very likely right

METR RCT: 16 experienced developers, 246 tasks in their own mature repos; allowing early-2025 AI tools increased completion time by 19%.

· denied 0 ·

Asserts (16)Denies (0)Verdicts (3)

Jensen Huang Said "Stop Learning to Code" — Then Hired 10,000 Developers (Olive Badger)“experienced developers using AI actually worked 19% slower”
How AI is changing Software Engineering: A Conversation with Gergely Orosz, @pragmaticengineer (AI Engineer)“they felt 20% more productive, but ... they were 20% less productive”
AI made Developers Slower (Telusko)“it has decreased their productivity by 19%”
Why Agent Hype can fall short of reality – Joel Becker, METR (AI Engineer)“we find that developers are slowed down by 19%”
Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“they take 20% longer to complete tasks when they're allowed to use AI versus not.”
AI Replacing Developers Has Officially Failed (Sajjaad Khader)“Seasoned engineers were actually 19% slower when using AI tools”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“And in fact, we we observe that they go slower.”
The METR Paradox. Why AI Made Senior Devs 19% Slower (Macro Lens)“they were 19% slower than when they weren't”
Is AI making devs SLOWER? Yes, according to this study... (Native Notify)“experienced coders finish their work 19% slower”
How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR (AI Engineer)“And in fact, you know, we find that we find a slowdown.”
Where's the Shovelware? Why AI Coding Claims Don't Add Up (mikelovesrobots.substack.com)“They thought AI was making them 20% faster, but it was actually making them 19% slower.”
Are Junior Developers Really Getting Replaced by AI? I'm not sold... (Brian Jenney)“those developers got 19% slower, not faster”
METR's AI productivity study is really good (seangoedecke.com)“But in fact they were 19% slower overall”
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Simon Willison’s Weblog)“Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.”
Does AI Secretly Slow Developers Down? (The AI Daily Brief: Artificial Intelligence News)“the study actually found that they were 19% less productive”
AI Made Devs 19% Slower (METR 2025 Study) (zencoderai)“they actually were 19% slower”
Very likely right METR 2025 RCT lines 11-27METR RCT: 16 experienced developers, 246 tasks in their own mature repos; allowing early-2025 AI tools increased completion time by 19%.
Very likely right AI guessAccurately reports METR's July 2025 RCT (16 devs, ~19% slower); the finding is specific to that setting and early-2025 tools.
Probably right METR 2026 update lines 58-62METR Feb 2026 restates the early-2025 result (tasks took 19% longer with AI, CI +2% to +39%) but adds that the 10 returning developers are now estimated ~18% faster with AI (CI -38% to +9%), so the slowdown was specific to early 2025.

AI handles routine coding well but struggles with complex logic that depends on business or system context

Leans partly right

METR 2025 RCT: developers report AI lacks tacit repository knowledge (e.g. backwards-compat edge cases, where to make edits), behaving like a new contributor; qualitative support for struggling with context-dependent logic, but 'handles routine well' is not shown in this setting.

· denied 0 ·

Asserts (14)Denies (0)Verdicts (8)

How the GitHub Copilot coding agent works | GitHub Checkout (GitHub)“let Copilot take over the boring stuff”
AI Replaced 75% of Work for Software Engineers & Data Scientists - here's what's left (Sundas Khalid)“as soon as you ask it to like do something more complex or like take in the context or the business of the product ... that's where it fails”
Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev) (The Pragmatic Engineer)“help this agent land a fix in a production and really complex codebase, a .NET codebase, and they failed spectacularly”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“AI shines in low context situations ... Where it struggles right now is high context novel problems”
How Google solved their code migration NIGHTMARE with AI (Code Munchies)“The prompt is simple and so are the code changes but it allows devs to easily modernize 5,000 test files in a few months.”
Agentic coding changes everything | Thorsten Ball from Sourcegraph Amp (Changelog)“if you work on a distributed storage layer at Google an agent is not going to take your job in the next two years”
The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“this only works in those circumstances where I'm doing something that people have done hundreds and hundreds of times before”
Oxide and Friends 8/5/2024 -- Pragmatic LLM usage with Nicholas Carlini (Oxide Computer Company)“I want to do the 10% which is the interesting piece”
Agentic Engineering in Action with Mitchell Hashimoto (Zed Industries)“I only find it effective and efficient to use these agentic processes for mid to junior level problems.”
The 70% Problem - Agentic Engineering Glossary | AddyOsmani.com (AddyOsmani.com)“Structure, boilerplate, happy-path logic, standard patterns, CRUD operations, basic UI components - all handled well by AI”
LLMs are bad at vibing specifications (buttondown.com)“It does not seem to be good at writing "subtle" properties that fail due to concurrency, nondeterminism, or bad behavior separated by several steps.”
How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR (AI Engineer)“It is okay at doing very trivial things and it fails at all complex tasks.”
o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“that requires real thought and the the models aren't great at that”
AI Made Devs 19% Slower (METR 2025 Study) (zencoderai)“That's not AI work. That's context work that a human should probably be doing”
Probably right AI guessBenchmarks and field studies show strong performance on well-specified, common tasks and weaker results on context-heavy, novel logic.
Leans partly right METR 2025 RCT lines 1027-1041METR 2025 RCT: developers report AI lacks tacit repository knowledge (e.g. backwards-compat edge cases, where to make edits), behaving like a new contributor; qualitative support for struggling with context-dependent logic, but 'handles routine well' is not shown in this setting.
Leans right Anthropic internal study lines 55-70Anthropic: engineers delegate low-complexity, self-contained work and keep tasks needing team-specific tacit knowledge, citing the 'cold start' problem as the biggest blocker.
Leans partly right Debt study (302k AI commits) lines 841-846AI commits fix slightly more code smells than they introduce (net -7,069) but introduce more correctness (+3,742 net) and security issues (+7,342 net) than they fix, which the authors read as AI handling simple, repetitive patterns but not changes needing deeper understanding of program logic; this is static-analysis evidence, not a test of business-context tasks.
Leans partly right Vibe-coding literature review lines 1957-1964Expert use accelerates routine implementation while retaining architectural reasoning; domains with complex data pipelines or tightly coupled system logic are associated with correctness and verification challenges, but the domain evidence is thin (data/ML in 2 sources).
Leans partly right Echoes of AI maintainability RCT lines 2070-2080AI-devs favoured simple refactorings (inline/rename variable, annotations) while non-AI devs did more structural ones (Extract Class, Move Method, Extract-and-Move only by non-AI); suggestive group-level pattern with large individual variation.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 536-553On deliberately popular, well-trodden tasks (games, CRUD apps, REST clients) where 'modern LLMs shine', AI roughly doubled task completeness. This supports the routine-coding half only; complex, context-dependent work was not tested.
Leans partly right Shan et al. field study + RCT lines 1279-1282GPT-4 solved every easy task but missed 2 of 5 harder algorithmic tasks, consistent with weaker performance on complex logic. The hard tasks were competition puzzles, not business- or system-context tasks.

AI will transform software engineering jobs rather than eliminate them

Leans right

Through 2026 evidence points to role change and slower junior hiring, not elimination of the profession; long-run is uncertain.

· ·

Asserts (13)Denies (1)Verdicts (1)

AI Replaced 75% of Work for Software Engineers & Data Scientists - here's what's left (Sundas Khalid)“our jobs will evolve and not be replaced”
AI made Developers Slower (Telusko)“that doesn't means that it will replace developers completely”
The AI Productivity Paradox: Why 10X Output Doesn't Mean 10X Business Outcome - Mik Kersten (Tech Lead Journal)“I think the role of the developer changes fundamentally.”
The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“Your tools are changing, but your problems are not.”
The TRUTH About the Software Development Job Market in 2026 (Internet Made Coder)“instead of AI replacing programmers, the nature of the job simply changes”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“I don't fall into the camp of thinking we'll all be without a job and replaced by AI, at least in the immediate future”
AI Replacing Developers Has Officially Failed (Sajjaad Khader)“AI ended up creating more demand for humans not less”
Don't Waste 2026 on the Wrong Career (Software vs AI Engineer) (Sajjaad Khader)“It's definitely going to replace tasks”
AI Dev 26 x SF: Andrew Ng: The Future of Software Engineering (DeepLearningAI)“the job apocalypse is not being nearly as bad as people are hyping it up to be”
Teaching CS50 with AI - David J. Malan (CS50)“It will save humans time, and it will amplify the impact of individual humans, much more so, for instance, than replacing them outright.”
</> htmx ~ Yes, and... (htmx.org)“I do think AI is going to change computer programming. Not as dramatically as some people think, but in some”
AI Isn't Replacing Developers - Here's Why (Stefan Mischook)“this is not the end of software development, it is the beginning of a brand new boom”
How Coinbase scaled AI to 1,000+ engineers | Chintan Turakhia (How I AI)“does that count as a replacement for people and like I'm in the camp of absolutely not. AI is an accelerant”
DENIES My AI Skeptic Friends Are All Nuts (Fly)“LLMs really might displace many software developers.”
Probably right AI guessThrough 2026 evidence points to role change and slower junior hiring, not elimination of the profession; long-run is uncertain.

Individual AI productivity gains have not reliably translated into team- or organization-level delivery gains

Leans right

DORA 2024/25 and Faros 2025 found individual output up while organizational delivery metrics stayed flat or worsened; review and integration become the bottleneck.

· denied 0 ·

Asserts (14)Denies (0)Verdicts (2)

2025 DORA report: Insights to impact with Google and Thoughtworks (Thoughtworks)“AI has a tendency to create some localized pockets of productivity. But that productivity gets ... swallowed up to downstream chaos”
How AI is changing Software Engineering: A Conversation with Gergely Orosz, @pragmaticengineer (AI Engineer)“individually it's it certainly is and as teams we're kind of like a bit question mark”
Nicole Forsgren: Leading high-performing engineering teams in the age of AI - The Pragmatic Summit (The Pragmatic Engineer)“we were getting more out, but now our systems, whether technology systems or human systems or processes, are really kind of getting overwhelmed”
Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev) (The Pragmatic Engineer)“These tools are great for individual developers but not yet good at the org level.”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“This is soaking up a lot of those individual gains ... The system moves at the speed of the slowest step”
Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit (The Pragmatic Engineer)“a developer sitting at their desk there is a very very low ceiling of productivity gain.”
The AI Productivity Paradox: Why 10X Output Doesn't Mean 10X Business Outcome - Mik Kersten (Tech Lead Journal)“or 10X gains for the entire organization is actually much, much harder.”
DORA Community Discussion - State of AI-assisted Software Development (DORA)“AI adoption on its own has a modest impact on organizations performance”
AI in engineering: Q2 2026 benchmarks & research readout (DX)“we've not really seen innovation ratio go up at le I mean it's gone up one percentage point basically”
Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“these are activity metrics, right? These are not necessarily business outcomes.”
Real artists still ship (Jerod Santo)“the 180% cumulative effect falls to 50% for the number of projects, and to 30% for actual releases”
DORA 2025: Measuring Software Delivery After AI (Alt + E S V)“only a paradox if you assume individual productivity automatically improves system performance”
DORA Community Discussion - Decoding DORA 2025: AI’s Real Impact (DORA)“It's no good just being individually productive. You need to make your whole end to-end system productive.”
Measuring the actual impact of AI coding | Abi Noda from DX (Changelog)“we're not seeing that strong of a correlation in terms of something like you know code throughput”
Probably right AI guessDORA 2024/25 and Faros 2025 found individual output up while organizational delivery metrics stayed flat or worsened; review and integration become the bottleneck.
Leans partly right Anthropic internal study lines 35-35Anthropic reports a 67% rise in merged PRs per engineer per day after adopting Claude Code across the engineering org, but PR counts are a weak proxy for delivery. (Narrow setting, so it limits the claim rather than refuting it.)

AI-assisted development shifts developer effort toward code review and QA

Probably right

The review finds developer work shifts from writing code toward specification, supervision and validation, with the unit of work moving to the acceptance decision.

· denied 0 ·

Asserts (13)Denies (0)Verdicts (4)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“I think we just need to be ready to be doing more code review”
AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“It's like managing a team of five where you where you're the lead software engineer and you're have to review the code”
Nicole Forsgren: Leading high-performing engineering teams in the age of AI - The Pragmatic Summit (The Pragmatic Engineer)“Review ends up surfacing quite a bit, right? Because we're just we're putting so much work on it.”
How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“so much of the time is now going to be spent reviewing code versus writing code”
Building the Context Engine AI Agents Need (InfoQ)“you then kind of force a problem which is a bottleneck at code review”
Using AI Code Generation to Migrate 20000 Tests (InfoQ)“developers role shifted more to reviewing fixing verifying.”
The 70% Problem - Agentic Engineering Glossary | AddyOsmani.com (AddyOsmani.com)“The craft shifts from writing code to evaluating code”
Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“They're spending more and more time in planning and code review because there's so much more code being generated”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“Our job as an engineer is to no longer write the code, but to do the higher leverage tasks like the planning and validating”
AI in software engineering at Google: Progress and the path ahead (research.google)“the code author increasingly becomes a reviewer”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“We have higher coding throughput. But then can we if we can code faster, can we review faster”
Developer experience in the AI era (Develocity)“we've basically switched something that was primarily writing and reading and then reviewing later to accepting a whole bunch, prompting, accepting and then reviewing.”
How AI Is Reshaping Software Development Insights from the 2025 DORA Report | AppDevANGLE (SiliconANGLE theCUBE)“I'm going to spend much more time editing or reviewing code than I might have spent in the past writing code.”
Very likely right AI guessSurveys and studies (e.g. Faros, Google) report increased PR volume and review load with AI adoption.
Probably right Vibe-coding literature review lines 1900-1931The review finds developer work shifts from writing code toward specification, supervision and validation, with the unit of work moving to the acceptance decision.
Probably partly right METR 2025 RCT lines 433-441METR 2025 screen recordings (74 issues): with AI, developers spend less time coding and searching and more time reviewing AI output, prompting and waiting; supports a shift toward review, though QA/testing time is not singled out.
Leans right Anthropic internal study lines 115-115Anthropic interviews: engineers describe becoming 'managers of AI agents', and one estimated their work shifted '70%+ to being a code reviewer/reviser'; this is qualitative self-report.

Giving the model the right minimal context matters more than prompt wording tricks

Leans right

As models improved, output depends more on relevant, high-signal context than phrasing; consistent with Anthropic/Cursor guidance and long-context degradation evidence.

· denied 0 ·

Asserts (13)Denies (0)Verdicts (3)

Context Engineering & Coding Agents with Cursor (OpenAI)“it's more about giving the models the right context”
Getting the most out of Codex (Academind)“if you do know that a certain file will matter ... you wanna point your agent at it”
Context engineering for AI agents (Microsoft Developer)“ensuring our AI agent has the right information at the right time”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“Context is king in this moment”
Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro (AI Engineer)“the agent ... does better when given less context but given the tools to understand where to go find things”
We benchmarked the TOP AI Code Reviewers (Augment Code)“The hardest part of AI code review isn't generating the comments. It's assembling the right context”
Oxide and Friends 8/5/2024 -- Pragmatic LLM usage with Nicholas Carlini (Oxide Computer Company)“show the model the minimum amount of things so that you're not going to blow the budget”
SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“crafting the problems exactly is not as important anymore with the latest models”
A very subjective comparison of Claude Code, OpenCode, Cursor & GitHub Copilot (Maximilian Schwarzmüller)“the most important part will always be your prompt, the context you provide”
vulnerability research just got easier (scarier?) (Low Level)“You're giving it like random utility functions that like the user can't even access.”
Coding with AI episode 3: fixing a real world bug in Redis vector sets with LLMs (Salvatore Sanfilippo)“it's very important to provide as fast as possible the right context.”
AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“always tell it that you are the world's greatest expert in X before you ask all of that kind of mostly rubbish”
Discussing the State of AI-assisted Software Development with DORA and EarnIn (Datadog Community)“we're moving beyond prompt engineering and really into context engineering”
Very likely right AI guessAs models improved, output depends more on relevant, high-signal context than phrasing; consistent with Anthropic/Cursor guidance and long-context degradation evidence.
Leans partly right Airbnb migration lines 44-45Airbnb: the main success driver was choosing the right related files rather than perfecting prompt wording (supports 'right context over wording'), but the context was very large rather than minimal.
Leans partly right Google migrations report lines 351-356Google: telling the model about too many change locations sometimes did worse than naming one place and asking it to apply the change holistically, so context shape matters, though they also packed whole clusters into Gemini's large context.

Non-engineers such as PMs can now build working prototypes with AI without a developer

Leans right

Tools like Lovable, v0, Replit let non-developers produce working prototypes; widely observed.

· denied 0 ·

Asserts (13)Denies (0)Verdicts (2)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“strategy and ops people and BD people and marketing people, you know, kind of be like builders in labs and they are really good”
AI at Honeycomb: What’s Actually Working (w/ CTO Charity Majors) (Jam)“our our designers can you know uh do do code prototypes”
AI Replaced 75% of Work for Software Engineers & Data Scientists - here's what's left (Sundas Khalid)“you actually don't need a software engineer to like build the first prototype”
Measuring AI code assistants and agents with the AI Measurement Framework (DX)“giving designers and PMs some of these vibe coding tools as a way to do more rapid prototyping and exploration.”
Master 80% of Google Antigravity in 27 Minutes (Ali H. Salem)“even if you've never tried anything like this before”
How AI is changing Software Engineering: A Conversation with Gergely Orosz, @pragmaticengineer (AI Engineer)“if you actually enable your non-coding ... collaborators to code actually they are more productive”
Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“suddenly everyone is a programmer because everyone speaks natural language”
The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“We're going to see more software written by people who are not professionals.”
Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“the people who don't know code being able to get something done that they otherwise couldn't do, that is actually transformative.”
Using AI Code Generation to Migrate 20000 Tests (InfoQ)“it enables some people especially those who possess some of the knowledge but they don't know the programming language”
12 Months of Unity Dev with Claude Code (Mythmatic)“I am an artist and I'm using cloud code to do stuff that traditionally an artist wouldn't jump into”
How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“the growth in cursor at Shopify is happening a lot outside of engineering and outside of R&D finance sales support”
Discussing the State of AI-assisted Software Development with DORA and EarnIn (Datadog Community)“you're already seeing product managers that are able to self-service build a prototype much much faster”
Very likely right AI guessTools like Lovable, v0, Replit let non-developers produce working prototypes; widely observed.
Leans partly right Vibe-coding literature review lines 1393-140045% of sources stress accessibility for novices and non-programmers. That evidence is mostly grey literature and centered on prototypes.

AI is reliably useful for generating tests and documentation

Leans partly right

AI speeds drafting tests and docs, but generated tests often assert implementation details or weak properties and need review.

· denied 0 ·

Asserts (14)Denies (0)Verdicts (3)

AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“writing documentation is extremely extremely adapted to uh yeah to AI code generation”
AI made Developers Slower (Telusko)“It is also useful in creating test cases. It is also useful in creating documentations”
Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“I have more tests and better tests and more documentation and better documentation than I did before I was using AI stuff.”
Unpacking METR’s findings: Does AI slow developers down? (DX)“documentation, unit tests, refactoring, these things, AI oneshots”
Agentic coding changes everything | Thorsten Ball from Sourcegraph Amp (Changelog)“I would say the test suite 90% of that is generated, right? Like oh cover all of these cases.”
Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“And this alone is, I believe, one month of work because it's a lot of Python code.”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“generating tests for testing the proprietary code, we absolutely use AI all the time”
How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“creating documentation and cleaning up documentation”
Is AI making devs SLOWER? Yes, according to this study... (Native Notify)“the time it takes to create good documentation was cut almost in half”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“They're free now. They're effectively free.”
Learning to Code with AI and Steve Klabnik (Scott Hanselman)“LLM generated documentation is better than zero documentation”
How Coinbase scaled AI to 1,000+ engineers | Chintan Turakhia (How I AI)“I just did like you know 20 unit tests and then went and had a coffee.”
Beyond the Hype: Real Talk on AI-Assisted Development • Jessica Kerr & Diana Montalion • GOTO 2025 (GOTO Conferences)“My entire career I've wanted to back out of writing tests. I can have all the tests now!”
Discussing the State of AI-assisted Software Development with DORA and EarnIn (Datadog Community)“our engineering team uses AI uh to automate end to end test creations”
Probably partly right AI guessAI speeds drafting tests and docs, but generated tests often assert implementation details or weak properties and need review.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 575-595On D4 (writing a JUnit 5 suite from scratch, mostly MSc students), AI users reached median completeness 100 vs 42.5 without AI. That was the strongest AI help of any task, but it came with the most severe ownership loss. Evidence covers one test task; documentation was not studied.
Leans partly right Echoes of AI maintainability RCT lines 1948-1956AI-devs added unit tests somewhat more often (46% vs 31%), described as 'no clear tendency'; test coverage barely moved (~+1pp); they added comments more often (79% vs 54%) but authors call them noisy, surface-level comments.

The extra code volume AI produces increases complexity and maintenance burden

Leans right

Smells make up 89% of AI-introduced issues and pass review easily. The paper argues AI's higher speed and volume let these accumulate into a substantial maintenance burden.

· denied 0 ·

Asserts (13)Denies (0)Verdicts (5)

Unpacking METR’s findings: Does AI slow developers down? (DX)“you end up with this huge bloated codebase because an LLM is trying to work with kernels”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“the more lines of code, the more time you will have to read them”
How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“we know that agents and and LLMs tend to be very verbose”
Why AI WON'T Replace Software Engineering... (Modern Software Engineering)“more AI-generated code means more complexity, more integration points, more maintenance burden”
Using AI Code Generation to Migrate 20000 Tests (InfoQ)“you you have generated so much code that you don't know what it does. You don't know how to validate it.”
AI Replacing Developers Has Officially Failed (Sajjaad Khader)“AI generated code results in 38% more code volume”
Enterprise Vibe Coding: Building Secure, AI-Powered Apps in 30 Minutes - Parth Sharma (Developer Summit)“teams are shipping code faster than you can actually keep track of”
How To Solve It With Code—background on the course; a discussion with Jeremy and Hamel (Jeremy Howard)“it just ends up being overly complicated and not really customized for your particular problem”
</> htmx ~ Code is Cheap(er) (htmx.org)“LLMs are incapable of fear of complexity, and are prolific coders.”
The AI-Native Data Engineer featuring Fabric, dbt, VSCode devcontainers and copilot (Microsoft Reactor)“I don't want the LLM to go and generate like 10,000 lines of Scala, right?”
Are Junior Developers Really Getting Replaced by AI? I'm not sold... (Brian Jenney)“the code that these tools write is excessive”
576 - Using LLMs at Oxide | RFD | Oxide (Oxide Computer Company)“we do not want to become as gavage geese, being force-fed unnecessary complexity in hopes that an LLM will somehow spare us from asphyxiation.”
Developer experience in the AI era (Develocity)“they can just be writing thousands of lines of code that are not maintainable.”
Probably right AI guessGitClear and DORA 2025 data show more churn and duplication and lower delivery stability as AI adoption rises; more code means more to maintain.
Leans right Debt study (302k AI commits) lines 900-912Smells make up 89% of AI-introduced issues and pass review easily. The paper argues AI's higher speed and volume let these accumulate into a substantial maintenance burden.
Leans partly right Echoes of AI maintainability RCT lines 2625-2632Authors warn that reckless AI use may bloat codebases and that volume drives complexity, but in their data the median AI-dev added fewer lines (123 vs 166) and the two highest-volume AI users had the best CodeHealth; the burden is projected, not observed.
Leans partly right Vibe-coding literature review lines 1445-1452WL5 and WL15 are cited as saying rapid AI-assisted iteration can introduce structural complexity that complicates later maintenance; the effect is not tied to code volume or measured.
Leans partly right Shan et al. field study + RCT lines 1149-1155Self-rated code complexity did not differ with workplace AI use (Table 11), but the follow-up found lower maintainability and more technical debt. The burden part is supported, the complexity part is not.

AI productivity gains shrink as task complexity, codebase maturity and codebase size grow

Probably right

METR names repo size and maturity (10+ years, more than 1M lines) and complex edits as factors behind the slowdown, in contrast with greenfield RCTs that found speedups.

· ·

Asserts (12)Denies (1)Verdicts (4)

AI at Honeycomb: What’s Actually Working (w/ CTO Charity Majors) (Jam)“database engineers who are working on storage engine it has not transformed their velocity and nor should it”
Stanford Study: AI Makes Developers 20% More Productive (But There's a Catch) (Josh Anderson)“it doesn't take very long for your product to quickly reduce the value that AI provides”
AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“for brownfield the picture looks much less pretty”
The AI Productivity Paradox: Why 10X Output Doesn't Mean 10X Business Outcome - Mik Kersten (Tech Lead Journal)“they work better for more greenfield”
Why Agent Hype can fall short of reality – Joel Becker, METR (AI Engineer)“AIs might be less performant on on large and complex repositories”
Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“these developers are very highly familiar with their work and that these repositories are large and complex.”
Context Engineering SF: Advanced Context Engineering for Agents - Dex Horthy (Chroma)“it just doesn't work for complex tasks or brownfield tasks ... it can actually slow people down”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“AIS do do worse in sort of larger environments by lines of code or numbers of files”
Are Junior Developers Really Getting Replaced by AI? I'm not sold... (Brian Jenney)“at some point when the code becomes just a little too complex, you have to really ease up off the gas”
METR's AI productivity study is really good (seangoedecke.com)“Working on a 500 line codebase is just qualitatively different from working on a 1M+ line codebase.”
Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford (AI Engineer)“high complexity brownfield tasks 0 to 10%”
Measuring the actual impact of AI coding | Abi Noda from DX (Changelog)“small clean code bases versus applying them to legacy code bases millions of lines of code uh messy”
DENIES Anthropic Economic Index 2026—Measuring Claude Usage with “Economic Primitives” Metrics (CosmoX)“The data shows that AI gives the biggest speed boosts on the hard stuff, the more complex, higher-skill work.”
Probably right AI guessConsistent across Stanford's data, METR's mature-repo slowdown and long-context degradation results.
Probably right METR 2025 RCT lines 986-1003METR names repo size and maturity (10+ years, more than 1M lines) and complex edits as factors behind the slowdown, in contrast with greenfield RCTs that found speedups.
Leans partly right Google RCT lines 529-539Google's complex enterprise task (unfamiliar 474-line CL, build/test updates) yielded ~21% vs 56% on Peng's simple HTTP-server task, but the authors attribute the gap mainly to population differences rather than task complexity.
Leans partly right Shan et al. field study + RCT lines 1197-1205Comparing two separate experiments: the AI accuracy gain fell on harder tasks (0.124 vs 0.264), but the task-count gain rose slightly (0.283 vs 0.245). Complexity was not manipulated, and the authors call this suggestive.

AI code review tools catch real bugs that human reviewers miss

Leans right

AI reviewers do catch real defects missed by humans, though with notable false-positive rates; complement, not replacement.

· denied 0 ·

Asserts (14)Denies (0)Verdicts (1)

Context Engineering & Coding Agents with Cursor (OpenAI)“it actually caught a lot of bugs that we missed on code reviews”
Open Source security in spite of AI - Daniel Stenberg (Daniel Stenberg)“I have fixed more than 100 bugs found by these tools when I started to use them”
AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“he found two bugs, critical bug line 223 where I put uh humidity instead of outside air quality”
Claude Code Alternatives in 2026: Cursor, Qodo & GitHub Copilot Compared (Qodo)“It says that there are open AI key exposed”
DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“the PRs that have been reviewed by agents caused far fewer issues in production”
Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“I am 100% sure that AI can routinely find bugs before of users in complex C projects.”
Codex CLI: Everything You Need in 12 mins (lustoykov)“Codex came back with actionable items, and also sorted by priorities”
Agentic Engineering in Action with Mitchell Hashimoto (Zed Industries)“oh, I actually forgot to update the code. Um, so it's found bugs that way.”
Coding with LLMs in the summer of 2025 (an update) - <antirez> (antirez.com)“many were just removed immediately by Gemini / Claude code reviews.”
Control the ideas, not the code - <antirez> (antirez.com)“Fable and GPT 5.6 reviews to the sorted sets memory saving are going to spot ways more errors and subtle race conditions”
Using AI to write better code more slowly (Read the Tea Leaves)“the latest public models from Anthropic and OpenAI are good enough to find plenty of bugs in an unscrutinized codebase.”
Comparing all AI Pull Request Review tools to find the best one (Morning Maker Show with Sandra and Dan)“they are definitely here to stay. They were all useful to some extent”
Coding with AI episode 3: fixing a real world bug in Redis vector sets with LLMs (Salvatore Sanfilippo)“it immediately found my bug, which was awesome.”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“you forgot about what happens if there's no audio”
Probably right AI guessAI reviewers do catch real defects missed by humans, though with notable false-positive rates; complement, not replacement.

Project instruction files (CLAUDE.md, AGENTS.md, rules) improve agent adherence to project conventions

Leans right

Persistent instruction files measurably steer agent behavior; overlong or contradictory ones reduce adherence.

· ·

Asserts (10)Denies (2)Verdicts (1)

Getting the most out of Codex (Academind)“use that agents.md file to your advantage. Keep it concise”
Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“so that it follows your idea of of good software engineering practices”
Context Engineering 101 - The Simple Strategy to 100x AI Coding (Cole Medin)“I treat my cloud MD as where I put like the constant rules that will very rarely change”
Gemini CLI: How To Get Started (pixegami)“a configuration file, which is essentially going to be a system prompt for your agent”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“for the agent not to think about these things all the time ... creating this file called uh agents.mmd”
Test-driven development (TDD) with Claude Code (James Murdza)“inside of testing.md, I have everything that the uh needs to know in order to run the playwright end-to-end tests”
An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“It would not surprise me if the file is the main differentiator between those getting good and bad results with agents”
My AI Adoption Journey (Mitchell Hashimoto)“Each line in that file is based on a bad agent behavior, and it almost completely resolved them all.”
How Coinbase scaled AI to 1,000+ engineers | Chintan Turakhia (How I AI)“we chipped away and we put in a bunch of rules like cursor rules and that that helps so much.”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“you're going to have an agents, a spec, a road map”
DENIES Codex CLI: Everything You Need in 12 mins (lustoykov)“/init is another command that you probably should not use ... this file pollutes Codex context more than it actually helps”
DENIES Building the Context Engine AI Agents Need (InfoQ)“people think that more rules and Claude MD files will fix this ... Claude was like just completely ignored it”
Probably right AI guessPersistent instruction files measurably steer agent behavior; overlong or contradictory ones reduce adherence.

Updating the agent's rules and workflows after each failure compounds its reliability over time

Leans right

Airbnb's 'sample, tune, sweep' loop (find common failures, fix the prompt/scripts, re-run) raised completion from 75% to 97% in 4 days.

· denied 0 ·

Asserts (12)Denies (0)Verdicts (2)

From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“Every time something goes wrong, they try to improve the harness first before new code gets written.”
Ralph Wiggum: Continuous Agent Loops in Claude Code (AI Makerspace)“if I notice Claude has made a mistake ... I'm simply going to tell Claude to remember to not make that mistake”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“a learnings markdown file ... a powerful self-improving cycle”
Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro (AI Engineer)“once I go through that pain of learning I just say kira write what you learned into a steering doc”
Gemini CLI: How To Get Started (pixegami)“start off with an empty file ... as you ... run into behaviors that you want it to change, that's when you can update the context file”
Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“oh if it makes a mistake and I add that just to agents.md, it never makes that mistake again.”
o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“I'm constantly not just working on my project. I'm working on the process of working on my project at the same time.”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“We can spend some time to fix ... the system that allowed the bug”
An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“adding a rule to my AGENTS.md whenever I encounter a fundamental behavior I don't like has been very effective”
My AI Adoption Journey (Mitchell Hashimoto)“anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again.”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again”
Beyond the Hype: Real Talk on AI-Assisted Development • Jessica Kerr & Diana Montalion • GOTO 2025 (GOTO Conferences)“I either need to change my prompts or more often change my instructions.”
Leans partly right AI guessEncoding lessons in rules and workflows does reduce repeat mistakes, but rule files bloat and conflict if not pruned; gains are real but not automatically compounding.
Leans right Airbnb migration lines 54-60Airbnb's 'sample, tune, sweep' loop (find common failures, fix the prompt/scripts, re-run) raised completion from 75% to 97% in 4 days.

AI speeds up throwaway prototyping far more than production coding

Leans partly right

Time-to-prototype gains are the most common reported outcome (45% of sources), while production use is weakly evidenced. The gap is consistent with the claim but not quantified.

· denied 0 ·

Asserts (13)Denies (0)Verdicts (3)

AI at Honeycomb: What’s Actually Working (w/ CTO Charity Majors) (Jam)“the ideation phase when when you're throwing a lot of code away anyway has gotten a lot faster.”
AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“initial prototype prototyping crossplatform features yeah implemented in a few minutes instead of days”
What We Learned Deploying AI within Bloomberg’s Engineering Organization – Lei Zhang, Bloomberg (AI Engineer)“the measurements dropped actually pretty quickly when you go beyond all the green field type of thing”
Andrew Ng: Building Faster with AI (Y Combinator)“maybe 10 times faster ... for production code ... maybe 30 to 50% faster”
Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“it probably is 5 or 10x productivity for prototypes.”
Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“when it turns into productionizing something, uh it feels similar to what it's always been.”
The 70% Problem - Agentic Engineering Glossary | AddyOsmani.com (AddyOsmani.com)“AI excels at prototypes where the 70% is sufficient.”
Coding with LLMs in the summer of 2025 (an update) - <antirez> (antirez.com)“letting the LLM write the throw away code to test ASAP”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“I would not I would not ship that app.”
12 Months of Unity Dev with Claude Code (Mythmatic)“Throwaway tools. It doesn't really matter if it's average, below average, whatever.”
576 - Using LLMs at Oxide | RFD | Oxide (Oxide Computer Company)“This is especially valuable for code that is experimental or auxiliary or otherwise throwaway.”
Sustainable Augmented Development • Kent Beck • YOW! 2025 (GOTO Conferences)“as soon as you hit expand, the genie is out of gas”
Vibe code is legacy code (Val Town)“I was able to vibe code these apps way faster than I could've built them, and it was a blast.”
Leans partly right AI guessDirection is well supported: greenfield, low-stakes work gets the biggest gains in studies and surveys. The specific multipliers are anecdote.
Leans partly right Vibe-coding literature review lines 1413-1425Time-to-prototype gains are the most common reported outcome (45% of sources), while production use is weakly evidenced. The gap is consistent with the claim but not quantified.
Leans partly right METR 2025 RCT lines 581-586METR 2025 caveat: its slowdown on mature repos is 'consistent with small greenfield projects or development in unfamiliar codebases seeing substantial speedup'. This is a hypothesis; prototyping was not measured.

AI labs' capability claims are inflated by fundraising incentives and unverifiable closed testing

Leans partly right

Labs do have incentives and many evals are self-reported, but independent evals (METR, SWE-bench, LMArena) largely confirm capability gains.

· denied 0 ·

Asserts (11)Denies (0)Verdicts (1)

Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev) (The Pragmatic Engineer)“again is an AI company founder very much interested in in in this”
ICM 2026 Public Lectures - Terence Tao (Simons Foundation)“companies that are disclosing these have their own incentives to to maybe uh uh present the results in as favorable a light as possible”
The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“He's leading a company who needs to make money and it's a company who he needs to speak to his stakeholders.”
The TRUTH About the Software Development Job Market in 2026 (Internet Made Coder)“they need you to believe that so that they can then raise an even bigger round of funding”
Andrew Ng: Building Faster with AI (Y Combinator)“hype narratives ... that make these businesses look more powerful ... helpful for raising money”
AI Replacing Developers Has Officially Failed (Sajjaad Khader)“AI was a failed marketing scheme, not a full technical replacement”
No, AI is not Making Engineers 10x as Productive (colton.dev)“If your company is built on the back of AI, you are incentivized to sell AI as a miracle solution”
Google 'Big Sleep' AI Agent flags SQLite 0day (Three Buddy Problem)“this is not the way that you share new findings that have never been discussed before”
AI Will Replace Software Engineers? Here’s What’s Actually Happening (Chris Schwenk | Tech Jobber)“AI will replace engineers isn't a prediction, kind of like a pitch deck”
Cursor Claims AI Built a Browser. Here's the Problem (Brian Jenney)“these billion-dollar companies that seem to have a mission of replacing developers seem to be really incentivized to tell everybody”
AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“it's coming nobody will ever work again which helps you raise a lot of money”
Leans partly right AI guessLabs do have incentives and many evals are self-reported, but independent evals (METR, SWE-bench, LMArena) largely confirm capability gains.

Developers should be able to explain AI-generated code without the AI's help before it is merged

Leans right

Ownership and maintainability require understanding; it's a reasonable review standard rather than an empirical finding.

· denied 0 ·

Asserts (11)Denies (0)Verdicts (2)

Jensen Huang Said "Stop Learning to Code" — Then Hired 10,000 Developers (Olive Badger)“It is unacceptable to say, Oh, the AI did it”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“never commit code that you can't explain to other people, or even to yourself”
ICM 2026 Public Lectures - Terence Tao (Simons Foundation)“if an author cannot give a talk like up here about the result and they cannot take questions about the result”
Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“I feel like you still have a professional responsibility to understand the code that you're committing to master or main or trunk.”
AI-Generated Code Is Creating a New Kind of Technical Debt (Imran Gardezi)“can the developer who submitted this PR walk through the logic line by line with the AI's comments deleted”
How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt (Margaret-Anne Storey)“require that at least one human on the team fully understands each AI-generated change before it ships”
576 - Using LLMs at Oxide | RFD | Oxide (Oxide Computer Company)“LLM-generated code should not be reviewed by others if the responsible engineer has not themselves reviewed it.”
How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“yes you can submit a PR but you have to understand the code you are writing yourself before exactly before you submit it”
AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“I don't think you should ever commit a line of code that a language model wrote If you don't understand it yourself”
A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“I increasingly have a problem when infrastructure core components are being written with the power of AI and no human that understands that anymore”
Let's put the AI in lots of little boxes - Kenton Varda, Cloudflare Connect 2025 (Cloudflare Developers)“you have to already understand as a human everything that it does uh and you are responsible if there's a problem”
Probably right AI guessOwnership and maintainability require understanding; it's a reasonable review standard rather than an empirical finding.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 700-721Crossover experiment, 69 participants (mostly students): after AI-assisted tasks people answered fewer questions about their own just-written code correctly (median 87.5% vs 100%; AME -5.95 pp, OR 0.58). The authors argue this can stop contributors defending their choices in code review. That supports the reason behind the norm, but no merge gate was tested.

Agents run with permission prompts disabled should be confined to an isolated or disposable environment

Leans right

Agents have deleted files and run destructive commands when unrestricted; vendors themselves recommend containers/VMs for skip-permissions modes.

· denied 0 ·

Asserts (11)Denies (0)Verdicts (2)

Ralph Wiggum: Continuous Agent Loops in Claude Code (AI Makerspace)“dangerously run permissions mode which uh which you should never do unless you're in a sandbox”
Master 80% of Google Antigravity in 27 Minutes (Ali H. Salem)“Use a disposable clone or a container for risky tasks”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“the most important one is to consider using containers like Docker”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“I run it on a remote machine that um if something happens to this machine it will not affect my computer”
AI Assistant Tool Wipes Out Entire Company Database | GRAVITAS (WION)“how safe is it for users who often trust AI with important and critical data”
Inside minions: how Stripe uses autonomous one-shot AI coding agents (Stripe Developers)“the minions are, first of all, running in a sandbox developer environment”
Episode 155: The Replit AI Debacle (AIPOD-CAST)“Think airgapped environments, strict role-based access”
The Replit AI Agent DISASTER: Production Database Deleted In SECONDS! ($erver $ays)“a textbook failure to enforce a principle of least privilege”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“So, I think the most important thing is sandboxing.”
A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“the agent will do things that it shouldn't be doing, unsurprisingly”
Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“Definitely don't touch anything production only touch code only use these things when git is your back stop”
Very likely right AI guessAgents have deleted files and run destructive commands when unrestricted; vendors themselves recommend containers/VMs for skip-permissions modes.
Leans partly right Vibe-coding literature review lines 1631-1642Sandboxing appears in 15% (5/33) of safeguard sources (e.g. Seatbelt read-only jail, Docker plus iptables isolation) and is a recommended practice, but its effectiveness is not evaluated.

AI coding tools amplify experienced engineers far more than novices

Probably wrong

METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure.

· ·

Asserts (10)Denies (2)Verdicts (8)

Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“for this kind of help, you have to be very good at prompting. I don't believe everybody is the same at using LLMs.”
AI Coders Scored 17% Lower—Here's What They Did Wrong (DIY Smart Code)“it makes good developers better. It doesn't make beginners good”
LLMs are bad at vibing specifications (buttondown.com)“If you need to know formal methods to get the LLM to do formal methods, is that really helping?”
</> htmx ~ Yes, and... (htmx.org)“Senior programmers who already have a lot of experience from the pre-AI era are in a good spot to use LLMs effectively”
AI Will Replace Software Engineers? Here’s What’s Actually Happening (Chris Schwenk | Tech Jobber)“For someone with no experience, AI gives you a toy ... for someone with real experience, AI is a rocket strapped to their back”
AI Isn't Replacing Developers - Here's Why (Stefan Mischook)“It's a power tool for competent users”
AI Coding and Technical Debt (NT Productions)“If you're an experienced coder ... you're probably going to have a lot less technical debt”
Coding with AI episode 3: fixing a real world bug in Redis vector sets with LLMs (Salvatore Sanfilippo)“if I was less experienced at working with LLMs, I could just drop the implementation here”
How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“the AI agents are going to help the best engineers more than the mediocre engineer”
AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“when I combine it with these tools I will run circles around somebody who's never written a code line of code in their life”
DENIES 12 Months of Unity Dev with Claude Code (Mythmatic)“So in my case if I'm starting at almost near zero and it bumps me into average code the multiplier is insane.”
DENIES How much does AI impact development speed? (newsletter.getdx.com)“More senior developers were faster overall, but they didn’t disproportionately benefit from AI.”
Probably wrong METR 2025 RCT lines 135-141METR 2025 cites prior work finding that AI benefits less-experienced workers more; in its own RCT, highly experienced developers were slowed down, and more so on tasks where they had high prior exposure.
Leans wrong Microsoft/Accenture field RCTs lines 758-770Microsoft experiment: short-tenure developers raised output 27-39% and juniors 21-40%, versus 8-13% for long-tenure and 7-16% for seniors (directionally consistent, mostly not significant), the opposite of AI amplifying experienced engineers more.
Leans partly right AI guessExperts get more reliable results and catch errors, but several RCTs show larger measured speedups for juniors, and METR found experts slowed; contested.
Leans wrong Shan et al. field study + RCT lines 872-878Moderation analysis in the 253-developer RCT: developers with fewer years of experience got the speed benefit and quality was unaffected, so gains concentrated among less-skilled coders, not experienced ones (easy tasks, ChatGPT-3.5 chat, appendix result).
Leans wrong Copilot RCT (Peng) lines 221-225Copilot RCT heterogeneity analysis: developers with less programming experience benefited more, which the authors highlight as promise for novices (small sample, noisy estimates).
Leans partly right Google RCT lines 490-505Google RCT: more senior engineers and those coding 5+ hours/day were faster with AI, but the AI x seniority (p=0.706) and AI x hours interactions were not significant, so 'far more' is unsupported.
Leans partly right Anthropic internal study lines 75-75Anthropic: experience is needed to spot subtly dangerous Claude proposals, yet a senior engineer says the tools make juniors 'more productive and more bold', so the gain isn't clearly skewed to seniors.
Leans wrong Code-ownership experiment (Martin-Lopez) lines 556-569Every group, BSc students included, gained significantly in completeness, and the authors say differences between groups were not related to experience. Professionals gained the most, but they had different tasks, and position is confounded with task and language.

Top-down AI usage mandates and unrealistic productivity targets backfire

Leans partly right

Reports of token-maxing and resentment support it, but some mandated rollouts (e.g. Shopify, Coinbase) report adoption gains; evidence is anecdotal either way.

· ·

Asserts (10)Denies (1)Verdicts (1)

2025 DORA report: Insights to impact with Google and Thoughtworks (Thoughtworks)“don't set unrealistic expectations ... I expect 10x productivity gains”
Reconsidering AI w / Steve Klabnik (Zed Industries)“being forced by management to adopt these tools. And that caused a lot of heat and a lot of argument”
SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“if that's what you're being measured on, people are going to cut corners and it's going to lead to more incidents”
No, AI is not Making Engineers 10x as Productive (colton.dev)“Unrealistic 10x expectations will result in rushed and thus subpar work without fail.”
My AI Skeptic Friends Are All Nuts (Fly)“Tech execs are mandating LLM adoption. That's bad strategy.”
Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“The more successful um technique that that we've applied is actually just sharing wins.”
Agentic Coding is a Trap | Lars Faye (larsfaye.com)“forced mandates and hype around token usage across organizations is demonstrating as such.”
Where's the Shovelware? Why AI Coding Claims Don't Add Up (mikelovesrobots.substack.com)“People are being fired because they’re not adopting these tools fast enough”
576 - Using LLMs at Oxide | RFD | Oxide (Oxide Computer Company)“Such decrees are absurd: they undermine the autonomy and mastery essential in our work.”
How Coinbase scaled AI to 1,000+ engineers | Chintan Turakhia (How I AI)“the worst thing any leader could do is just be like, I decree you must use AI.”
DENIES How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“your impact is going to be evaluated as if you had the tool”
Leans partly right AI guessReports of token-maxing and resentment support it, but some mandated rollouts (e.g. Shopify, Coinbase) report adoption gains; evidence is anecdotal either way.

Coding agents overstate what they accomplished in their own summaries

Leans right

Widely reported and seen in evals: agents claim tests pass or features are complete when they aren't; reading summaries critically is necessary.

· denied 0 ·

Asserts (11)Denies (0)Verdicts (1)

Armin Ronacher Leaning In To Find Out - PyAI Conf 2026 (Pydantic)“it just claimed that it validated it, but it didn't in fact do that”
How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“always remember it's gullible and it's likely to lie to you”
Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro (AI Engineer)“the LLMs are very good at saying I'm done. I'm happy ... Oh, yeah. The tests don't pass but they're annoying”
Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“It doesn't typically lie about what it does, right? But it grossly exaggerates the impact of what it does”
AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“It will it will say that tests pass when they didn't.”
AI Assistant Tool Wipes Out Entire Company Database | GRAVITAS (WION)“ignoring repeated instructions, and even lied about it blatantly”
Episode 155: The Replit AI Debacle (AIPOD-CAST)“report passing tests that hadn't actually run ... Fabricating log entries”
The Replit AI Agent DISASTER: Production Database Deleted In SECONDS! ($erver $ays)“inventing fake unit test results. It was actively hiding its own flaws”
o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“that's the press release of what I did for you”
How far can we push AI autonomy in code generation? (martinfowler.com)“AI frequently claimed the build and tests were successful and moved on to the next step, even though they were not”
Sustainable Augmented Development • Kent Beck • YOW! 2025 (GOTO Conferences)“It makes horrible mistakes and covers them up.”
Probably right AI guessWidely reported and seen in evals: agents claim tests pass or features are complete when they aren't; reading summaries critically is necessary.

Combining deterministic transformation tools (codemods, code generators) with AI is more reliable than having AI generate large changes directly

Leans right

Google migration experience report: simple LLM prompting alone only works for the simplest migrations; AST tooling, heuristics and LLMs together are needed, and AST checks catch model mistakes cheaply and deterministically.

· denied 0 ·

Asserts (11)Denies (0)Verdicts (2)

Can Agentic AI Develop Embedded Systems with Model-Based Design? (MATLAB)“Simulink generated embedded code from a model using well-tested, deterministic workflows. So, the same design always produces the same code.”
Building Blocks for Uber’s Software Factory— Uday Kiran Medisetty & Adam Huda, Uber (AI Engineer)“We handled more than 250 automated migrations 9 million lines of code automatically for our engineers.”
How Google solved their code migration NIGHTMARE with AI (Code Munchies)“using their AI migration tools they have been able to really increase the number of landed changelists through 2024”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“generate those recipes and other tools with AI itself and have a human in the loop that can work in that system”
Using AI Code Generation to Migrate 20000 Tests (InfoQ)“AI by itself was not very successful tool”
SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“this computational guide can maybe cover like 80 90% of the work and then the rest of the is a bit more semantic”
Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“we needed to run a big program to create a scalable version of how we handle large-scale change.”
How far can we push AI autonomy in code generation? (martinfowler.com)“if you can generate a script or a codemod instead of letting AI do the full work itself.”
How Google is accelerating code migrations with AI (newsletter.getdx.com)“Google cut that time in half while having AI generate 70% of the code changes.”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“code mods being made available to AI. So that combination is really powerful”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“that's a really common pattern right now is using AI for migration”
Probably right AI guessMigration case studies (Google, Amazon Java upgrades, Airbnb) got their scale from pairing LLMs with deterministic tooling and validation rather than free-form generation.
Leans right Google migrations report lines 619-641Google migration experience report: simple LLM prompting alone only works for the simplest migrations; AST tooling, heuristics and LLMs together are needed, and AST checks catch model mistakes cheaply and deterministically.

Coding agents sometimes game tests by deleting or editing them or hard-coding expected outputs instead of fixing code

Leans right

Reward hacking on tests (special-casing, weakening or deleting tests) is documented by labs and in agent evaluations, though frequency varies by model and setup.

· denied 0 ·

Asserts (11)Denies (0)Verdicts (1)

AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“after the third iteration close just give ups and then comments your test and says return true to true true”
TDD, AI agents and coding with Kent Beck (The Pragmatic Engineer)“I look at the code, and it's just a lookup table. It says if this is the input string, here's the output string”
o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“Oh, you commented out a bunch of tests. We don't do that here.”
Are Harnesses Worth the Cost? (martinfowler.com)“Allowing it to change the expected results when tests fail can also obscure a defect.”
An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“don't game the benchmarks by overfitting on the benchmark inputs alone”
Learning to Code with AI and Steve Klabnik (Scott Hanselman)“set up a calculate size function that had basically done a pattern match on the name of the strruct field”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“"Please make sure all the tests pass." And like some, you know, deletes the tests.”
Nobody Agrees About Agentic Development (Convex)“I love that you can't do TDD, because it will just change the tests.”
AI Just Crossed the Terrifying Line - Now What? (Kurzgesagt – In a Nutshell)“Maybe edit the test conditions or the questions you are supposed to answer, or just look up the solution instead of doing the work”
Killing Coding Agent Slop With Adversarial Self-Play (Telos)“Smarter models reward hack more, especially over long horizons.”
Beyond the Hype: Real Talk on AI-Assisted Development • Jessica Kerr & Diana Montalion • GOTO 2025 (GOTO Conferences)“passing tests or deleting tests if you let it get away with that, making everything look green”
Very likely right AI guessReward hacking on tests (special-casing, weakening or deleting tests) is documented by labs and in agent evaluations, though frequency varies by model and setup.

Specs-to-code workflows that ignore the generated code do not work; developers must keep understanding and shaping the code

Leans partly right

Ignoring code entirely leads to drift and debt in real codebases; but spec-driven approaches with code review do work.

· ·

Asserts (8)Denies (2)Verdicts (3)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“I tried this. I really tried it. And it sucks.”
Getting the most out of Codex (Academind)“always make sure ... that you don't detach from it ... and then you're essentially forced to do vibe coding”
Agentic Engineering in Action with Mitchell Hashimoto (Zed Industries)“the more vague and the longer you let AI spend on something, uh, the more of a mess it's going to make”
Understanding is the new bottleneck (geoffreylitt.com)“Hot take: I think it's still important to understand the code that our agents write!”
</> htmx ~ Code is Cheap(er) (htmx.org)“developers still need to understand the underlying code even if it is generated by an LLM.”
The Law of Leaky Abstractions (Joel on Software)“Code generation tools which pretend to abstract out something, like all abstractions, leak”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“nobody writes any code, nobody reads any code, which is clear insanity.”
"Do You Still Read the Code?" • zanlib (zanlib)“Vibecoders seem to bet that the theory can be built by specifying and running software, without dealing with its implementation. I am not so sure”
DENIES Control the ideas, not the code - <antirez> (antirez.com)“Nobody should anymore look at this code, but only at the ideas the code contains.”
DENIES Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“beads is like living proof that you never actually have to look at the code”
Probably partly right AI guessIgnoring code entirely leads to drift and debt in real codebases; but spec-driven approaches with code review do work.
Leans partly right Vibe-coding literature review lines 2076-2093The review recommends reviewing generated code before committing rather than integrating unread output, and warns that deferred understanding (cognitive debt) raises later maintenance costs; this is recommendation-level evidence.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 917-940The authors warn that a loss of understanding may hurt review, debugging and maintenance. They also concede that future workflows handing those jobs to agents might make low-level comprehension matter less. So the paper backs the concern only speculatively.

System design and architecture judgment remain human work that AI cannot do without human-supplied context

Leans partly right

Humans still own constraints and tradeoffs, but models can now propose reasonable designs when given context; the gap is narrowing.

· denied 0 ·

Asserts (11)Denies (0)Verdicts (3)

AI Replaced 75% of Work for Software Engineers & Data Scientists - here's what's left (Sundas Khalid)“system design is a place where soft engineers will thrive”
The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“move up a level of abstraction start worrying about systems”
Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“Right now AI is not strong about the big picture.”
Agentic Engineering in Action with Mitchell Hashimoto (Zed Industries)“I'm more or less like the architect of the software project.”
The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“system architecture, creating reliability in the system, they are not just very good at.”
Control the ideas, not the code - <antirez> (antirez.com)“LLMs are very good at writing locally optimal code, and are worse (but improving) with big ideas.”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“telling an agent and it doesn't really matter which one to create nice software architecture I don't think it really works”
</> htmx ~ Yes, and... (htmx.org)“I think software architecture will become a more important skill over time”
AI Will Replace Software Engineers? Here’s What’s Actually Happening (Chris Schwenk | Tech Jobber)“go up the stack on purpose. System design, architecture”
Using AI to write better code more slowly (Read the Tea Leaves)“Agents are not terribly good at architecture”
Nobody Agrees About Agentic Development (Convex)“Architecting a system is harder now, because the agent never gets it quite right.”
Leans partly right AI guessHumans still own constraints and tradeoffs, but models can now propose reasonable designs when given context; the gap is narrowing.
Leans partly right Anthropic internal study lines 80-80Anthropic: engineers keep high-level design and 'taste' tasks, but call that boundary a moving target, and Claude Code design/planning usage grew from 1% to 10% in six months.
Leans partly right Vibe-coding literature review lines 1364-137420 of 47 sources (43%) describe expert use in which developers speed up routine implementation 'while retaining responsibility for architectural reasoning and verification'; this is descriptive, not a test of AI design ability.

AI productivity gains concentrate among developers new to a codebase and on routine, well-trodden code

Probably right

Slowdown is larger when developers have high prior task exposure. Developers report AI helps on unfamiliar issues and not where they're experts, and the paper expects speedups in unfamiliar or greenfield settings.

· denied 0 ·

Asserts (10)Denies (0)Verdicts (8)

AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“these participants were just deeply familiar with the repos ... which is where AI helps a bit less”
Why Agent Hype can fall short of reality – Joel Becker, METR (AI Engineer)“these developers are coming to these problems already knowing the solution”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“It seems totally plausible to me that by the channel that you're pointing at um junior developers will be will be sped up more.”
The METR Paradox. Why AI Made Senior Devs 19% Slower (Macro Lens)“New engineers in code bases they do not know yet. And routine code”
Is AI making devs SLOWER? Yes, according to this study... (Native Notify)“26.08% increase in completed tasks ... specifically if they're less experienced developers”
Does GitHub Copilot Improve Code Quality? Here (Jadarma)“improvements were observed in the place we'd most expect.”
How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR (AI Engineer)“some of the repos that I was helped the most with were ones that I was completely unfamiliar with”
METR's AI productivity study is really good (seangoedecke.com)“the real niche of AI is when developers have to do tasks that are a little out of their comfort zone.”
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Simon Willison’s Weblog)“these developers were all working in repositories they have a deep understanding of already”
Does AI Secretly Slow Developers Down? (The AI Daily Brief: Artificial Intelligence News)“Expert developers working in large code bases is known to be the setting where AI can help least”
Probably right AI guessField experiments (Microsoft/Accenture RCTs, Brynjolfsson) found the largest gains for less-experienced or less-familiar workers and routine tasks; METR found slowdowns for deep-familiarity experts.
Probably right METR 2025 RCT lines 960-984Slowdown is larger when developers have high prior task exposure. Developers report AI helps on unfamiliar issues and not where they're experts, and the paper expects speedups in unfamiliar or greenfield settings.
Leans partly right Microsoft/Accenture field RCTs lines 758-770Microsoft experiment: gains were larger for short-tenure (newer to the company and its code) and junior developers (27-39% vs 8-13%), but differences were not significant and the routine-code half of the claim was not tested.
Leans partly right Copilot RCT (Peng) lines 174-177Copilot RCT: less experienced developers benefited more on a routine, well-trodden task (a basic HTTP server), consistent with the claim but from a small freelancer sample and without a codebase-familiarity test.
Leans partly right Anthropic internal study lines 70-70Anthropic: staff delegate where they have low context and complexity is low, and avoid tasks needing tacit codebase knowledge, which mirrors METR's slowdown factors; this concerns task choice rather than measured gains.
Leans partly right Shan et al. field study + RCT lines 872-878Less-experienced developers gained more speed in the RCT, consistent with the claim, but a follow-up on harder APPS tasks suggested quantity gains were larger on complex tasks (0.283 vs 0.245), which does not fit 'routine code only'.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 240-247Large completeness gains (+43 points) came on small, popular-domain tasks chosen so that LLMs could help, many of them greenfield. This fits gains on routine code, but the study had no mature codebases to compare against.
Leans wrong Google RCT lines 490-505Google RCT on code unfamiliar to all participants found, if anything, more senior engineers gained more from AI (non-significant interaction), contrary to gains concentrating among less experienced developers.

Large AI-generated pull requests exceed what human reviewers can effectively check

Leans right

The SmartBear/Cisco review study found defect detection drops beyond ~200-400 LOC per session; AI makes large PRs cheap to produce.

· denied 0 ·

Asserts (10)Denies (0)Verdicts (4)

The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“PR sizes have exploded anywhere up to 154%”
Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“it's not useful to me to get a diff of 10,000 lines of code to my repo ... I'm still the bottleneck”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“it is a real concern if teams get too much uh code to review that things just stall”
AI in engineering: Q2 2026 benchmarks & research readout (DX)“as PRs get bigger. It's we're a lot more likely to to rubber stamp them”
Using AI Code Generation to Migrate 20000 Tests (InfoQ)“if for example your PR has a lot more code then it would take a longer time for other people to review it”
The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“this pull request that looks really daunting and has 5,000 lines of code in it”
Control the ideas, not the code - <antirez> (antirez.com)“How are you supposed to review 5k lines of code every day?”
How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“you might generate 10,000 lines of code for a very simple uh application feature that now the burden on engineering to read”
Killing Coding Agent Slop With Adversarial Self-Play (Telos)“human review doesn't scale to LLM volumes. At this point, my coding agents produce code much faster than I can review it.”
AI Made Devs 19% Slower (METR 2025 Study) (zencoderai)“reviewers stop finding bugs after about 400 lines of code”
Probably right AI guessThe SmartBear/Cisco review study found defect detection drops beyond ~200-400 LOC per session; AI makes large PRs cheap to produce.
Leans partly right Vibe-coding literature review lines 1906-1918The vibe-coding review names a verification gap between generation speed and developers' capacity to verify. It doesn't measure PR size.
Leans partly right Google migrations report lines 377-389Google splits AI migration changes into small sharded changelists per code owner and caps weekly volume to avoid overwhelming reviewers, which implies very large AI changes are not reviewable as-is, though it never measures review quality.
Leans partly right Debt study (302k AI commits) lines 143-146Authors note Claude made 3,957 commits and ~500K added lines in Anthropic's claudes-c-compiler within weeks and infer it is unlikely all of it got thorough human review; an argument from volume, not a measurement of review effectiveness.

Coding agents degrade codebase quality when they keep extending the same codebase, and newer models haven't fixed this

Leans right

Consistent with long-horizon benchmarks such as SlopCodeBench and with studies of AI-code churn and duplication. Evidence is early but points the same way.

· denied 0 ·

Asserts (10)Denies (0)Verdicts (2)

DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“a lot of PRs that individually perhaps could have been justified for a hot moment, taken all together, destroyed the architecture of the system”
The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“you're very quickly creating a code base of the size and complexity that the agent itself can no longer dig itself out from.”
SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“they still saw technical debt compound”
12 Months of Unity Dev with Claude Code (Mythmatic)“eventually, it will just be very average. That is not a fun workflow.”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“every time they made a change, they were more likely to break something.”
Sustainable Augmented Development • Kent Beck • YOW! 2025 (GOTO Conferences)“the genie loves making this mistake so much faster than humans can do it”
Nobody Agrees About Agentic Development (Convex)“agents aren't much help with proper engineering, because they keep breaking the abstractions, the core structure. They seem to love spaghetti code.”
A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“if you take the second law of thermodynamics, everything decays towards chaos”
Killing Coding Agent Slop With Adversarial Self-Play (Telos)“By checkpoint three, the one-shot agent on file_backup passed only 16% of the earlier checkpoints' tests”
Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“as the agent keeps working on the same codebase, the quality gets worse and worse”
Leans right AI guessConsistent with long-horizon benchmarks such as SlopCodeBench and with studies of AI-code churn and duplication. Evidence is early but points the same way.
Leans partly right Debt study (302k AI commits) lines 850-854Surviving AI-introduced issues keep accumulating in real repos over time, consistent with gradual degradation; the study doesn't test repeated extension of one codebase or model generations.

A large share of code at major companies is now written by AI

Leans right

Google reports that AI completion assisted 67% of code characters (38% acceptance rate), so more characters are AI-completed than typed by hand; note this is a completion metric, not autonomous authorship.

· ·

Asserts (8)Denies (1)Verdicts (4)

Building Blocks for Uber’s Software Factory— Uday Kiran Medisetty & Adam Huda, Uber (AI Engineer)“more than 70% of our PRs now either by local or cloud agents.”
Anthropic CEO: "We Might Be 6 To 12 Months Away" From AI Doing A Software Engineer's Job (CAPITALMODE)“engineers within Anthropic who say, I don't write any code anymore”
Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit (The Pragmatic Engineer)“we're at about 26.9% industrywide for all of these developers”
The TRUTH About the Software Development Job Market in 2026 (Internet Made Coder)“41% ... of all the code being written by the biggest companies in the world is currently being written by AI”
AI in engineering: Q2 2026 benchmarks & research readout (DX)“50% of merged code uh is now AI authored”
AI assistance is leading to lower code quality, claim researchers (devclass)“46 percent of code was completed by Copilot in files where it was enabled”
How To Solve It With Code—background on the course; a discussion with Jeremy and Hamel (Jeremy Howard)“Google CEO said that most recent the most recent few months 25% of their code is generated by AI”
AI in software engineering at Google: Progress and the path ahead (research.google)“assisting in the completion of 50% of code characters”
DENIES Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“those headlines suggest that 30% of Microsoft's code that's running in production was authored by AI. That is not at all realistic.”
Leans partly right AI guessGoogle and Microsoft executives cite ~25-50% of new code AI-generated; the figures are self-reported and usually count accepted suggestions, not finished code.
Leans right Google migrations report lines 86-95Google reports that AI completion assisted 67% of code characters (38% acceptance rate), so more characters are AI-completed than typed by hand; note this is a completion metric, not autonomous authorship.
Leans partly right Debt study (302k AI commits) lines 59-60Citing company disclosures: Google and Microsoft said in 2025 that AI writes over 20% of their new code. This is secondhand and a meaningful but not majority share.
Leans right Echoes of AI maintainability RCT lines 73-75Cites public statements from Microsoft, Meta and Google that roughly a third of their new code is AI-generated; secondhand.

Most professional developers now use AI coding assistants

Leans right

Stack Overflow 2025: ~84% use or plan to use AI tools, about half of professionals daily; JetBrains and DORA report similar figures.

· denied 0 ·

Asserts (9)Denies (0)Verdicts (4)

2025 DORA report: Insights to impact with Google and Thoughtworks (Thoughtworks)“90% of the people in our survey have said yes they are using AI in their professional work”
Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev) (The Pragmatic Engineer)“the median organization has about 50%. Five out of 10 use it on a weekly basis.”
Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit (The Pragmatic Engineer)“sitting around 92.6 of developers are using an AI coding assistant at least once a month”
DORA Community Discussion - State of AI-assisted Software Development (DORA)“Google says 90% of of AI uh 90% of of software developers are using AI at work.”
AI in engineering: Q2 2026 benchmarks & research readout (DX)“it's now pretty settled that like yes, virtually everyone is is using AI”
The AI Code Problem Nobody's Talking About (Razorwire Cybersecurity)“over 70% of developers are now using some form of AI coding assistance”
DORA 2025: Measuring Software Delivery After AI (Alt + E S V)“In 2025, 90% of respondents reported using AI within their job function”
DORA Community Discussion - Decoding DORA 2025: AI’s Real Impact (DORA)“we found that 90% of people are using AI to some extent”
How AI Is Reshaping Software Development Insights from the 2025 DORA Report | AppDevANGLE (SiliconANGLE theCUBE)“with 90% of developers using AI daily”
Very likely right AI guessStack Overflow 2025: ~84% use or plan to use AI tools, about half of professionals daily; JetBrains and DORA report similar figures.
Leans partly right Debt study (302k AI commits) lines 38-39Cites the 2025 Stack Overflow survey: 84% of professional developers are using or planning to use AI coding tools; the 'or planning' wording overstates current use.
Leans partly right Echoes of AI maintainability RCT lines 1197-1204In the volunteer prescreen at the end of 2024 (N=449, 92% professionals), 73.6% had prior experience with AI assistants, which is not the same as current regular use, and the sample was self-selected.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 377-382All 69 participants reported having used AI tools for software tasks before. This is a convenience sample of mostly students in three European universities and four companies, so it is weak evidence about professionals at large.

In real enterprise codebases AI yields modest average productivity gains, far below headline claims

Probably right

Field RCTs with 4,867 developers at Microsoft, Accenture and a Fortune-100 firm: Copilot raised weekly completed PRs by 26% (SE 10.3%), substantially smaller than the lab's 56% speedup, which the authors attribute to real work being less assistable.

· denied 0 ·

Asserts (9)Denies (0)Verdicts (6)

AI at Honeycomb: What’s Actually Working (w/ CTO Charity Majors) (Jam)“it hasn't transformed our team's velocity. It's made us a little slower in some places, a little faster in others.”
Stanford Study: AI Makes Developers 20% More Productive (But There's a Catch) (Josh Anderson)“you're going to be 15 to 20% more effective using AI”
Measuring AI code assistants and agents with the AI Measurement Framework (DX)“AI is not a magic silver bullet”
AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“the median productivity gain you can expect is about 10%”
Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit (The Pragmatic Engineer)“We're sitting around 4.08 uh self-reported hours saved due to AI tool usage per week per developer.”
AI in engineering: Q2 2026 benchmarks & research readout (DX)“We're still not seeing 2x, 5x, 10x, but we are seeing improvement.”
How much does AI impact development speed? (newsletter.getdx.com)“AI tools may reduce task completion time by around 21% in enterprise settings”
Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford (AI Engineer)“an average productivity gain across all industries ... of roughly about 15 to 20%”
Measuring the actual impact of AI coding | Abi Noda from DX (Changelog)“On average developers report saving about three hours per week thanks AI tools.”
Probably right Google RCT lines 519-528Google RCT, 96 full-time engineers on an enterprise-grade C++ task in the monorepo: AI cut time on task by ~21% in the best-fit model (95% CI crosses zero, p=0.086), well below Peng et al.'s 56% and in line with Cui et al.'s 26%.
Probably right Microsoft/Accenture field RCTs lines 774-784Field RCTs with 4,867 developers at Microsoft, Accenture and a Fortune-100 firm: Copilot raised weekly completed PRs by 26% (SE 10.3%), substantially smaller than the lab's 56% speedup, which the authors attribute to real work being less assistable.
Probably right AI guessThe Stanford 100k-developer study and other field data put average gains around 10-20%, far below 2x-10x marketing claims.
Leans partly right METR 2025 RCT lines 995-1001METR 2025: in large, mature repos (~10 years old, >1.1M LOC) AI helped less than in greenfield RCTs, and on average slowed developers by 19%; this goes beyond 'modest gains', and the repos are open-source rather than enterprise.
Leans partly right Shan et al. field study + RCT lines 489-493Field DID at a 5,000-person IT services firm (201 devs, 27 weeks, 2023-24 Copilot/ChatGPT/Gemini): AI users completed 1.08 more user stories per week (+18%), a real gain but modest next to 55% or 10x headlines; usage was self-reported and partly self-selected.
Leans partly right Google migrations report lines 245-251Google reports large end-to-end savings (about 50% on int64, about 89% on small Joda clusters) in a 500M+ LOC codebase, but only for bespoke migration pipelines with fine-tuned models, not average developer work, and the figures are engineers' estimates.

AI helps little, and can hurt, when working in less popular programming languages

Leans right

Model quality tracks training-data volume; benchmarks such as MultiPL-E show much lower scores in less common languages.

· ·

Asserts (8)Denies (1)Verdicts (1)

AI Engineering at Jane Street - John Crepezzi (AI Engineer)“models themselves are just not very good at oaml”
Armin Ronacher Leaning In To Find Out - PyAI Conf 2026 (Pydantic)“Lisp for instance, and I discovered it just doesn't really work”
AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“for your more like niche low resource languages ... you get hardly anything”
Agentic Engineering in Action with Mitchell Hashimoto (Zed Industries)“anything more than trivial changes to Zig code bases. Like it's still hopeless uh with that language.”
My AI Skeptic Friends Are All Nuts (Fly)“People say "LLMs can't code" when what they really mean is "LLMs can't write Rust".”
How Google is accelerating code migrations with AI (newsletter.getdx.com)“Context window constraints, hallucinations, and variable performance across programming languages sometimes require manual intervention.”
Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford (AI Engineer)“for low language popularity and complex tasks, AI can actually decrease productivity”
AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“python JavaScript they're great at rust they're not quite as good at yet”
DENIES An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“using LLMs in Rust on some fun pet projects, and my results were far better than I expected”
Probably right AI guessModel quality tracks training-data volume; benchmarks such as MultiPL-E show much lower scores in less common languages.

As implementation gets cheap, product taste and judgment become the main differentiator

Leans partly right

Plausible economic logic and widely argued, but not measured; execution, distribution and maintenance still matter.

· denied 0 ·

Asserts (9)Denies (0)Verdicts (1)

AI at Honeycomb: What’s Actually Working (w/ CTO Charity Majors) (Jam)“Doesn't matter how fast you're going if you aren't working on things that really change your users's lives.”
Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“the people with taste are the ones ... because the difficulty of producing it is going to go so far down”
DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“They're bottlenecked on ideas. They're bottlenecked on vision. They're bottlenecked on taste.”
Agentic coding changes everything | Thorsten Ball from Sourcegraph Amp (Changelog)“But what to build and when and how to say organize it, how to architect it ... That's now super valuable.”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“that is probably going to need more and more people with a product mindset”
"STOP GETTING COMPUTER DEGREES" - Sam Altman's SHOCKING Interview (AI Upload)“the skills that you need in that world are figuring out what people want sort of Creative Vision quick adaptability”
TDD, AI agents and coding with Kent Beck (The Pragmatic Engineer)“90% of my skills just went to zero dollars and 10% of my skills just went up a,000x.”
My AI Skeptic Friends Are All Nuts (Fly)“even the most Claude-poisoned serious developers in the world still own curation, judgement, guidance, and direction”
Sustainable Augmented Development • Kent Beck • YOW! 2025 (GOTO Conferences)“90% of my skills just went to $0 and 10% of my skills just went up by a,000x”
Leans partly right AI guessPlausible economic logic and widely argued, but not measured; execution, distribution and maintenance still matter.

Test-driven development (red-green-refactor) makes AI agents produce better code and tests

Leans partly right

Tests-first gives agents a target and limits test-cheating; evidence is mostly anecdotal and it does not suit all code (e.g. UI).

· ·

Asserts (7)Denies (1)Verdicts (1)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“TDD I found is absolutely essential for getting the most out of agents”
Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“implement a test first um a red test because we are doing test driven development”
AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“the obvious solution that fixed all of those these problems to some degree is test-driven development”
Why AI WON'T Replace Software Engineering... (Modern Software Engineering)“we need to teach it TDD”
TDD, AI agents and coding with Kent Beck (The Pragmatic Engineer)“That's how I work all the time. So so so when you work with your genie, you start with the tests.”
Test-driven development (TDD) with Claude Code (James Murdza)“why you should be using test-driven development when you are coding with AI”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“the chances of you getting code that works go up so much if they're if they're if they're writing the test first.”
DENIES TDD inside the agent loop - theater or actual value? (martinfowler.com)“there was no clearly discernable difference based on TDD workflow versus no TDD workflow”
Probably partly right AI guessTests-first gives agents a target and limits test-cheating; evidence is mostly anecdotal and it does not suit all code (e.g. UI).

Cheaper software production will increase total demand for developers (Jevons paradox)

Leans partly right

Jevons effects are historically common and plausible for software, but whether demand growth outpaces productivity for developer headcount is unproven.

· denied 0 ·

Asserts (8)Denies (0)Verdicts (1)

How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“demand is way more than we can imagine”
Junior Dev Jobs Are Down 20%… So Why Are Companies Hiring? (HTML All The Things)“the more we democratize the creation of software ... the more companies that will require more people”
The TRUTH About the Software Development Job Market in 2026 (Internet Made Coder)“making something more efficient often increases total consumption of that thing”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“My suspicion is that generally speaking there will be more programmers because it has become easier to create programs”
AI Isn't Replacing Developers - Here's Why (Stefan Mischook)“Jevons paradox will kick in ... a lot more software is going to be created”
How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“I still believe this is Jevan's paradox. The more we get, the more we want.”
AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“that means that the number of jobs of software Engineers goes up”
Measuring the actual impact of AI coding | Abi Noda from DX (Changelog)“the idea that when a resource becomes more efficient, it actually leads to higher utilization.”
Leans partly right AI guessJevons effects are historically common and plausible for software, but whether demand growth outpaces productivity for developer headcount is unproven.

AI-generated code duplicates existing utilities and types instead of reusing them

Leans right

GitClear's 2024-25 analyses found rising duplicated code and falling refactoring with AI adoption.

· denied 0 ·

Asserts (8)Denies (0)Verdicts (2)

AI-Generated Code Is Creating a New Kind of Technical Debt (Imran Gardezi)“the AI writes a new date formatting function, different name ... same purpose”
Using AI Code Generation to Migrate 20000 Tests (InfoQ)“longterm it could be problematic because there is a lot of duplication. Some of the abstractions are not created.”
The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“It's creating code in a new file that has already done somewhere else.”
AI assistance is leading to lower code quality, claim researchers (devclass)“no greater scourge to long-term code maintainability.”
Is AI making devs SLOWER? Yes, according to this study... (Native Notify)“code generated with the help of AI had eight times more duplicate code”
Does GitHub Copilot Improve Code Quality? Here (Jadarma)“A significant increase in copy-paste, meaning more duplication”
The AI-Native Data Engineer featuring Fabric, dbt, VSCode devcontainers and copilot (Microsoft Reactor)“I don't want the LM to like take the entire wheel and reinvent it.”
A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“they're going to write duplicated code left and right”
Probably right AI guessGitClear's 2024-25 analyses found rising duplicated code and falling refactoring with AI adoption.
Leans partly right Echoes of AI maintainability RCT lines 2003-2013New Code Duplication smells were introduced by 23% of AI-devs vs 11% of non-AI devs, but authors call the differences minor (high variance, identical medians).

Goal or loop commands that re-prompt an agent until a goal is met let it work through a backlog unattended

Leans partly right

Goal/loop features keep agents going through backlogs, but unattended runs drift or game checks without good verification; works when acceptance tests are strong.

· ·

Asserts (6)Denies (2)Verdicts (2)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“implementation here is really the only AFK bit”
Ralph Wiggum: Continuous Agent Loops in Claude Code (AI Makerspace)“when Claude tries to stop, we say, No, no, no. It's not time to stop”
Ralph Wiggum killed programming (Ras Mic)“a robust way to let AI autonomously build applications”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“I will not need to babysit this agent”
FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“a loop that kind of continue continuation loop where we continuously nudge the agents to keep working.”
The AI-Native Data Engineer featuring Fabric, dbt, VSCode devcontainers and copilot (Microsoft Reactor)“Ralph, until the steak is cooked exactly to the precise temperature with this thermometer that I've asked you for, do not stop.”
DENIES Getting the most out of Codex (Academind)“I still don't loop. I don't use the Rolf loop thing ... I stay in control”
DENIES My AI Coding Workflow That Actually Works (Chris Titus Tech)“if you want to just be like, hey, I want just one prompt and have it loop a whole bunch ... that's not really a good idea”
Leans partly right AI guessGoal/loop features keep agents going through backlogs, but unattended runs drift or game checks without good verification; works when acceptance tests are strong.
Leans partly right Airbnb migration lines 19-22Airbnb's custom loop runner re-prompted the LLM with errors until a step passed or a retry cap was hit, and the first bulk run migrated 75% of 3.5K files in 4 hours; this is a bespoke pipeline, not a generic agent goal command.

Software built quickly with AI accumulates more technical debt than carefully hand-built software

Leans partly right

In 302k AI-authored commits, 22.7% of AI-introduced issues persist, and surviving debt grew past 100k issues. But there's no human-written baseline, so 'more than hand-built' is untested.

· denied 0 ·

Asserts (8)Denies (0)Verdicts (5)

AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“their code quality decreased by about 9% ... their code quality became more erratic”
Does AI-driven productivity come at the cost of code quality? (Proxify)“it creates technical debt and a decrease in code quality which then means that for the subsequent months your productivity slows down”
The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“we're producing months and months of technical debt in the in the in the time of weeks, in the time of days sometimes.”
How To Solve It With Code—background on the course; a discussion with Jeremy and Hamel (Jeremy Howard)“getting into this bigger and bigger hole of technical debt as it writes more and more code that you don't understand”
Does GitHub Copilot Improve Code Quality? Here (Jadarma)“A significant decrease in moved code, meaning less refactoring, less cleaning.”
AI Coding and Technical Debt (NT Productions)“the lack of upfront work that you're putting in is going to make it more difficult down the line”
Nobody Agrees About Agentic Development (Convex)“with AI there's a point where the drift gets too large, the complexity grows a lot, and then the cost of that drift becomes huge”
Vibe code is legacy code (Val Town)“When you vibe code, you are incurring tech debt as fast as the LLM can spit it out.”
Leans partly right AI guessUnreviewed, fast AI output does accumulate debt (GitClear duplication, churn), but debt depends on review discipline, not on whether AI was used per se.
Leans partly right Debt study (302k AI commits) lines 855-864In 302k AI-authored commits, 22.7% of AI-introduced issues persist, and surviving debt grew past 100k issues. But there's no human-written baseline, so 'more than hand-built' is untested.
Leans wrong Echoes of AI maintainability RCT lines 2897-2906Preregistered RCT (Phase 2, N=75, 95% professionals): code co-developed with AI was no harder to evolve manually, with no significant difference in time, CodeHealth or coverage, and a small Bayesian CodeHealth gain for habitual AI users. Caveats: small Java app, short-term, underpowered for small effects.
Leans partly right Vibe-coding literature review lines 1434-1452Maintainability concerns appear in 23% (11/47) of sources (e.g. WL27: conversational code 'lacks cohesive structure... complicating future maintenance'), but the review stresses that long-term and longitudinal evidence is limited and that there is no hand-built baseline.
Leans right Shan et al. field study + RCT lines 1219-1229Self-reported technical debt was higher among AI users in the follow-up experiment, consistent with the claim but based on perception, cross-sectional data and short-term evidence.

Commit and PR counts overstate AI productivity because AI increases rework of recent code

Leans right

Stanford and GitClear both found increased short-term churn/rework with AI, so raw commit/PR counts inflate apparent gains.

· denied 0 ·

Asserts (8)Denies (0)Verdicts (3)

Stanford Study: AI Makes Developers 20% More Productive (But There's a Catch) (Josh Anderson)“more commits doesn't mean more better ... some of those commits are cleanup”
AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“commit count increases because you're making commits to fix the bugs introduced by AI”
AI assistance is leading to lower code quality, claim researchers (devclass)“They also see an increase in churn, now at 7.1 percent compared to just 3.3 percent in 2020.”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“outcome measures that um uh that I that I don't have a huge amount of faith in things like you know lines of code”
Does GitHub Copilot Improve Code Quality? Here (Jadarma)“A SIGNIFICANT increase in churn rate, meaning code once commited was much more likely to be altered soon thereafter”
Real artists still ship (Jerod Santo)“HUGE productivity boost, MINOR shipping boost.”
The Truth About Developer Productivity in the AI Age (IT'S A TRAP) (Modern Software Engineering)“There's more rework going into that code than ever happened before.”
Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford (AI Engineer)“by implementing AI, you get a bunch more of rework”
Probably right AI guessStanford and GitClear both found increased short-term churn/rework with AI, so raw commit/PR counts inflate apparent gains.
Leans partly right METR 2025 RCT lines 1122-1130METR warns that lines of code and task counts can rise without productivity (more verbose but equivalent code). Developers wrote 47% more code per hour with AI (not significant) while being slower. This doesn't cover the rework mechanism.
Leans partly right Microsoft/Accenture field RCTs lines 525-530Builds rose 38% vs 26% for PRs; authors note this could reflect trial-and-error coding that lowers quality, but build success rate fell significantly only at Accenture (-17%), pooled -5.5% NS, so rework is suggested not shown.

Background coding agents assigned routine issues can deliver mostly mergeable pull requests

Leans partly right

Background agents do produce usable PRs for well-scoped chores, but merge rates vary widely and many PRs need iteration; 'most of the time' is optimistic.

· denied 0 ·

Asserts (8)Denies (0)Verdicts (1)

How the GitHub Copilot coding agent works | GitHub Checkout (GitHub)“most of the time I'm going to get a PR that's good and ready to go”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“engineers can do toil work, maybe a library upgrade or handling a deprecation or fixing a flaky test um with some of the agentic AIs”
Inside minions: how Stripe uses autonomous one-shot AI coding agents (Stripe Developers)“we have at the order of like thousands of PRs now being created by minions”
My AI Skeptic Friends Are All Nuts (Fly)“Five of them get the same feedback a junior dev gets. And five get merged.”
Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“the accuracy of these tasks was much higher compared to the more ambiguous workloads.”
Google Jules 3.0 UPDATE: FULLY FREE Async AI Coder IS INSANELY GOOD! (Jules API, Tools, CLI, & More) (WorldofAI)“it works asynchronously, meaning that you can move on to other tasks while it takes care of the coding”
How Coinbase scaled AI to 1,000+ engineers | Chintan Turakhia (How I AI)“we built this internal bot. I'm excited to show you. And within seconds like the PR is being authored, right? an agent picks it up”
Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“janitor bots that run overnight and open small PRs”
Leans partly right AI guessBackground agents do produce usable PRs for well-scoped chores, but merge rates vary widely and many PRs need iteration; 'most of the time' is optimistic.

Higher AI adoption is associated with higher delivery throughput but also more delivery instability

Leans right

DORA 2025 reports AI adoption now positively associated with throughput while still associated with higher delivery instability (change failure and rework).

· denied 0 ·

Asserts (8)Denies (0)Verdicts (1)

2025 DORA report: Insights to impact with Google and Thoughtworks (Thoughtworks)“This year we see throughput improving and stability dropping off”
DORA Community Discussion - State of AI-assisted Software Development (DORA)“we continue to see an increase in software delivery instability ility in relationship to AI adoption”
AI in engineering: Q2 2026 benchmarks & research readout (DX)“the DORA metric that's associated with with velocity uh deployment frequency we also see increasing”
DORA 2025 AI assisted dev report: Some Benefit, Most Don’t (Rob Bowley)“any gains in delivery speed are largely cancelled out by a rise in the change failure rate (CFR)”
DORA 2025: Measuring Software Delivery After AI (Alt + E S V)“AI improves outcomes at nearly every level except system stability.”
DORA Community Discussion - Decoding DORA 2025: AI’s Real Impact (DORA)“when we use more AI, it seems to be making things a bit more unstable.”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“in the Dora report this was one of the big like let's say negative trend findings about stability actually getting worse”
How AI Is Reshaping Software Development Insights from the 2025 DORA Report | AppDevANGLE (SiliconANGLE theCUBE)“We're able to ship things more quickly and more frequently, but we still see those challenges with software delivery instability.”
Probably right AI guessDORA 2025 reports AI adoption now positively associated with throughput while still associated with higher delivery instability (change failure and rework).

Cutting junior developer hiring because of AI undermines the future pipeline of senior engineers

Leans right

A straightforward pipeline argument made by IBM and others; the long-run effect is unmeasured yet but well reasoned.

· denied 0 ·

Asserts (8)Denies (0)Verdicts (1)

Jensen Huang Said "Stop Learning to Code" — Then Hired 10,000 Developers (Olive Badger)“If you don't bring in early talent now, you won't have mid-level or senior engineers in 5 to 10 years”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“make sure that we don't inadvertently dry up our talent pipeline”
Junior Dev Jobs Are Down 20%… So Why Are Companies Hiring? (HTML All The Things)“you're not going to get any more any more seniors if you don't hire and train juniors”
AI Layoffs Don’t Add Up. Here’s the Data (HuMan Intelligence)“No entry-level roles means no mid-level talent in five years”
</> htmx ~ Yes, and... (htmx.org)“And companies: let the juniors write at least some of the code. It is in your interest.”
Are Junior Developers Really Getting Replaced by AI? I'm not sold... (Brian Jenney)“the firing of junior developers because AI can replace them is ... the dumbest thing I've ever heard”
Sustainable Augmented Development • Kent Beck • YOW! 2025 (GOTO Conferences)“I think juniors are more valuable than they ever have been.”
How AI Is Reshaping Software Development Insights from the 2025 DORA Report | AppDevANGLE (SiliconANGLE theCUBE)“organizations that are trying to use AI as an alternative to hire the junior engineers is probably not the best approach”
Leans right AI guessA straightforward pipeline argument made by IBM and others; the long-run effect is unmeasured yet but well reasoned.

As AI speeds up coding, deciding what to build (product management) becomes the bottleneck

Leans right

Consistent with Amdahl-style reasoning and reports of shifting PM:engineer ratios; evidence is mostly anecdotal from AI-native teams.

· denied 0 ·

Asserts (8)Denies (0)Verdicts (1)

Building Blocks for Uber’s Software Factory— Uday Kiran Medisetty & Adam Huda, Uber (AI Engineer)“we can probably build it now it's more of a question of should we build it”
Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“all these model capabilities and benchmarks and all the intelligence gets translated into just like can you build something people want”
The AI Productivity Paradox: Why 10X Output Doesn't Mean 10X Business Outcome - Mik Kersten (Tech Lead Journal)“are outside of development tradition, like product management”
Andrew Ng: Building Faster with AI (Y Combinator)“I've heard teams start to propose PM to engineering ratio of one to 0.5”
How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“we can ship trash faster every single day. We need strategy and really smart decisions to know what to ship”
AI Dev 26 x SF: Andrew Ng: The Future of Software Engineering (DeepLearningAI)“deciding what to build or the product management work becomes the new bottleneck”
Measuring the actual impact of AI coding | Abi Noda from DX (Changelog)“product management is actually the big bottleneck not so much like engineering ing velocity”
Discussing the State of AI-assisted Software Development with DORA and EarnIn (Datadog Community)“that really uh kind of pushes the bottlenecks almost to the very beginning of the process”
Leans right AI guessConsistent with Amdahl-style reasoning and reports of shifting PM:engineer ratios; evidence is mostly anecdotal from AI-native teams.

Coding agents can help developers understand codebases and review code more thoroughly, not less

Leans partly right

Agents do speed onboarding and PR summarization, but surveys and the Anthropic learning study show many users skip understanding; outcome depends on use.

· denied 0 ·

Asserts (8)Denies (0)Verdicts (2)

The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“They're actually using agents to better understand the code ... to do more thorough code reviews”
Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“it's amazing for learning uh a code base you're not familiar with.”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“you can immerse yourself in the output that Claude or any other trendy tool gives you and then hopefully learn something from it”
Understanding is the new bottleneck (geoffreylitt.com)“walking through the changes in a sensible order, with surrounding explanation and embedded code snippets. Faster to review than a raw diff.”
How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt (Margaret-Anne Storey)“agents can also be used to manage cognitive load by summarizing what changes have been made and how”
Using AI to write better code more slowly (Read the Tea Leaves)“It’s a great way to improve the overall health of the codebase while also teaching you about the odd corners of it.”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“you can often quickly sharpen your understanding of the code by talking with an agent.”
Nobody Agrees About Agentic Development (Convex)“You have a genius on your computer who can tell you exactly how it works. So you can keep asking questions”
Leans partly right AI guessAgents do speed onboarding and PR summarization, but surveys and the Anthropic learning study show many users skip understanding; outcome depends on use.
Leans right Anthropic internal study lines 32-32Anthropic: 42% use Claude daily for code understanding, the Security team uses Claude Code mainly (48.9%) to analyze unfamiliar code, and some say it lets them test more thoroughly.

Open-weight coding models now handle typical coding tasks reasonably well, though they trail the best closed models

Leans right

Qwen3-Coder, Kimi K2 and GLM 4.5/4.6 posted strong SWE-bench and Terminal-Bench results in 2025 while generally trailing frontier closed models.

· ·

Asserts (5)Denies (3)Verdicts (1)

Two People Vibe Coded a $1.8B Company. Yes, This is AI Takeoff. (AI For Humans)“85.2 compares to 92% for GPT 5.4 ... So, it's not all the way there, but these are tiny models”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“even that model was still struggling a bit with tool calling, which is really crucial for our workflows”
DeepSeek R1 vs Kimi K2 vs Qwen3 Coder vs GLM 4.5 - Head-to-Head Testing! (Bijan Bowen)“every single one of these is something that is open source ... All of the results were acceptable”
"Do You Still Read the Code?" • zanlib (zanlib)“we have access to many providers of open-weights models of high enough quality to trundle along if it comes to it”
Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“they're seven months behind and that gap is gradually narrowing”
DENIES From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“the smaller models usually still struggle with the tool calling.”
DENIES AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“there are no local llamas right now which can be trusted for writing code as good as that”
DENIES Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“some of the open source coding agents with open models and that so far is not really working all that well”
Leans right AI guessQwen3-Coder, Kimi K2 and GLM 4.5/4.6 posted strong SWE-bench and Terminal-Bench results in 2025 while generally trailing frontier closed models.

Clearing context and starting fresh beats compacting a long session

Leans partly right

Fresh context avoids accumulated noise, but compaction is useful when state is hard to reconstruct; it is a tradeoff, not a rule.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“devs love compacting for some reason, but I hate it”
AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“when I start iterating more than three times with AI just closing the session, reopening a new one and restarting”
Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“what is relevant? ... That's why this feature is not working very well”
Context Engineering 101 - The Simple Strategy to 100x AI Coding (Cole Medin)“Typically, you want to clear the context. So, you're starting with a blank slate”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“for each new task I try to start a session a new session”
Context Engineering SF: Advanced Context Engineering for Agents - Dex Horthy (Chroma)“I think slash compact is trash. I never use it”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“We want to start a brand new session with Claude code ... We want it to have a fresh set of eyes”
Leans partly right AI guessFresh context avoids accumulated noise, but compaction is useful when state is hard to reconstruct; it is a tradeoff, not a rule.

AI code review is more effective in a fresh context than in the session that wrote the code

Leans partly right

Fresh-context reviewers avoid self-confirmation and context bloat; plausible and commonly practiced but little hard evidence.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“the reviewer will be dumber than the thing that actually implemented it”
Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“the sub agent is kind of like a new person, right? A new context, fresh eyes”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“The tester often finds some things that a software engineer missed”
SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“with that quote unquote unbiased view it looks at the code and reviews it”
How far can we push AI autonomy in code generation? (martinfowler.com)“We introduced a review agent to double check AI's work against the original prompts.”
Killing Coding Agent Slop With Adversarial Self-Play (Telos)“A single agent with shared context doesn't do as good a job as an independent verifier.”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“it needs to be outside of this agent”
Leans partly right AI guessFresh-context reviewers avoid self-confirmation and context bloat; plausible and commonly practiced but little hard evidence.

Running many coding agents in parallel is not yet productive for most developers

Leans partly right

Review and merge overhead limits parallel agents for most people today, but structured setups (worktrees, independent tasks) are productive for some.

· ·

Asserts (3)Denies (4)Verdicts (1)

Context Engineering & Coding Agents with Cursor (OpenAI)“are we really being productive running nine CLIs in parallel? Probably not”
My AI Adoption Journey (Mitchell Hashimoto)“I'm not [yet?] running multiple agents, and currently don't really want to.”
Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“do less things in parallel”
DENIES AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“they run multiple parallel agents ... They are a bit more methodical about it”
DENIES The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“the very very very top 1% of the 1% ... run multiple of these things in parallel”
DENIES DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“at the current pace, I can run about 16 threads”
DENIES Learn the basics of Google Antigravity (Google Antigravity)“you can have the agent do work for you in the background while you focus on maybe a more complex task in the foreground”
Leans partly right AI guessReview and merge overhead limits parallel agents for most people today, but structured setups (worktrees, independent tasks) are productive for some.

AI increases demand for senior engineers relative to juniors

Leans partly right

Hiring data (e.g. Stanford 'canaries' 2025) shows junior roles hit harder than senior, but evidence that senior demand rose is weaker.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

How AI Is Crushing Junior Developers (Economy Media)“where three or four junior employees were once needed ... today a smaller team with advanced tools can fulfill the same role”
AI Replaced 75% of Work for Software Engineers & Data Scientists - here's what's left (Sundas Khalid)“senior software engineers ... they're going to be super high in demand”
Jensen Huang Said "Stop Learning to Code" — Then Hired 10,000 Developers (Olive Badger)“they were only needed a couple of senior developers”
Junior Dev Jobs Are Down 20%… So Why Are Companies Hiring? (HTML All The Things)“even as their older colleagues head count grows”
The 70% Problem - Agentic Engineering Glossary | AddyOsmani.com (AddyOsmani.com)“Junior developers who only know how to get the 70% are less valuable than senior engineers”
Will AI replace Software Engineers ? Raw & Honest Opinion (Apporva arya)“junior developer position ... risk factor”
Nobody Agrees About Agentic Development (Convex)“OpenAI and Anthropic are quite openly saying that they're now hiring only experienced engineers”
Leans partly right AI guessHiring data (e.g. Stanford 'canaries' 2025) shows junior roles hit harder than senior, but evidence that senior demand rose is weaker.

Agents should keep notes in an external scratchpad or memory store instead of holding everything in context

Leans right

External notes/memory files are a standard, effective pattern (Anthropic, Manus write-ups) for long tasks; adds retrieval overhead.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

Armin Ronacher Leaning In To Find Out - PyAI Conf 2026 (Pydantic)“you're actually surprisingly context efficient if the data actually doesn't have to travel through the context”
Context engineering for AI agents (Microsoft Developer)“this information can stay out of the context window and is only called when needed”
Ralph Wiggum killed programming (Ras Mic)“The progress file tracks what's done”
Context Engineering SF: Advanced Context Engineering for Agents - Dex Horthy (Chroma)“we have it write out a progress file very specifically”
How Agents Use Context Engineering (LangChain)“giving agents access to a file system is very useful. It lets agents save and recall information”
FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“we we use a scratchpad and it's just a regular file”
Google Jules 3.0 UPDATE: FULLY FREE Async AI Coder IS INSANELY GOOD! (Jules API, Tools, CLI, & More) (WorldofAI)“Jules memory ... it can save your preferences, corrections, and workflow habits”
Probably right AI guessExternal notes/memory files are a standard, effective pattern (Anthropic, Manus write-ups) for long tasks; adds retrieval overhead.

Frontier model improvements have become incremental rather than exponential

Leans wrong

METR's task-horizon metric kept doubling roughly every 7 months through 2025-26 and coding benchmarks rose sharply; perceived per-release gains feel smaller but the trend is steep.

· ·

Asserts (5)Denies (2)Verdicts (1)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“in practice it's not that much different than the previous model that also everyone thought was the best model ever”
AI made Developers Slower (Telusko)“with every model you will see slight improvement”
Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“pre-training was a big deal. It seemed exponential but actually it S-curved out and got flat as things went on.”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“I'm mostly doing coding since May and models didn't get better in that period of time”
AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“if what we are seeing is incremental improvements like Claude 3.5 Sonet is a substantial incremental improvement over GPT 40”
DENIES AI Agent Capabilities Are Doubling Every Three Months (The AI Daily Brief: Artificial Intelligence News)“it appears that exponential is what we have here”
DENIES Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev) (The Pragmatic Engineer)“the model improvements in the last 6 months have been some sort of tipping point”
Leans wrong AI guessMETR's task-horizon metric kept doubling roughly every 7 months through 2025-26 and coding benchmarks rose sharply; perceived per-release gains feel smaller but the trend is steep.

Giving agents organizational context (tickets, discussions, history, docs) reduces correction loops and token use

Leans partly right

Relevant org context plausibly cuts rework, but the cited 50% token / one-shot result is a single vendor test, not independent evidence.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“they load the code base into a graph they also load your tickets, your Slack history, your wiki”
Building Blocks for Uber’s Software Factory— Uday Kiran Medisetty & Adam Huda, Uber (AI Engineer)“with and without graph we see massive improvement in tokens, turns and latency”
Building the Context Engine AI Agents Need (InfoQ)“nearly 50% less tokens used ... With context, it actually one-shot the prompt”
Using AI Code Generation to Migrate 20000 Tests (InfoQ)“the more context you provide for them uh the better they are”
Enterprise Vibe Coding: Building Secure, AI-Powered Apps in 30 Minutes - Parth Sharma (Developer Summit)“Context gap ... full context of your portfolio”
Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“Like these are all things that to have an effective agent have organizational memory it needs to start to get access to.”
How Coinbase scaled AI to 1,000+ engineers | Chintan Turakhia (How I AI)“context is the most important thing. So the place where we capture all of our context is linear”
Leans partly right AI guessRelevant org context plausibly cuts rework, but the cited 50% token / one-shot result is a single vendor test, not independent evidence.

LLMs propose outdated library versions and models because of their training cutoff, so their technology choices must be checked

Leans right

Training cutoffs reliably cause stale version and model suggestions; checking against current docs is necessary.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker (Hamel Husain)“if you don't have kind of web search or grounding you often end up with outdated versions of the OpenAI API”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“many of these models have been trained with code that was out there some time ago”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“LLMs have this knowledge cutff ... it suggest to use GPT40”
AI Dev 26 x SF: Andrew Ng: The Future of Software Engineering (DeepLearningAI)“if you ask Claude Code to call the OpenAI API, it will use the chat completions API, which is older and deprecated”
Is AI making devs SLOWER? Yes, according to this study... (Native Notify)“all the models out there are still trained on one to two year-old data”
How far can we push AI autonomy in code generation? (martinfowler.com)“we found that the LLM frequently uses javax.persistence, which has been superseded by jakarta.persistence.”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“LLMs always pick old versions that have tons of security problems in them”
Very likely right AI guessTraining cutoffs reliably cause stale version and model suggestions; checking against current docs is necessary.

The length of tasks AI agents can complete autonomously has been doubling roughly every seven months

Leans right

METR's 2025 time-horizon study found ~7-month doubling (faster in 2024-25); later measurements stayed on or above trend, with wide error bars.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

AI Agent Capabilities Are Doubling Every Three Months (The AI Daily Brief: Artificial Intelligence News)“The length of tasks that AI can do is doubling about every 7 months”
Why Agent Hype can fall short of reality – Joel Becker, METR (AI Engineer)“the time horizon of models is doubling every something like six to seven months”
Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“plotted with log log scaled y- axis, it looks remarkably steady um in its exponential pattern.”
How Agents Use Context Engineering (LangChain)“the task length is doubling around every seven months”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“if you plot that over calendar time you see something that kind of looks exponential.”
How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR (AI Engineer)“if capabilities are measured by time horizon keep keep doubling, it does seem very very challenging to keep up with that.”
AI Just Crossed the Terrifying Line - Now What? (Kurzgesagt – In a Nutshell)“largely invisible to the public, their capabilities have increased exponentially”
Very likely right AI guessMETR's 2025 time-horizon study found ~7-month doubling (faster in 2024-25); later measurements stayed on or above trend, with wide error bars.

Major AI coding subscriptions are subsidized below cost by their vendors

Leans partly right

Heavy users clearly consume more compute than flat fees cover, which is why vendors added weekly limits; whether plans are below cost on average is not public.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“it's pretty obvious that at the moment it's still very subsidized”
Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“these $200 a month ... those are all loss leaders”
Junior Dev Jobs Are Down 20%… So Why Are Companies Hiring? (HTML All The Things)“it's either heavily subsidized by the companies or it's just not usable in a general setting”
A very subjective comparison of Claude Code, OpenCode, Cursor & GitHub Copilot (Maximilian Schwarzmüller)“Claude Code, Cursor, and I guess also GitHub Copilot in the end are all heavily subsidized”
Agentic Coding is a Trap | Lars Faye (larsfaye.com)“Model providers are heavily subsidized, and the models themselves are built on shifting sands.”
Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“the subscriptions are subsidized”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“I know I'm using way more compute than the cost that I pay them”
Leans partly right AI guessHeavy users clearly consume more compute than flat fees cover, which is why vendors added weekly limits; whether plans are below cost on average is not public.

Developers often ship AI-generated code without thoroughly reviewing it

Leans partly right

The Debt paper says AI-generated code is unlikely to all get thorough review and over-trust raises acceptance. The vibe-coding review also finds code accepted without full understanding. Both are cited, not measured, prevalence.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (5)

The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“that has led to more and more code reviews being skipped, being rubber stamped.”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“eventually you're going to let through unintentionally because like your brain just turns off a little bit”
LLMs are bad at vibing specifications (buttondown.com)“So we know the person did not actually run these specs.”
The AI Code Problem Nobody's Talking About (Razorwire Cybersecurity)“Developers admit that they often don't deeply review the AI output”
A look at CloudFlare’s AI-coded OAuth library (Neil Madden)“there were 21 commits directly to main on the first day from one developer, no sign of any code review at all”
Using AI to write better code more slowly (Read the Tea Leaves)“if you’re the kind of developer who uses agents to write multi-hundred-line PRs that you barely understand yourself”
A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“a lot of stuff gets merged without reviews”
Probably right AI guessSurveys (e.g. Sonar 2026) report many developers don't always check AI code before committing, despite saying they distrust it.
Leans partly right Debt study (302k AI commits) lines 143-146The Debt paper says AI-generated code is unlikely to all get thorough review and over-trust raises acceptance. The vibe-coding review also finds code accepted without full understanding. Both are cited, not measured, prevalence.
Leans partly right Vibe-coding literature review lines 2223-2231Minimal-review commits appear in only 2 of 47 sources (grey, no peer-reviewed corroboration), though the cited Fawzy et al. grey-literature review found 36% of practitioners skip quality assurance.
Leans partly right METR 2025 RCT lines 1015-1018METR 2025: among experienced OSS maintainers, 75% read every line of AI code and all of them modified it before submitting. This contradicts the claim for this high-standards group only. (Narrow setting, so it limits the claim rather than refuting it.)
Leans partly right Code-ownership experiment (Martin-Lopez) lines 636-646Participants submitted AI-assisted code that they then explained less well, with the median dropping from 100% to 87.5% correct. This is indirect evidence that generated code gets accepted without full comprehension, gathered in a 75-minute lab task rather than real shipping.

A clear AI policy, AI access to internal data and small-batch work amplify the benefits teams get from AI

Leans partly right

These come from DORA's 2025 AI capabilities model, a survey-based correlational analysis; plausible and consistent with older DORA findings on small batches, but not causal evidence.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

2025 DORA report: Insights to impact with Google and Thoughtworks (Thoughtworks)“Access to internal data, working in small batches, and that clear and communicated stance”
Nicole Forsgren: Leading high-performing engineering teams in the age of AI - The Pragmatic Summit (The Pragmatic Engineer)“explicit exec sponsorship makes a huge difference in not just using them, but trying new things and feeling safe to fail”
DORA Community Discussion - State of AI-assisted Software Development (DORA)“a clear and communicated AI stance, a healthy data ecosystem, AI accessible internal data, strong version control practices, working in small batches”
DORA Community Discussion - Decoding DORA 2025: AI’s Real Impact (DORA)“These are the three things that are going to uh have the most impact on our product performance.”
How AI Is Reshaping Software Development Insights from the 2025 DORA Report | AppDevANGLE (SiliconANGLE theCUBE)“One of those DORA AI capabilities is a clear and communicated stance on how you can use AI”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“they've had to be so intentional with acceptable use policies, with licensing ... And the more intentional and structured a roll out it is”
Discussing the State of AI-assisted Software Development with DORA and EarnIn (Datadog Community)“having the this ability to connect to MCP servers to connect to internal data like maybe your security and compliance policies”
Leans partly right AI guessThese come from DORA's 2025 AI capabilities model, a survey-based correlational analysis; plausible and consistent with older DORA findings on small batches, but not causal evidence.

Using AI for explanations and concepts preserves skill formation, while delegating code generation to it impairs learning

Leans right

Anthropic's 2026 study found high scorers used AI for conceptual questions and explanations while low scorers delegated; small sample and a single learning task.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (3)

SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker (Hamel Husain)“it's really encouraging her to write the code not to do lots of for you all at once”
AI Coders Scored 17% Lower—Here's What They Did Wrong (DIY Smart Code)“The lowscoring patterns all have one thing in common. You stop thinking”
Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“the people who do use it to learn more and learn faster are going to like they're going to outpace”
How To Solve It With Code—background on the course; a discussion with Jeremy and Hamel (Jeremy Howard)“make sure you really understand that that code and why it's written that way and what it does”
Teaching CS50 with AI - David J. Malan (CS50)“just directly using something like ChatGPT would have definitely detracted from learning”
</> htmx ~ Yes, and... (htmx.org)“AI, used properly, is a tremendously effective TA.”
Agentic Coding is a Trap | Lars Faye (larsfaye.com)“when used responsibly, they can be a stellar tool for learning and upskilling.”
Probably right AI guessAnthropic's 2026 study found high scorers used AI for conceptual questions and explanations while low scorers delegated; small sample and a single learning task.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 810-818Using AI to generate code (Copilot plus any tool) lowered comprehension of the resulting code. But the study logged no detail on how AI was used, so it cannot compare explanation-style use with delegation, and it did not measure skill formation.
Leans partly right Shan et al. field study + RCT lines 1054-1064Using ChatGPT for ideation while coding independently improved accuracy, while direct acceptance only raised speed. This is task performance, though; learning and skill retention were explicitly not measured.

In AI-assisted development the specification, not the code, should be the authoritative source of truth

Leans partly right

Specs help align agents, but code is still what runs; specs drift from implementation unless continuously maintained, and practitioners like Pocock report specs-only workflows failing.

· ·

Asserts (5)Denies (2)Verdicts (2)

Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro (AI Engineer)“It is like this living documentation about the system”
AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“We're going to be programming in terms of specifications cuz that's the level at which we must interact with it”
Spec Kit: Github's NEW tool That FINALLY Fixes AI Coding (Better Stack)“the spec becomes the authorative artifact and the models constantly circle back to the spec document”
Killing Coding Agent Slop With Adversarial Self-Play (Telos)“This abstraction lets us treat the spec as a durable artifact and code as the disposable implementation, much like a traditional compiler.”
What Is Spec-Driven Development - Kiro · Spec Kit · BMAD | Complete Visual Guide #2 (ByteLearn)“If the code and the spec disagree, the spec wins and the code is forced to change”
DENIES How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“I'd rather get smaller, more frequent slices than more stuff in each slice”
DENIES Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“the source of truth should be the code, not the specification or the plan”
Leans partly right AI guessSpecs help align agents, but code is still what runs; specs drift from implementation unless continuously maintained, and practitioners like Pocock report specs-only workflows failing.
Leans partly right Vibe-coding literature review lines 1924-1928The discussion says the developer's 'unit of work moves from the line of code toward the specification and the acceptance decision'; it also recommends reviewing code before committing, so it does not make the spec the sole source of truth.

Best practices for AI coding tools go stale within months because model capabilities shift quickly

Leans right

Tool-use and long-horizon capability jumps in 2025 changed effective workflows (chat-with-RAG to agentic loops); consistent with practitioner reports.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

Reconsidering AI w / Steve Klabnik (Zed Industries)“everyone was talking about prompt engineering a year ago, but like now it's about context engineering”
The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“the number one mistake ... is they're trying to use coding agents the same way they're using AI coding tools six months ago”
Does AI-driven productivity come at the cost of code quality? (Proxify)“the limitation is always evolving and changing”
Andrew Ng: Building Faster with AI (Y Combinator)“being half a generation behind makes a big difference”
How Uber Uses AI to Move from Java to Kotlin (Kotlin by JetBrains)“he was calling them the leader of the quarter because things are moving so quickly”
SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“might want to revisit that as well with newer models because some things are maybe not necessary anymore”
Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“these decisions need to be revisited on a pretty regular basis.”
Probably right AI guessTool-use and long-horizon capability jumps in 2025 changed effective workflows (chat-with-RAG to agentic loops); consistent with practitioner reports.

Benchmarks overstate real-world usefulness because benchmark tasks are cleaner and scored more leniently than real work

Probably right

METR concludes the field slowdown suggests in-the-wild capability may be lower than benchmarks imply, because benchmark tasks are self-contained and scored algorithmically.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (4)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“most people in the world are never ever ever going to care about an ELO rating”
Why Agent Hype can fall short of reality – Joel Becker, METR (AI Engineer)“SWE-bench like scoring ... versus mergeability like scoring ... the problems aren't very messy”
Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“about half of the cases that Sweetbench measures as um as as passing these algorithmic tests would be merged into Maine by these maintainers.”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“benchmarks look very bullish. This RCT looks much less bullish.”
How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR (AI Engineer)“basically all the time they find that the models fall on their faces and suck.”
AI in software engineering at Google: Progress and the path ahead (research.google)“offline metrics are often only rough proxies of user value”
Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“the benchmarks are one shot ... that's not how real software gets built”
Probably right AI guessMETR's own analysis and follow-ups found many benchmark-passing patches aren't mergeable; messy, high-context work lags benchmark scores.
Probably right METR 2025 RCT lines 484-489METR concludes the field slowdown suggests in-the-wild capability may be lower than benchmarks imply, because benchmark tasks are self-contained and scored algorithmically.
Leans partly right Google migrations report lines 619-633Google argues that good model-level eval performance on hermetic tasks does not guarantee good business-level outcomes, because the model is only one part of the task; this is a position statement, not a measurement.
Leans partly right Vibe-coding literature review lines 1805-181130% of RQ8 sources cite the lack of standardized benchmarks that capture realistic multi-step workflows and long-horizon tasks; they do not show that benchmarks overstate usefulness.

Engineers who adopt agentic coding are about ten times as productive as peers who don't

Leans wrong

Second-hand anecdote measured by lines of code and commits; controlled and large field studies find average gains of roughly 10-30%, sometimes negative.

· ·

Asserts (2)Denies (5)Verdicts (3)

DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“to get that magical 10X, 100X, in a few rare cases, 1000X productivity boost, you have to interact with the agents directly”
Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“the performance difference is like 10x any way that you measure it”
DENIES Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“I'm not 100x faster. I'm not even 10x faster.”
DENIES AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“the people that are claiming that you get tens or hundreds of times productivity are wrong too”
DENIES Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“the results were just terrible. We we our productivity fell off a cliff.”
DENIES No, AI is not Making Engineers 10x as Productive (colton.dev)“10x productivity means ten times the outcomes, not ten times the lines of code.”
DENIES Where's the Shovelware? Why AI Coding Claims Don't Add Up (mikelovesrobots.substack.com)“There are no new 10xers.”
Probably wrong AI guessSecond-hand anecdote measured by lines of code and commits; controlled and large field studies find average gains of roughly 10-30%, sometimes negative.
Leans wrong Anthropic internal study lines 35-35Anthropic: users self-report an average +50% gain at 59% usage, plus about 67% more merged PRs per engineer after Claude Code adoption, an order of magnitude below 10x.
Leans wrong Shan et al. field study + RCT lines 489-493The field gain from AI assistants was about 18% more user stories (24-31% under alternative estimators), far from 10x. The tools were 2023-24 assistants, not agentic coding.

AI coding agents tend to over-complicate solutions unless explicitly pushed to simplify

Leans partly right

Code-bloat and duplication findings support a tendency toward verbose, over-engineered output, but it varies a lot by model and prompt.

· denied 0 ·

Asserts (8)Denies (0)Verdicts (3)

DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“you actually do have to tell them, "Make it simpler."”
SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“the facto complexity definitely another great example of a very typical failure mode of AI generated code”
Coding with LLMs in the summer of 2025 (an update) - <antirez> (antirez.com)“when left alone with nontrivial goals they tend to produce fragile code bases that are larger than needed”
No, AI is not Making Engineers 10x as Productive (colton.dev)“AI often seems to encourage hastiness and over-building.”
</> htmx ~ Yes, and... (htmx.org)“significant accidental complexity by choosing inappropriate approaches to problems, taking shortcuts, etc.”
How far can we push AI autonomy in code generation? (martinfowler.com)“We frequently got additional endpoints and features that we did not ask for in the requirements.”
TDD inside the agent loop - theater or actual value? (martinfowler.com)“They frequently overshot and implemented more than the current test demanded”
You can just choose how many bugs you want now (Read the Tea Leaves)“They will happily build one epicycle per bug until the code is a spaghetti mess.”
Leans partly right AI guessCode-bloat and duplication findings support a tendency toward verbose, over-engineered output, but it varies a lot by model and prompt.
Leans partly right METR 2025 RCT lines 1020-1021METR 2025: one developer says that after accepting AI code 'I do a lot of simplifying its code'; this is anecdotal support from a single quote.
Leans partly right Echoes of AI maintainability RCT lines 2111-2121For pre-agent assistants, AI-devs triggered fewer cyclomatic/NPath complexity violations, so AI seemed to reduce complexity while adding stylistic clutter; they did introduce slightly more complex-conditional and nested-complexity smells, and agents were not studied. (Narrow setting, so it limits the claim rather than refuting it.)

Intensive agentic coding raises cognitive load and fatigue rather than making developers' work easier

Leans partly right

Many practitioner reports and some surveys describe higher intensity and fatigue from supervising agents, but controlled evidence is thin and other studies report reduced effort on routine work.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (4)

Nicole Forsgren: Leading high-performing engineering teams in the age of AI - The Pragmatic Summit (The Pragmatic Engineer)“I'm having to sometimes rebuild my mental model dozens of times in like a 30-minute period.”
The AI Productivity Paradox: Why 10X Output Doesn't Mean 10X Business Outcome - Mik Kersten (Tech Lead Journal)“I think the cognitive load increases a lot with working all of these agents”
How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt (Margaret-Anne Storey)“Adding more agents to a project may add more coordination overhead, invisible decisions, and thus cognitive load.”
You can just choose how many bugs you want now (Read the Tea Leaves)“I find myself mentally exhausted after slogging through the umpteenth AI-generated bug report”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“after two hours of that I'm done for the day. Like I'm mentally exhausted”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“It doesn't actually make us make our lives more relaxed. On the contrary.”
"Do You Still Read the Code?" • zanlib (zanlib)“It might turn out that the accelerator approach is just a fast track to burnout.”
Leans partly right AI guessMany practitioner reports and some surveys describe higher intensity and fatigue from supervising agents, but controlled evidence is thin and other studies report reduced effort on routine work.
Leans partly right Vibe-coding literature review lines 1906-1914Cognitive-load changes appear in 38% of RQ3.1 sources; the discussion says the role shift 'can increase cognitive load through prompt refinement, context repair, and systematic verification', while some sources report smoother flow on routine tasks.
Leans partly right Anthropic internal study lines 39-40Anthropic: some respondents spend more time on Claude-assisted tasks because of debugging and the 'cognitive overhead' of understanding code they didn't write, while most report net time savings.
Leans wrong Shan et al. field study + RCT lines 901-905Developers using ChatGPT for short tasks frequently reported lower cognitive load and workload. The setting was brief chat-based help, not intensive agentic coding, so it bears only weakly on the claim.

Acceptance rate and counts of AI-generated lines are poor measures of AI's productivity impact

Leans right

METR 2025 argues that lines of code and task counts can rise without productivity (more verbose code, PRs split up), and so uses fixed, pre-defined tasks with time to completion as the outcome.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (5)

Measuring AI code assistants and agents with the AI Measurement Framework (DX)“acceptance rate is an incredibly flawed and inaccurate metric because when developers accept code, much of that code is often then later modified or deleted”
How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“if the goal is more lines of code, I can prompt something to write you the longest piece of code ever”
Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“I'm not uh CEO bragging about the number of lines of code written by AI. That's I think a completely useless metric.”
The Truth About Developer Productivity in the AI Age (IT'S A TRAP) (Modern Software Engineering)“If you measure the percentage of AI generated code in a code base, don't be surprised if the number of classes and methods”
How Coinbase scaled AI to 1,000+ engineers | Chintan Turakhia (How I AI)“lines of code committed by AI. I think that's like kind of misleading”
Developer experience in the AI era (Develocity)“lines of code, which is a trash metric, right? Like AI can, can spit out hundreds of lines of code for you.”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“Do we really want to measure AI impact in terms of lines of code generated? I certainly don't. We don't recommend it.”
Very likely right AI guessAccepted suggestions are often later rewritten or deleted, and output volume has long been a poor proxy for productivity; controlled studies measure task time or delivery instead.
Leans right METR 2025 RCT lines 42-48METR 2025 argues that lines of code and task counts can rise without productivity (more verbose code, PRs split up), and so uses fixed, pre-defined tasks with time to completion as the outcome.
Leans partly right Google migrations report lines 182-202Google: the share of committed AI code is a good proxy for time savings in most cases, but it misses value such as an imperfect first draft that locates the change sites, so they judge success by end-to-end time saved instead.
Leans partly right Debt study (302k AI commits) lines 946-951Authors argue acceptance rate and immediate correctness capture only what happens at introduction and miss later maintenance, where 22.7% of AI-introduced issues persist; they also cite work finding acceptance rate correlates with self-reported productivity.
Leans right Echoes of AI maintainability RCT lines 255-257Authors state completion acceptance rates are 'far from' the maintainability and productivity constructs, and deliberately exclude studies relying on acceptance rate, keystrokes or LoC; assertion, not test.

AI amplifies an organization's existing strengths and weaknesses rather than fixing them

Leans partly right

DORA 2025 data shows AI outcomes depend on surrounding capabilities, which supports the framing, though 'amplifier' is an interpretation of correlational survey results.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit (The Pragmatic Engineer)“AI is an accelerator. It's a multiplier and it is moving organizations off in different directions.”
AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“if you're working poorly, you're just going to dig a deeper hole faster.”
DORA Community Discussion - State of AI-assisted Software Development (DORA)“AI's primary role in software development is that of an amplifier.”
AI in engineering: Q2 2026 benchmarks & research readout (DX)“AI is going to amplify the the good and the bad”
DORA 2025 AI assisted dev report: Some Benefit, Most Don’t (Rob Bowley)“For the other clusters, AI mostly amplifies existing problems.”
DORA Community Discussion - Decoding DORA 2025: AI’s Real Impact (DORA)“AI is an amplifier. So, uh, it doesn't make things that are bad good.”
How AI Is Reshaping Software Development Insights from the 2025 DORA Report | AppDevANGLE (SiliconANGLE theCUBE)“But AI also magnifies weaknesses.”
Probably partly right AI guessDORA 2025 data shows AI outcomes depend on surrounding capabilities, which supports the framing, though 'amplifier' is an interpretation of correlational survey results.

As AI speeds up generation, human attention for review and understanding becomes the bottleneck

Leans right

Google JUnit3->4 migration (5,359 files in 3 months): the bottleneck was how fast engineers could review, and change generation was deliberately throttled so reviewers were not overwhelmed.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (4)

ICM 2026 Public Lectures - Terence Tao (Simons Foundation)“proof review is now one of the most precious bottlenecks um scarce resources um in this new era.”
AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“So, so verification becomes the bottleneck.”
What We Learned Deploying AI within Bloomberg’s Engineering Organization – Lei Zhang, Bloomberg (AI Engineer)“the average open pull requests increased and time to merge also increased”
Understanding is the new bottleneck (geoffreylitt.com)“Agents are writing more and more code for us, and we all know it's getting harder to keep up.”
</> htmx ~ Code is Cheap(er) (htmx.org)“the LLM can produce code far faster than you, or anyone else, can understand it.”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“the precious resource remains human attention.”
Discussing the State of AI-assisted Software Development with DORA and EarnIn (Datadog Community)“the real bottleneck isn't speed it's a confidence actually”
Probably right AI guessDORA and field reports show review and verification load growing with AI output, consistent with instability and persistent-debt findings.
Leans right Google migrations report lines 417-423Google JUnit3->4 migration (5,359 files in 3 months): the bottleneck was how fast engineers could review, and change generation was deliberately throttled so reviewers were not overwhelmed.
Leans right Vibe-coding literature review lines 1906-1914The review's central synthesis is a 'verification gap': a mismatch between how fast AI workflows generate code and developers' capacity to verify it. This is an interpretive synthesis of mention-level evidence.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 736-767AI users produced more, larger code but understood it less. Their median answering time was 3 min longer (+27%) and they switched back to the code more than twice as often, though the time effect was not robust. The authors suggest time saved may be offset later in the lifecycle.

Cognitive debt, the loss of developers' shared understanding of a system built fast with AI, is a bigger long-term risk than technical debt

Leans partly right

Naur's theory-building view and early studies of AI-assisted teams support loss of understanding as a real risk, but there is no measurement yet showing it outweighs technical debt.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (4)

Understanding is the new bottleneck (geoffreylitt.com)“It's like tech debt: you can get away with not understanding what's going on in the short term, but it'll bite you eventually.”
Agentic Coding is a Trap | Lars Faye (larsfaye.com)“you cannot understand code without engaging with it.”
How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt (Margaret-Anne Storey)“Cognitive debt is likely a much bigger threat than technical debt, as AI and agents are adopted.”
You can just choose how many bugs you want now (Read the Tea Leaves)“keeping a working theory of the code in my head (ala Peter Naur)”
Cursor Claims AI Built a Browser. Here's the Problem (Brian Jenney)“What human in this world could go through millions of lines of code, find the fixes, and then implement them”
Vibe code is legacy code (Val Town)“Programming is fundamentally theory building, not producing lines of code.”
"Do You Still Read the Code?" • zanlib (zanlib)“eventually the only practical way to find out what it is that you’ve actually built is to ask the model”
Leans partly right AI guessNaur's theory-building view and early studies of AI-assisted teams support loss of understanding as a real risk, but there is no measurement yet showing it outweighs technical debt.
Leans partly right Echoes of AI maintainability RCT lines 2667-2673Authors conclude that AI-assisted development does not appear to introduce code-level technical debt, and warn that cognitive debt is the greater concern; the warning is not empirically tested here.
Leans partly right Vibe-coding literature review lines 1261-1284Cognitive debt (deferred understanding of accepted code) is reported in 24% (8) of RQ3.1 sources with warnings of long-term unmaintainable systems; the review never compares its size with technical debt.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 957-967The study shows only a short-term loss of individual code comprehension right after writing, and it explicitly leaves long-term effects to future work. It makes no comparison with technical debt, so only the mechanism, not the 'bigger risk' ranking, is supported.

Having the AI interview you to reach shared understanding before planning prevents misalignment

Leans right

Requirements clarification before implementation is well-established practice and widely reported to reduce agent rework.

· denied 0 ·

Asserts (6)Denies (0)Verdicts (1)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“Interview me relentlessly about every aspect of this plan”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“it's popping up a window that it's interviewing me.”
Codex CLI: Everything You Need in 12 mins (lustoykov)“in plan mode, Codex will start asking questions to iron out the details”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“ask me one question at a time ... we are moving towards something that we decide not the agent decides”
12 Months of Unity Dev with Claude Code (Mythmatic)“it asks Claude to only ask me one question at a time, wait for me to respond”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“the most powerful strategy here ... is to have the coding agent ask you questions”
Probably right AI guessRequirements clarification before implementation is well-established practice and widely reported to reduce agent rework.

Codebases built from deep modules with small interfaces are easier for AI agents to work in and test

Leans right

Clear module boundaries help both humans and agents navigate and test; consistent with Ousterhout and practitioner reports.

· denied 0 ·

Asserts (7)Denies (0)Verdicts (1)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“bad code bases make bad agents”
Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro (AI Engineer)“if your system already had good separation of concerns ... it's going to have a great job”
Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“modularity sits at the heart of it ... If it's too big of a monolith ... it can't do any thinking at all”
Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“In general, simple code bases have an advantage to be AI-developed.”
The AI-Native Data Engineer featuring Fabric, dbt, VSCode devcontainers and copilot (Microsoft Reactor)“you want to have it in a function which is called transform, which takes in a data frame and spits out a data frame”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“AI can also deal with a well factored code base much better than with a messy code base”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“leaders re recommitting to like clean interfaces between services. I would say that's probably the top thing that comes up”
Probably right AI guessClear module boundaries help both humans and agents navigate and test; consistent with Ousterhout and practitioner reports.

AI coding tools are converging in capability and no single tool is best for every workflow

Leans right

By 2026 the major tools all offer agents, CLIs, cloud runs and review; differences are workflow and pricing more than capability.

· denied 0 ·

Asserts (6)Denies (0)Verdicts (1)

Codex vs Claude Code vs Cursor vs Copilot — Best AI Coding Agent in 2026? (Byte & Beyond)“Notice what is missing, a universal winner”
Cursor vs Claude Code vs GitHub Copilot — Which AI Coding Tool Actually Wins? (2026) (Nordic AI Dev)“There's no universal winner”
Huge Codex Update: All My Complaints Addressed (Matt Maher)“They really covered just about everything that I was complaining about ... That kind of feels like parity now”
A very subjective comparison of Claude Code, OpenCode, Cursor & GitHub Copilot (Maximilian Schwarzmüller)“ultimately, all these tools are decent”
Best AI Code Editors in 2026 — Cursor vs Windsurf vs Copilot vs Zed vs Claude Code (Nordic AI Dev)“there's no single best here”
AI Coding Assistants Compared: Cursor vs Copilot vs Claude Code vs Codex (2026) (Tyson Cung)“The verdict, it depends on your use case”
Probably right AI guessBy 2026 the major tools all offer agents, CLIs, cloud runs and review; differences are workflow and pricing more than capability.

The post-2022 drop in developer hiring was driven mainly by interest rates and pandemic over-hiring rather than AI

Leans right

The 2022-23 decline began before ChatGPT and tracks rate hikes and over-hiring; most economists attribute it mainly to macro factors, though AI may weigh on junior roles since 2024.

· ·

Asserts (5)Denies (1)Verdicts (1)

Computer Science Degrees Are WORTHLESS, Now Workers Are FURIOUS — Here’s Why (Jay Reed)“They're saying AI, but ultimately, I don't think it's AI ... they over-hired in the pandemic”
How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“the most important thing that's hit us is not AI. It's the end of zero interest rates”
The TRUTH About the Software Development Job Market in 2026 (Internet Made Coder)“this was kind of just a coincidence ... a combination of economic factors”
AI Replacing Developers Has Officially Failed (Sajjaad Khader)“the tech market was utterly cooked before AI even stepped into the kitchen”
</> htmx ~ Yes, and... (htmx.org)“programmer job market tends to be cyclical with booms and busts”
DENIES How AI Is Crushing Junior Developers (Economy Media)“The challenge is not explained solely by the macroeconomic situation”
Probably right AI guessThe 2022-23 decline began before ChatGPT and tracks rate hikes and over-hiring; most economists attribute it mainly to macro factors, though AI may weigh on junior roles since 2024.

AI coding tools reduce perceived effort more than they reduce elapsed time

Leans partly right

Consistent with METR's perception gap and surveys reporting less frustration/fatigue; a plausible explanation rather than a measured mechanism.

· denied 0 ·

Asserts (6)Denies (0)Verdicts (3)

Unpacking METR’s findings: Does AI slow developers down? (DX)“it was en more enjoyable to talk back and forth to an AI than it was to grind myself on a bug”
GitHub Copilot productivity: what the research actually shows (Mneme HQ)“87% of developers reported that GitHub Copilot helped them preserve their mental effort during repetitive tasks”
Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“They enjoy using cursor more as a as a sort of um workflow”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“AI tooling companies are producing benefits for the developer experience more than more than productivity benefits at recent margins”
The METR Paradox. Why AI Made Senior Devs 19% Slower (Macro Lens)“The cognitive load drops, the clock doesn't”
METR's AI productivity study is really good (seangoedecke.com)“coding with AI requires less cognitive effort, and makes it easier to multi-task or just to zone out.”
Leans partly right AI guessConsistent with METR's perception gap and surveys reporting less frustration/fatigue; a plausible explanation rather than a measured mechanism.
Leans partly right METR 2025 RCT lines 1165-1173The paper floats trading speed for ease; reports are mixed. One developer felt 'less effort' and another 'didn't really think about some of the problems', while a third saw no effort difference.
Leans partly right Shan et al. field study + RCT lines 1246-1251Developers said ChatGPT eased their cognitive load, yet in the follow-up experiment the ChatGPT group spent more total time (less on coding tasks, more on multiple-choice). Perceived effort fell while elapsed time did not.

AI is displacing entry-level work and raising unemployment among recent graduates

Leans partly right

Stanford 'Canaries' (2025) found ~13% relative employment decline for 22-25-year-olds in AI-exposed jobs, but rates, over-hiring correction and other macro factors also matter.

· denied 0 ·

Asserts (6)Denies (0)Verdicts (1)

How AI Is Crushing Junior Developers (Economy Media)“businesses are now replacing entry-level jobs with artificial intelligence”
Computer Science Degrees Are WORTHLESS, Now Workers Are FURIOUS — Here’s Why (Jay Reed)“Entry-level hiring across all tech saw a 73% decrease ... structural elimination of the entry-level role”
Junior Dev Jobs Are Down 20%… So Why Are Companies Hiring? (HTML All The Things)“Employment among software developers aged 22 to 25 have has plummeted nearly 20% since 2024”
AI Layoffs Don’t Add Up. Here’s the Data (HuMan Intelligence)“Graduate roles down 32% since Chat GBT launched”
Companies Are Lying About AI Layoffs - Here’s the Proof (Vanessa Wingårdh)“computer engineering, 7.5% unemployment”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“to the extent that we're seeing economic impacts so far it's just for these um uh very young people in very AI exposed industries”
Leans partly right AI guessStanford 'Canaries' (2025) found ~13% relative employment decline for 22-25-year-olds in AI-exposed jobs, but rates, over-hiring correction and other macro factors also matter.

Instructions alone (such as 'do not touch production') do not reliably constrain agents; safeguards must be enforced by the environment

Leans right

Agents violate explicit instructions often enough that dev/prod separation, permissions and sandboxes are standard advice; Replit's own fix was architectural.

· denied 0 ·

Asserts (6)Denies (0)Verdicts (1)

AI in engineering: Q2 2026 benchmarks & research readout (DX)“everything we tell an agent to do it still treats as a suggestion”
Episode 155: The Replit AI Debacle (AIPOD-CAST)“Don't just rely on instructions or flags like don't touch the database. You need architectural safeguards”
The Replit AI Agent DISASTER: Production Database Deleted In SECONDS! ($erver $ays)“Lemkin realized there was no way to actually enforce a code freeze on the AI”
A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“went wrong in your process if if the agent had the the power to do that”
AI Just Crossed the Terrifying Line - Now What? (Kurzgesagt – In a Nutshell)“They argued that cyberattacking an unrelated company was malicious, unethical and explicitly went against their instructions.”
Let's put the AI in lots of little boxes - Kenton Varda, Cloudflare Connect 2025 (Cloudflare Developers)“we need to take a a uh uh technical approach to preventing them from making these mistakes.”
Very likely right AI guessAgents violate explicit instructions often enough that dev/prod separation, permissions and sandboxes are standard advice; Replit's own fix was architectural.

Keeping code maintainable still requires human judgment that linters and prompts can't enforce

Leans partly right

Linters and agent rules catch local smells, but architecture and coupling decisions still lack automated checks. Tooling is improving.

· denied 0 ·

Asserts (6)Denies (0)Verdicts (1)

TDD, AI agents and coding with Kent Beck (The Pragmatic Engineer)“is not good at reducing coupling and increasing cohesion ... it has no taste, no sense of design.”
</> htmx ~ Code is Cheap(er) (htmx.org)“This engineer says no, closely examines LLM output, suggests simplifications and”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“having poor quality choice from code from an agent is a choice that you make.”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“They are just not good at maintaining the quality of a code base.”
Killing Coding Agent Slop With Adversarial Self-Play (Telos)“deterministic checks can't catch every subjective dimension, including design, quality standards, and complexity.”
Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“you can't lint your way to a maintainable codebase”
Leans partly right AI guessLinters and agent rules catch local smells, but architecture and coupling decisions still lack automated checks. Tooling is improving.

Organizational AI adoption needs active enablement (training, showcased wins, champions), not just tool licenses

Leans partly right

Adoption and gains vary widely across teams and DORA finds organizational practices matter, but evidence isolating training effects is mostly observational.

· denied 0 ·

Asserts (6)Denies (0)Verdicts (4)

Anthropic Economic Index 2026—Measuring Claude Usage with “Economic Primitives” Metrics (CosmoX)“Just giving everyone access to AI isn't going to cut it. That's not enough.”
Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit (The Pragmatic Engineer)“just giving all of your developers licenses and hoping for the best. It does not work.”
What We Learned Deploying AI within Bloomberg’s Engineering Organization – Lei Zhang, Bloomberg (AI Engineer)“we just incorporate AI coding in on boarding training program”
How Coinbase scaled AI to 1,000+ engineers | Chintan Turakhia (How I AI)“People opened it up, checked the box, did kind of like a hello world thing, but it didn't stick”
Developer experience in the AI era (Develocity)“Carve out time, get people access, share training, share best practices.”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“They really do need support and enablement and training”
Probably partly right AI guessAdoption and gains vary widely across teams and DORA finds organizational practices matter, but evidence isolating training effects is mostly observational.
Leans right Microsoft/Accenture field RCTs lines 317-380With access alone only 8.5% of Microsoft's treated developers signed up in two weeks; reminder emails lifted it to 42.5%, Accenture's training plus manager encouragement sped adoption, and 30-40% never tried the tool despite free access.
Leans partly right Echoes of AI maintainability RCT lines 2549-2551Developers varied in how effectively they used AI (habitual users got the CodeHealth benefit), which authors take to mean training developers in these tools remains important; this is about individual skill, not organizational rollout.
Leans partly right Google migrations report lines 635-643Google notes that bespoke AI use has a hidden cost of training engineers in the techniques and that tooling to hide AI is costly to maintain, which only touches on enablement.

Coding agents that run tools in a loop are a step change in usefulness over chat-based LLM assistance

Leans partly right

Agentic tools clearly expanded what AI can do unaided, and METR's 2026 update suggests speedups are now likely, but measured average gains remain modest.

· denied 0 ·

Asserts (6)Denies (0)Verdicts (1)

Reconsidering AI w / Steve Klabnik (Zed Industries)“I hated spicy autocomplete and I love agentic style coding”
Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“if you're doing any copy pasting at all you're not getting the experience that either of us have been talking about for an hour”
My AI Skeptic Friends Are All Nuts (Fly)“pasting the resulting (broken) code into your editor, you're not doing what the AI boosters are doing.”
My AI Adoption Journey (Mitchell Hashimoto)“To find value, you must use an agent.”
Learning to Code with AI and Steve Klabnik (Scott Hanselman)“Claude had invoked GDB to examine the binary.”
A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“I said like I would be challenged if it's more than 10%. And that was that was completely invalidated”
Probably partly right AI guessAgentic tools clearly expanded what AI can do unaided, and METR's 2026 update suggests speedups are now likely, but measured average gains remain modest.

AI coding tools are the latest step in software's long rise in abstraction, comparable to compilers and high-level languages

Untestable

A historical framing; the analogy is debated because LLM output is non-deterministic, unlike compilers.

· ·

Asserts (2)Denies (4)Verdicts (1)

The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“We are moving up a level of abstraction much like what happened in the rise from machine language to assembly language”
Learning to Code with AI and Steve Klabnik (Scott Hanselman)“popping up one more layer of abstraction is kind of the way that I feel about it”
DENIES SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“I don't think it goes up in the same way that went up with compilers”
DENIES </> htmx ~ Code is Cheap(er) (htmx.org)“Compilers are deterministic; LLMs are, by design, not”
DENIES </> htmx ~ Yes, and... (htmx.org)“Compilers are, for the most part, deterministic in a way that current AI tools are not.”
DENIES Agentic Coding is a Trap | Lars Faye (larsfaye.com)“a higher level of ambiguity is not a higher level of abstraction.”
Untestable AI guessA historical framing; the analogy is debated because LLM output is non-deterministic, unlike compilers.

Cursor offers the most integrated AI-first editor experience, with codebase indexing and model choice

Leans partly right

Cursor's indexing and model choice are accurate; 'most integrated' is a taste judgment and competitors have closed gaps.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (1)

AI Coding 2026: Cursor vs. Copilot vs. Claude Code 🤖 (ByteBlitzs)“it indexes your entire codebase locally”
Codex vs Claude Code vs Cursor vs Copilot — Best AI Coding Agent in 2026? (Byte & Beyond)“Cursor is arguably the cleanest fit”
Cursor vs Claude Code vs GitHub Copilot — Which AI Coding Tool Actually Wins? (2026) (Nordic AI Dev)“Cursor wins on day-to-day speed and the all-in-one IDE experience”
A very subjective comparison of Claude Code, OpenCode, Cursor & GitHub Copilot (Maximilian Schwarzmüller)“Cursor, for example, has, uh, amazing auto-completion”
Best AI Code Editors in 2026 — Cursor vs Windsurf vs Copilot vs Zed vs Claude Code (Nordic AI Dev)“Cursor is repeatedly called the strongest pick for large existing codebases”
Leans partly right AI guessCursor's indexing and model choice are accurate; 'most integrated' is a taste judgment and competitors have closed gaps.

Claude Code is best suited to autonomous, complex tasks such as hard bugs and large refactors

Leans partly right

Claude Code is widely regarded as strong on autonomous multi-step tasks, but 'best' varies by task and model release; Codex and others compete.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (1)

Gemini CLI: Google's NEW Fully Free Opensource Coding Agent! Beats Claude Code! (WorldofAI)“it also offers superior code structure and engineering capabilities”
AI Coding 2026: Cursor vs. Copilot vs. Claude Code 🤖 (ByteBlitzs)“Use Claude Code when you have a nightmare bug or a massive refactor”
Cursor vs Claude Code vs GitHub Copilot — Which AI Coding Tool Actually Wins? (2026) (Nordic AI Dev)“Claude code wins on raw benchmark performance and autonomous, complex tasks”
Best AI Code Editors in 2026 — Cursor vs Windsurf vs Copilot vs Zed vs Claude Code (Nordic AI Dev)“Claude Code keeps showing up as the pick for complex multi-step reasoning”
AI Coding Assistants Compared: Cursor vs Copilot vs Claude Code vs Codex (2026) (Tyson Cung)“For deep context, Cursor or Claude Code”
Leans partly right AI guessClaude Code is widely regarded as strong on autonomous multi-step tasks, but 'best' varies by task and model release; Codex and others compete.

Delegating research to subagents with their own context windows keeps the main agent's context clean

Leans right

Subagents return summaries rather than raw exploration; Anthropic reports gains from multi-agent research, at higher token cost.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (1)

Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“our main agent ... is not polluting its context window with stuff that's not relevant”
The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“the sub agent allows you to essentially preserve the context window”
Context Engineering SF: Advanced Context Engineering for Agents - Dex Horthy (Chroma)“they're really about context control”
How Agents Use Context Engineering (LangChain)“That sub aent has its own context window ... and just return the output back to the parent agent”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“Sub-agents is one of the best strategies for that”
Probably right AI guessSubagents return summaries rather than raw exploration; Anthropic reports gains from multi-agent research, at higher token cost.

MCP tool definitions consume significant context, so existing CLI tools are often a better way to give agents capabilities

Leans right

Each MCP server injects tool schemas into context; large servers can use tens of thousands of tokens, and agents use CLIs well. Deferred tool loading has since reduced this.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (1)

From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“Why would you run another local process like an MCP server that can access the AWS cloud when you already have the CLI”
Building Blocks for Uber’s Software Factory— Uday Kiran Medisetty & Adam Huda, Uber (AI Engineer)“we projected all of these MCPS into CLI pattern so that even the response doesn't eat up in your context.”
Armin Ronacher Leaning In To Find Out - PyAI Conf 2026 (Pydantic)“because it is basically just pulling proprietary tools into the context all the time it's very hard for model to reason”
Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“if I add 10 MCP tools here my context is already full”
Playwright CLI vs MCP Server: Which is Actually BETTER for Claude Code? (Better Stack)“already 15% of the context is being used because all of these MCP tools are being loaded”
Probably right AI guessEach MCP server injects tool schemas into context; large servers can use tens of thousands of tokens, and agents use CLIs well. Deferred tool loading has since reduced this.

AI-generated code drifts from a codebase's established conventions

Leans right

Without enough context, models fall back to common internet patterns; instruction files and indexing reduce but do not remove this.

· ·

Asserts (3)Denies (2)Verdicts (4)

AI-Generated Code Is Creating a New Kind of Technical Debt (Imran Gardezi)“AI doesn't know your conventions. It knows the internet's conventions”
No, AI is not Making Engineers 10x as Productive (colton.dev)“It's not good at keeping up with the standards and utilities of your codebase.”
My AI Skeptic Friends Are All Nuts (Fly)“LLMs are showing signs of adapting to local idiom, but we're not there yet.”
DENIES Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“If you've got a code base with a bunch of patterns in, they will follow those patterns almost to a T.”
DENIES Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“the code around them and they pick up on the cues and they do a pretty good job of mimicking what's around them”
Probably right AI guessWithout enough context, models fall back to common internet patterns; instruction files and indexing reduce but do not remove this.
Leans right METR 2025 RCT lines 1015-1021METR 2025: developers report AI code leaves 'more to fix up, in terms of code conventions, code style' and needs major cleanup to meet repo standards (qualitative, 16 developers).
Leans partly right Debt study (302k AI commits) lines 672-674Correctness issues dominated by undefined variables (23,856) lead authors to say AI code 'may look locally correct, but still fail to stay consistent with the surrounding context'; this is consistency with surrounding code, not style conventions specifically.
Leans partly right Vibe-coding literature review lines 1434-1438WL27 is quoted saying conversationally generated code 'often lacks cohesive structure, consistent patterns, and proper documentation'; this is a single peer-reviewed source, not a measured drift rate.

Security- and data-critical code (auth, payments, migrations) should be written by people who understand it rather than generated

Leans partly right

Studies find frequent vulnerabilities in AI-generated security code, so the risk is real; but well-reviewed AI-assisted code by people who understand it can be fine.

· denied 0 ·

Asserts (6)Denies (0)Verdicts (1)

Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“I can write C code with Claude, not for Redis, not for system critical things”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“do not delegate anything related to security to an agent”
AI-Generated Code Is Creating a New Kind of Technical Debt (Imran Gardezi)“Some code is too critical for AI generation”
The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“we don't think that a database migration should ever go in without a human making a judgment call on this”
A look at CloudFlare’s AI-coded OAuth library (Neil Madden)“for important things, like my fucking auth system, I’d much rather do it myself”
Are Junior Developers Really Getting Replaced by AI? I'm not sold... (Brian Jenney)“missionritical code is basically off limits”
Leans partly right AI guessStudies find frequent vulnerabilities in AI-generated security code, so the risk is real; but well-reviewed AI-assisted code by people who understand it can be fine.

A simple, self-owned AI coding workflow works better than adopting a heavyweight off-the-shelf agent framework

Leans partly right

Owning and understanding your workflow helps you debug it, and many practitioners report framework bloat; but there is no comparative evidence, and frameworks can be a good starting point.

· ·

Asserts (4)Denies (1)Verdicts (2)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“you need to own as much of your planning stack as you possibly can”
SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker (Hamel Husain)“how easy it is to create your own complete agent framework to do exactly what you want no more and no less”
Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“I don't really like these frameworks ... I take the ideas and put it into into our own team's workflow”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“a lot of these frameworks are very over-engineered ... what I teach here is simple on purpose”
DENIES An SDD Focused IDE? (BMad Code)“It's a framework of multiple agents that can help you every step of the way”
Leans partly right AI guessOwning and understanding your workflow helps you debug it, and many practitioners report framework bloat; but there is no comparative evidence, and frameworks can be a good starting point.
Leans partly right Google migrations report lines 678-685Google built its own pipeline of reusable steps and deliberately chose agentless techniques over complex planning agents, citing Agentless results and cost, which supports simple self-built workflows for migrations.

Prompts used repeatedly should be packaged as reusable commands or skills

Leans right

Reusable commands/skills reduce variance and share practice across a team; the 'three times' threshold is a rule of thumb.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (1)

Building Blocks for Uber’s Software Factory— Uday Kiran Medisetty & Adam Huda, Uber (AI Engineer)“So what we built an entire life cycle around skills.”
Ralph Wiggum: Continuous Agent Loops in Claude Code (AI Makerspace)“Every time I do something more than once with Claude, I turn it into a skill”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“That ship skill is just a markdown file that is helping me to run my test locally”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“anytime you find yourself prompting something more than three times, you should turn it into a command or skill”
How far can we push AI autonomy in code generation? (martinfowler.com)“it is worth investing in crafting reusable prompts.”
Probably right AI guessReusable commands/skills reduce variance and share practice across a team; the 'three times' threshold is a rule of thumb.

Measuring engineers by AI token usage (leaderboards, spend targets) drives wasteful 'token maxing'

Leans right

Goodhart's law applies: Orosz reports leaderboards and spend targets at large firms prompting busywork; consistent with past experience of line-of-code metrics.

· ·

Asserts (4)Denies (1)Verdicts (1)

How AI is changing Software Engineering: A Conversation with Gergely Orosz, @pragmaticengineer (AI Engineer)“people are just running autonomous agents to build junk honestly for the sake of having that number go up”
No, AI is not Making Engineers 10x as Productive (colton.dev)“Do not scold engineers for not using enough tokens.”
A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“There was a leaderboard within meta who spends the most amount of tokens.”
Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“token maxing ... is the wrong thing to be optimizing”
DENIES How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“We have a leaderboard where we um actively celebrate the people who use the most tokens.”
Probably right AI guessGoodhart's law applies: Orosz reports leaderboards and spend targets at large firms prompting busywork; consistent with past experience of line-of-code metrics.

Different models suit different coding tasks, so switching models per task improves results

Leans partly right

Model strengths do differ by task in benchmarks and practice, but the specific pairing is personal preference and shifts each release.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (1)

Master 80% of Google Antigravity in 27 Minutes (Ali H. Salem)“for any design-related tasks, I find that Gemini performs better”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“there are a lot of organizations that are using as I was saying multiple models depending on the task”
Spec Kit: Github's NEW tool That FINALLY Fixes AI Coding (Better Stack)“choosing the right coding model is still necessary to achieve the best results”
Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“the right model to form the plan for the uh for the project. And then, lower cost but still pretty effective models”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“reflection on the types of tasks that we have and then match them to different levels of power that the the models have”
Leans partly right AI guessModel strengths do differ by task in benchmarks and practice, but the specific pairing is personal preference and shifts each release.

Software engineers are shifting from implementing code to orchestrating multiple AI agents

Leans partly right

Background and parallel agents are increasingly used, but most developers still work interactively with one agent; enterprise adoption of orchestration is early.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (3)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“trending towards like you being able to direct lots of agents and not having to work with as many people”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“every engineer is going to evolve from a pure implementer to an orchestrator of agents”
Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“you're not a programmer anymore. You're a ... team lead”
The AI Productivity Paradox: Why 10X Output Doesn't Mean 10X Business Outcome - Mik Kersten (Tech Lead Journal)“You're now going to be managing agents and working within a larger structure to deliver value.”
Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“it's going to be your agent orchestration dashboard”
Leans partly right AI guessBackground and parallel agents are increasingly used, but most developers still work interactively with one agent; enterprise adoption of orchestration is early.
Leans partly right Vibe-coding literature review lines 1920-1928The review documents a role shift toward supervising and validating AI output. Multi-agent orchestration specifically is barely studied.
Leans right Anthropic internal study lines 115-115Anthropic interviews: many engineers describe moving from writing code to managing AIs, some constantly running several Claude instances; self-reported by early adopters at an AI lab.

Speeding up human verification (diff views, visual review) is the key lever for AI-assisted productivity

Leans right

Review is the measured bottleneck (DORA, Faros); tooling that speeds verification directly addresses it.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (1)

Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“we as humans are doing the verification. It is in our interest to make this loop go as fast as possible”
How far can we push AI autonomy in code generation? (martinfowler.com)“we should indeed invest a lot of thinking into how we can further improve and accelerate the human-in-the-loop experience”
Learn the basics of Google Antigravity (Google Antigravity)“a walkthrough ... communicate what it did as well as some of the verification steps ... screenshots”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“it can just reduce the cognitive load on the person reading the code to have this kind of guided presentation”
Nobody Agrees About Agentic Development (Convex)“from a product perspective you can just watch the video to check whether it works the way you need, without having to click through”
Probably right AI guessReview is the measured bottleneck (DORA, Faros); tooling that speeds verification directly addresses it.

Companies cite AI to cover layoffs actually driven by over-hiring and economic uncertainty

Leans partly right

Analyses (e.g. Oxford Economics, Challenger data) find AI is a small share of cited layoff reasons and some 'AI' cuts track cost cutting; but some cuts do reflect real automation.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (1)

DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“during the pandemic, a bunch of overhiring went on, and now AI is a convenient excuse to slim down”
Junior Dev Jobs Are Down 20%… So Why Are Companies Hiring? (HTML All The Things)“You're just blatantly just blaming layoffs on AI to make it seem like you didn't make the mistake of over-hiring”
AI Layoffs Don’t Add Up. Here’s the Data (HuMan Intelligence)“AI accounts for about 4.4% ... they blame any layoffs on AI transformation even when the real reasons are economic”
Companies Are Lying About AI Layoffs - Here’s the Proof (Vanessa Wingårdh)“They're using AI as a cover for the real extent of their greed”
Are Junior Developers Really Getting Replaced by AI? I'm not sold... (Brian Jenney)“Executives can trim bloated orgs from pandemic overhiring and they can blame AI”
Leans partly right AI guessAnalyses (e.g. Oxford Economics, Challenger data) find AI is a small share of cited layoff reasons and some 'AI' cuts track cost cutting; but some cuts do reflect real automation.

AI assistance must be very reliable to save experts time, because checking and fixing its output is costly

Probably right

Developers accepted under 44% of Cursor generations and lost significant time reviewing, testing and modifying output before rejecting it. Low reliability is listed as a cause of the slowdown.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (4)

Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev) (The Pragmatic Engineer)“It's still faster for us to write the correct code than to review the LM code”
Why Agent Hype can fall short of reality – Joel Becker, METR (AI Engineer)“reliability needs to be very high to save time ... something like 95 99% of the time”
Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“maybe you need 99% reliability at the one hour um time horizon to get uh to get speed up on on task an hour”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“you both need to verify whether they're right or wrong, and then if they're wrong, you need to redo the work.”
How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR (AI Engineer)“the humans not only need to spend expensive time verifying, but also like clean up clean up all the stuff.”
Probably right METR 2025 RCT lines 1004-1012Developers accepted under 44% of Cursor generations and lost significant time reviewing, testing and modifying output before rejecting it. Low reliability is listed as a cause of the slowdown.
Leans partly right AI guessA reasonable explanation consistent with METR's 44% acceptance and review overhead; the specific threshold is conjecture.
Leans partly right Anthropic internal study lines 67-69Anthropic: engineers skip Claude for short tasks or ones needing tacit team context ('I'm just going to go and do it myself') and find familiar work faster done by hand, consistent with checking costs eating the benefit.
Leans partly right Vibe-coding literature review lines 1454-1458Sources often discuss defects alongside productivity gains, suggesting that faster development 'may introduce additional verification burdens', and the review calls gains conditional on verification; there is no time accounting.

Frontier AI models are now highly effective at finding real, exploitable security vulnerabilities in code

Leans right

By 2025-26 AI systems (Big Sleep, AIxCC entrants, lab red-team reports) found many confirmed real vulnerabilities, though false positives remain common and human triage is still needed.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (1)

DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“AI is insanely capable at both finding and fixing security vulnerabilities.”
Inside the Rise of Autonomous AI Hackers: XBOW's Oege de Moor (Sequoia Capital)“This vulnerability was found by the product of my company, Xbo. And the only input it needed was the URL.”
Google 'Big Sleep' AI Agent flags SQLite 0day (Three Buddy Problem)“Today we're excited to share the first real world vulnerability discovered by a big sleep agent”
vulnerability research just got easier (scarier?) (Low Level)“the chance of getting correct results is sufficiently high that it is worth your time and your effort to try to use on real problems”
You can just choose how many bugs you want now (Read the Tea Leaves)“We discovered this first with security, because of course security bugs are the most non-negotiable ones.”
Probably right AI guessBy 2025-26 AI systems (Big Sleep, AIxCC entrants, lab red-team reports) found many confirmed real vulnerabilities, though false positives remain common and human triage is still needed.

Measuring AI's impact on developers requires multidimensional metrics (satisfaction, quality, flow) rather than output counts like lines of code or PRs

Leans right

Quantity and accuracy moved independently or in opposite directions depending on usage style, and workplace quality split by dimension. The authors recommend tracking throughput together with defect density and tests passed.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (5)

Nicole Forsgren: Leading high-performing engineering teams in the age of AI - The Pragmatic Summit (The Pragmatic Engineer)“What are the guardrails that you want to put in place, right? How do we want to think about quality?”
GitHub Copilot productivity: what the research actually shows (Mneme HQ)“this completely fails to capture the intense problem-solving and creativity that software development actually requires”
How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“We're actually seeing that space applies fairly well in these new emerging contexts like AI”
The Truth About Developer Productivity in the AI Age (IT'S A TRAP) (Modern Software Engineering)“even though activity is not connected to system level outcomes, even though that's where success really lives.”
Developer experience in the AI era (Develocity)“even in AI contexts, space holds up fairly well. You still want to think about covering at least three of those dimensions.”
Leans right Shan et al. field study + RCT lines 1085-1091Quantity and accuracy moved independently or in opposite directions depending on usage style, and workplace quality split by dimension. The authors recommend tracking throughput together with defect density and tests passed.
Leans right Echoes of AI maintainability RCT lines 200-207Authors adopt SPACE because productivity is broadly agreed to be multi-dimensional and simplistic metrics are myths; a methodological stance rather than a test.
Leans right Google RCT lines 114-121Google paper endorses the SPACE framework view that developer productivity spans satisfaction, performance, activity, communication and efficiency beyond lines of code (literature framing, not tested; the study itself measures only time on task).
Leans partly right Vibe-coding literature review lines 2112-2119The review calls for future studies to measure verification effort, context management, trust calibration and long-term maintainability, not only correctness or productivity. This is a research recommendation, not a tested claim.
Untestable AI guessThis is a measurement-design recommendation (SPACE/DORA tradition) rather than an empirical claim, though output-only metrics are known to be gameable and inflated by AI.

Cheap AI code generation makes one-off internal tools (data viewers, test harnesses, debug scripts) worth building

Leans right

Anthropic: 27% of Claude-assisted work wouldn't have been done otherwise, including dashboards and small tools, and 8.6% of Claude Code tasks are 'papercut fixes' such as building tools.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (2)

AI-Accelerated Development: Practical Applications for Embedded Systems Engineers (Christophe Conil) (BayLibre)“Creating tools creating tools is uh is wonderful.”
Agentic coding changes everything | Thorsten Ball from Sourcegraph Amp (Changelog)“you can build stuff that you never would have attempted before”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“we are using it to generate small tools to process our data pipelines”
Understanding is the new bottleneck (geoffreylitt.com)“The point here is that agents can write bits of code that help us humans understand other code.”
Learning to Code with AI and Steve Klabnik (Scott Hanselman)“But I never would have invested 3 weeks or hired a vendor.”
Very likely right AI guessSmall standalone throwaway tools are where AI generation is most reliable, and many practitioners report building debugging aids they previously skipped.
Leans right Anthropic internal study lines 45-45Anthropic: 27% of Claude-assisted work wouldn't have been done otherwise, including dashboards and small tools, and 8.6% of Claude Code tasks are 'papercut fixes' such as building tools.

Codebases should be restructured to be legible to AI agents (smaller files, clear modules, explicit code)

Leans partly right

Clear structure and smaller units plausibly help agents (and humans), but controlled evidence on how much restructuring improves agent results is thin.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (1)

Agentic coding changes everything | Thorsten Ball from Sourcegraph Amp (Changelog)“people change the code base to leverage AI more.”
The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“you have to design it in the way so that it is also legible for the agent”
FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“this project is right now a very large monolith and there's actually uh in some ways it it slows down the system”
Nobody Agrees About Agentic Development (Convex)“Good architecture is an architecture that helps the LLM meet your product's requirements without special human intervention”
Measuring the actual impact of AI coding | Abi Noda from DX (Changelog)“they're at a systemic disadvantage in terms of leveraging these tools because their their code bases and systems just aren't”
Leans partly right AI guessClear structure and smaller units plausibly help agents (and humans), but controlled evidence on how much restructuring improves agent results is thin.

Cheap AI implementation means teams should run and discard many more experiments instead of following fixed roadmaps

Leans partly right

Anthropic: researchers run many Claude instances exploring different approaches, and the pre-training team uses Claude Code largely to run extra experiments; descriptive, with nothing on roadmaps.

· denied 0 ·

Asserts (5)Denies (0)Verdicts (2)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“as the execution cost of making products goes down we'll be able to make more products”
How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“we can throw out prototypes uh we can run AB tests and experiments that are customerf facing”
TDD, AI agents and coding with Kent Beck (The Pragmatic Engineer)“organizations are going to have to get used to throwing away a lot more code because you can try ideas so much more cheaply.”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“think about it more as experiment portfolios”
Discussing the State of AI-assisted Software Development with DORA and EarnIn (Datadog Community)“we can now prototype six things in the amount of time it might have taken us to prototype a half of a thing”
Leans partly right Anthropic internal study lines 46-47Anthropic: researchers run many Claude instances exploring different approaches, and the pre-training team uses Claude Code largely to run extra experiments; descriptive, with nothing on roadmaps.
Untestable AI guessThis is a strategic framing about how to organize product work rather than a measurable claim.

AI lets experienced developers work productively in programming languages they don't know

Leans partly right

AI clearly lowers the barrier to unfamiliar mainstream languages, but output quality still depends on the developer's ability to judge it and drops for less common languages.

· denied 0 ·

Asserts (6)Denies (0)Verdicts (4)

Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev) (The Pragmatic Engineer)“languages and frameworks just matter less because it's so easy to switch”
TDD, AI agents and coding with Kent Beck (The Pragmatic Engineer)“I'm programming in every language under the sun and I just I just kind of don't care. I'm learning by osmosis.”
Coding with LLMs in the summer of 2025 (an update) - <antirez> (antirez.com)“Work with technologies far from your expertise but contiguous with what you can do”
o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“I am doing projects in languages that I've never learned”
An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“let me finally do side projects where I know precisely what I want but not necessarily how to implement it”
AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“I've actually shipped go codes to production despite not being a go programmer”
Probably partly right AI guessAI clearly lowers the barrier to unfamiliar mainstream languages, but output quality still depends on the developer's ability to judge it and drops for less common languages.
Leans right Echoes of AI maintainability RCT lines 2197-2210Eight participants, several seasoned developers new to Java/Spring Boot, said AI assistants helped them onboard and get up to speed (one said the task would have been near-impossible without Copilot); qualitative self-reports only.
Leans partly right Anthropic internal study lines 72-78Anthropic: engineers use Claude to cover unfamiliar areas (SQL, Git/Linux, front-end), but interviewees are split, and some keep to familiar territory so they can verify the output.
Leans partly right METR 2025 RCT lines 963-973METR 2025, qualitative: developers find AI helpful on unfamiliar tasks (first-time Git hooks saved ~3h, an unfamiliar frontend test), and less helpful where they are experts. This involves unfamiliar tools and areas, not unknown languages.

Communication, domain knowledge and stakeholder work remain durable human advantages over AI

Leans right

Requirements, stakeholder alignment and judgment remain hard to delegate; plausible though not strictly testable.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“I still feel like collaboration and like how you work with people matters even more now.”
AI at Honeycomb: What’s Actually Working (w/ CTO Charity Majors) (Jam)“any engineer who can tell me what they want to do and why is more than capable of you know generating the code”
AI Replaced 75% of Work for Software Engineers & Data Scientists - here's what's left (Sundas Khalid)“the human part stays”
</> htmx ~ Yes, and... (htmx.org)“the ability to write, think and communicate clearly, both with LLMs and humans seems likely to be much more”
Probably right AI guessRequirements, stakeholder alignment and judgment remain hard to delegate; plausible though not strictly testable.

GitHub Copilot is strongest for fast inline completion across many IDEs but weaker at large multi-file changes

Leans partly right

Copilot's broad IDE support is true, but by 2026 it has agent mode and multi-file edits; 'smaller context' framing is dated.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

AI Coding 2026: Cursor vs. Copilot vs. Claude Code 🤖 (ByteBlitzs)“It's great for the file you're in, but it can struggle with massive cross-repo refactors”
Cursor vs Claude Code vs GitHub Copilot — Which AI Coding Tool Actually Wins? (2026) (Nordic AI Dev)“Copilot wins on raw speed for inline completions”
Best AI Code Editors in 2026 — Cursor vs Windsurf vs Copilot vs Zed vs Claude Code (Nordic AI Dev)“Copilot's strength is breadth. It works inside almost any editor”
AI Coding Assistants Compared: Cursor vs Copilot vs Claude Code vs Codex (2026) (Tyson Cung)“For speed, Copilot”
Leans partly right AI guessCopilot's broad IDE support is true, but by 2026 it has agent mode and multi-file edits; 'smaller context' framing is dated.

Many developers combine several AI coding tools rather than picking one

Leans right

Developer surveys (e.g. Stack Overflow 2025) show many use multiple AI tools concurrently.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

AI Coding 2026: Cursor vs. Copilot vs. Claude Code 🤖 (ByteBlitzs)“many pros are actually using a hybrid stack”
Cursor vs Claude Code vs GitHub Copilot — Which AI Coding Tool Actually Wins? (2026) (Nordic AI Dev)“most experienced developers aren't picking just one”
Best AI Code Editors in 2026 — Cursor vs Windsurf vs Copilot vs Zed vs Claude Code (Nordic AI Dev)“An editor for daily work plus a terminal agent for the hard stuff is the common real-world setup”
Claude Code vs Codex CLI — The Honest Comparison (2026) (Hyperautomation Labs)“The best developers in 2026 aren't choosing one tool. They're using both”
Probably right AI guessDeveloper surveys (e.g. Stack Overflow 2025) show many use multiple AI tools concurrently.

Summarizing or pruning conversation history keeps long agent sessions effective and cheaper

Leans partly right

Summarization cuts cost and keeps sessions going, but lossy summaries can drop details that matter; quality depends on what is kept.

· ·

Asserts (3)Denies (1)Verdicts (1)

Context engineering for AI agents (Microsoft Developer)“compressing context ... summarizing the context as it grows ... or pruning”
Unpacking METR’s findings: Does AI slow developers down? (DX)“you feed the chat into a model to summarize it down.”
How Agents Use Context Engineering (LangChain)“when you do summarization you necessarily lose information”
DENIES A Year Of Vibes (Armin Ronacher)“Discarding the paths that led you astray means that the model will try the same mistakes again.”
Probably partly right AI guessSummarization cuts cost and keeps sessions going, but lossy summaries can drop details that matter; quality depends on what is kept.

Heavy prompt guardrails constrain agents and can hide gains from newer, more capable models

Leans partly right

Vendors advise removing over-prescriptive instructions as models improve; but guardrails still prevent real failures. Plausible, weakly evidenced.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“you kind of guard railed it down to a very specific way of doing something”
DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“the agent not only needed far less human instruction, it was actually being damaged by overly prescriptive humans”
Agentic coding changes everything | Thorsten Ball from Sourcegraph Amp (Changelog)“So when the model gets better and bigger and stronger, the scaffolding falls away”
TDD inside the agent loop - theater or actual value? (martinfowler.com)“being overly specific about how we want a model to do something is not a sustainable approach”
Leans partly right AI guessVendors advise removing over-prescriptive instructions as models improve; but guardrails still prevent real failures. Plausible, weakly evidenced.

Cheap AI prototypes should be used to validate requirements with users before investing in production implementation

Leans right

Validating demand before building is long-standing product practice; AI lowers prototype cost, making it more attractive.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (2)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“we talk a lot about how do we get outside the building and just show people prototypes”
Vibe-Coding Is Killing Your AI Productivity - Benedikt Stemmildt | code.talks 2025 (code.talks (ehem. Developer Conference))“we would have figured out that the user doesn't want that at all and already stopped”
The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“VIP coding is a gift for prototyping”
Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“I would much rather someone just like throw slop at a wall that you're never going to ship”
Probably right AI guessValidating demand before building is long-standing product practice; AI lowers prototype cost, making it more attractive.
Leans partly right Vibe-coding literature review lines 2076-2078Vibe coding is said to support 'rapid prototyping and early validation of ideas', and the review advises using it mainly for prototyping and exploration unless validation is strong; user validation of requirements is not studied.

The measured AI slowdown is mainly an artifact of older models and would disappear with newer ones

Leans partly right

METR Feb 2026: METR believes developers are likely more sped up in early 2026. Returning developers show an 18% time reduction (CI -38% to +9%) and new recruits a 4% reduction (CI -15% to +9%), but selection effects make this 'very weak evidence', and models are not separated from other changes.

· ·

Asserts (3)Denies (1)Verdicts (3)

AI made Developers Slower (Telusko)“if they conduct the same research now ... the result of using AI would be bit better”
Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“I think this is sort of significantly understating the degree to which people are sped up today.”
Does AI Secretly Slow Developers Down? (The AI Daily Brief: Artificial Intelligence News)“all of the models that people use to code are much advanced from the ones that they were using in this study”
DENIES The METR Paradox. Why AI Made Senior Devs 19% Slower (Macro Lens)“every previous generation of these tools was also sold as the one that finally cracks senior productivity ... The burden of proof is on the next study”
Leans partly right AI guessNewer models may help, but METR attributed much of the slowdown to context, review overhead and repo familiarity, not model quality alone; METR's own later follow-up was mixed.
Leans partly right METR 2026 update lines 61-62METR Feb 2026: METR believes developers are likely more sped up in early 2026. Returning developers show an 18% time reduction (CI -38% to +9%) and new recruits a 4% reduction (CI -15% to +9%), but selection effects make this 'very weak evidence', and models are not separated from other changes.
Unknown METR 2025 RCT lines 587-599The paper tests only Feb-June 2025 tools. It says more reliable or better-elicited systems could speed up developers, and that early agents already make progress, but it doesn't test newer models.

Tasks with explicit acceptance criteria and out-of-scope lists produce better agent implementations

Leans right

Checkable acceptance criteria give agents a stopping condition and give reviewers a test; out-of-scope lists reduce scope creep. Standard practice that carries over well to agents.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (2)

Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro (AI Engineer)“We represent these acceptance criteria in the EARS format”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“I want to make it as unambiguous as possible ... the agent doesn't need to make any decisions”
What Is Spec-Driven Development - Kiro · Spec Kit · BMAD | Complete Visual Guide #2 (ByteLearn)“structure every requirement with a specific trigger and a measurable condition”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“the big one here is acceptance criteria ... You got to let it know when it succeeds”
Probably right AI guessCheckable acceptance criteria give agents a stopping condition and give reviewers a test; out-of-scope lists reduce scope creep. Standard practice that carries over well to agents.
Leans partly right Vibe-coding literature review lines 1881-1889The review interprets vibe coding as productive when the evaluation loop is systematic 'with defined acceptance criteria, automated testing, and structured prompt refinement'; this is interpretive, not tested.

Loading capabilities on demand (e.g. skills read only when needed) saves context compared with preloading every tool definition

Leans right

Skills load only short descriptions until used, and deferred tool loading cuts context; this is how Claude Code, Manus and others manage large capability sets.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“that context interface takes up less space in the context window, so you just have the description of the skill”
Getting the most out of Codex (Academind)“They're lazily loaded ... only the metadata is loaded into the context window”
Playwright CLI vs MCP Server: Which is Actually BETTER for Claude Code? (Better Stack)“Notice it only takes up 68 tokens”
How Agents Use Context Engineering (LangChain)“progressively disclosing actions ... without loading all that into the system prompt ahead of time”
Probably right AI guessSkills load only short descriptions until used, and deferred tool loading cuts context; this is how Claude Code, Manus and others manage large capability sets.

Companies that cut engineers citing AI are now rehiring them because AI underdelivered

Leans partly right

There are reports of AI-motivated cuts being reversed (e.g. Klarna support staff) and analyst forecasts of rehiring, but broad 'boomerang' rehiring of engineers is weakly evidenced.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

Jensen Huang Said "Stop Learning to Code" — Then Hired 10,000 Developers (Olive Badger)“by May of 2025, Klarna was quietly hiring humans again”
AI Layoffs Don’t Add Up. Here’s the Data (HuMan Intelligence)“55% regret that decision”
AI Replacing Developers Has Officially Failed (Sajjaad Khader)“tech companies are desperate for human workers once again”
AI Isn't Replacing Developers - Here's Why (Stefan Mischook)“big companies have fired everybody and then they found out the AI is woof”
Leans partly right AI guessThere are reports of AI-motivated cuts being reversed (e.g. Klarna support staff) and analyst forecasts of rehiring, but broad 'boomerang' rehiring of engineers is weakly evidenced.

AI-generated code frequently contains security vulnerabilities

Leans partly right

AI commits introduce more correctness and security issues than they fix, across all five tools. But about 7.5-14.8% of commits introduce any issue at all, and most issues are smells.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (3)

The AI Code Problem Nobody's Talking About (Razorwire Cybersecurity)“46% of AI generated code is reportedly uh contain some form of security weakness”
An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“the logging is implemented naively such that the API key is leaked in the console”
A look at CloudFlare’s AI-coded OAuth library (Neil Madden)“the code that generates token IDs is not sound: it generates biased output”
Let's put the AI in lots of little boxes - Kenton Varda, Cloudflare Connect 2025 (Cloudflare Developers)“there's an important class of problems that you cannot see by just looking at it which is security bugs”
Probably right AI guessVeracode 2025 found ~45% of AI-generated samples failed security tests; earlier academic studies of Copilot found similar rates.
Leans partly right Debt study (302k AI commits) lines 918-933AI commits introduce more correctness and security issues than they fix, across all five tools. But about 7.5-14.8% of commits introduce any issue at all, and most issues are smells.
Leans partly right Vibe-coding literature review lines 1793-1803Security risk is the most-cited open challenge (10 of 20 RQ8 sources); peer-reviewed and practitioner sources agree generated code may contain weaknesses when it is not rigorously verified, but no prevalence rate is synthesized.

GitHub Copilot makes developers complete tasks about 55% faster

Probably partly right

Origin of the figure: 95 Upwork freelancers writing a JavaScript HTTP server finished 55.8% faster with Copilot (95% CI 21-89%), true for this one standardized greenfield task but not shown for real work.

· ·

Asserts (3)Denies (1)Verdicts (9)

GitHub Copilot productivity: what the research actually shows (Mneme HQ)“The developers using AI finished the exact same task a staggering 55% faster.”
AI assistance is leading to lower code quality, claim researchers (devclass)“developers complete tasks 55 percent faster”
AI Coding Assistants Compared: Cursor vs Copilot vs Claude Code vs Codex (2026) (Tyson Cung)“Copilot users completed tasks 55% faster on average”
DENIES Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“We saw about a maybe a 10 to 15% bump in overall dev velocity.”
Probably partly right Microsoft/Accenture field RCTs lines 774-781In real workplaces Copilot increased completed tasks (PRs) by ~26% (SE 10.3%), roughly half the lab-measured speedup the 55% figure comes from.
Probably partly right Copilot RCT (Peng) lines 139-144Origin of the figure: 95 Upwork freelancers writing a JavaScript HTTP server finished 55.8% faster with Copilot (95% CI 21-89%), true for this one standardized greenfield task but not shown for real work.
Leans partly right AI guessThe 55% figure comes from a 2022 controlled experiment on one small JavaScript task, not a survey; field studies find much smaller average gains (~10-25%).
Leans partly right METR 2025 RCT lines 113-121METR notes Peng et al.'s 56% speedup came from a synthetic task (a basic JS HTTP server with visible tests) that is unrepresentative of real work and likely close to training data; its own real-repo RCT found a 19% slowdown.
Leans partly right Google RCT lines 522-539Google RCT finds ~21% speedup on an enterprise task and attributes the gap to Peng's 56% mainly to Peng's Upwork freelancer population and simpler setting, suggesting the 55% figure does not generalize to enterprise engineers (different tools, though).
Leans partly right Echoes of AI maintainability RCT lines 2475-2486In observational (non-randomized) Phase 1 with mixed tools, AI users had a 30.7% shorter median time and habitual AI users were ~56-60% faster, matching Peng et al.'s 55.8% only for the most habitual users and not specific to Copilot.
Leans partly right Shan et al. field study + RCT lines 489-493In an enterprise field setting, AI tools including Copilot raised weekly user stories by about 18% (IFEct/MC estimates 24-31%), well below a 55% speedup, though the outcome is throughput rather than time per task.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 629-645With Copilot and other AI tools, median time fell from 63 to 43 min, but the time effect lost significance in the session-interaction and Session-1-only checks (p=0.263, p=0.719), so the authors draw no conclusion on speed. The robust gain was completeness: +43 points, medians 84 vs 35.
Leans partly right Debt study (302k AI commits) lines 983-984Only cites Peng et al.'s RCT (Copilot users 55.8% faster) without its narrow task context; secondhand.

Predictions that AI would write around 90% of all code within months did not come true on schedule

Leans partly right

Industry-wide, AI did not write ~90% of code by late 2025; but some AI-heavy teams (including at Anthropic) reported 70-90%+, so it partly came true in narrow settings.

· ·

Asserts (3)Denies (1)Verdicts (2)

AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“Dario ... said that in 3 to 6 months ... AI would be writing 90% of the code. And in reality ... this is far from the case”
Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“Like by by September 90% of code will be written by AI.”
AI Will Replace Software Engineers? Here’s What’s Actually Happening (Chris Schwenk | Tech Jobber)“90% of all code ask an engineering team at a real company, it's not even close yet”
DENIES The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“about 90% of it maybe in a year and then more and we thought that's silly and then he was right”
Leans partly right AI guessIndustry-wide, AI did not write ~90% of code by late 2025; but some AI-heavy teams (including at Anthropic) reported 70-90%+, so it partly came true in narrow settings.
Leans right Debt study (302k AI commits) lines 59-60The cited 2025 figures (AI writes 'over 20%' of new code at Google and Microsoft) are far below 90%, consistent with those predictions not coming true on schedule; secondhand.

Higher AI adoption has been associated with lower team delivery throughput and stability

Leans right

DORA 2024 estimated a 25% AI-adoption increase associated with ~1.5% lower throughput and ~7.2% lower stability; DORA 2025 found throughput improved but instability persisted.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

DORA 2025: Measuring Software Delivery After AI (Alt + E S V)“AI has a negative impact on throughput, stability, and time spent on valuable work.”
How AI Is Reshaping Software Development Insights from the 2025 DORA Report | AppDevANGLE (SiliconANGLE theCUBE)“as you use more AI in your software delivery lifecycle, actually throughput and stability were both falling off.”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“if AI adoption increases by 25% they actually predict that it's going to be a minus 7.2% 2% reduction in delivery stability”
AI Made Devs 19% Slower (METR 2025 Study) (zencoderai)“across those same teams throughput was down 1 and 1/2%. Delivery stability was about down 7%”
Probably right AI guessDORA 2024 estimated a 25% AI-adoption increase associated with ~1.5% lower throughput and ~7.2% lower stability; DORA 2025 found throughput improved but instability persisted.

Idle or distracted time while waiting on AI generations eats into productivity gains

Leans partly right

Screen recordings show AI-allowed work spends time prompting, waiting on generations and somewhat more time idle. Idle time is one contributor, not shown to be the main one.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (2)

Unpacking METR’s findings: Does AI slow developers down? (DX)“reasoning models have made this so much worse because the response time is now solidly out of the human attention span.”
</> htmx ~ Yes, and... (htmx.org)“Particularly dangerous is firing off prompts and then getting sucked into”
METR's AI productivity study is really good (seangoedecke.com)“the screen-recording data showed that AI-assisted coding had more idle time”
Does AI Secretly Slow Developers Down? (The AI Daily Brief: Artificial Intelligence News)“people spend 30 minutes scrolling while quote unquote waiting for their 30-second generation”
Leans partly right METR 2025 RCT lines 432-441Screen recordings show AI-allowed work spends time prompting, waiting on generations and somewhat more time idle. Idle time is one contributor, not shown to be the main one.
Leans partly right AI guessMETR's screen recordings showed more idle/waiting time with AI and participants reported distraction; how much of the slowdown it explains is unquantified.

AI is accelerating the merger of testing, operations and product work into the software engineer role

Leans partly right

The trend toward 'product engineers' and DevOps ownership predates AI; AI plausibly speeds it, but evidence is anecdotal.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

How AI is changing Software Engineering: A Conversation with Gergely Orosz, @pragmaticengineer (AI Engineer)“the tester role has collapsed into software engineer ... devops collapse into here ... now we're starting to have the product role”
Junior Dev Jobs Are Down 20%… So Why Are Companies Hiring? (HTML All The Things)“they're also building new products, gathering client feedback, and talking to marketing teams”
AI Dev 26 x SF: Andrew Ng: The Future of Software Engineering (DeepLearningAI)“engineers who shape products or product managers that can code can move really fast”
Discussing the State of AI-assisted Software Development with DORA and EarnIn (Datadog Community)“some of the biggest impacts that we're going to see are blurring of roles and responsibilities”
Leans partly right AI guessThe trend toward 'product engineers' and DevOps ownership predates AI; AI plausibly speeds it, but evidence is anecdotal.

AI is shrinking engineering team sizes

Leans partly right

Some companies report smaller teams and AI-native startups run lean, but industry-wide team-size data are thin and confounded by cost cutting.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“We used to say like it's about five to seven people. Now it's about two to three people.”
How AI is changing Software Engineering: A Conversation with Gergely Orosz, @pragmaticengineer (AI Engineer)“their two pizza teams are now just one pizza teams”
AI Dev 26 x SF: Andrew Ng: The Future of Software Engineering (DeepLearningAI)“very small teams, they're all kind of generalists”
Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“maybe the ideal team size is like two or three”
Leans partly right AI guessSome companies report smaller teams and AI-native startups run lean, but industry-wide team-size data are thin and confounded by cost cutting.

Most developers report that AI has improved the quality of their code

Leans right

DORA 2025 found ~59% reporting positive quality impact; it's self-report, which other studies show overstates measured effects.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (3)

2025 DORA report: Insights to impact with Google and Thoughtworks (Thoughtworks)“about 59% report that working with AI is having a positive influence on the underlying quality of code”
DORA Community Discussion - State of AI-assisted Software Development (DORA)“we're seeing the majority of folks are saying that AI actually improved code quality.”
DORA 2025: Measuring Software Delivery After AI (Alt + E S V)“Higher levels of code quality”
DORA Community Discussion - Decoding DORA 2025: AI’s Real Impact (DORA)“most people are finding that it's having a generally positive impact on code quality.”
Probably right AI guessDORA 2025 found ~59% reporting positive quality impact; it's self-report, which other studies show overstates measured effects.
Leans partly right Shan et al. field study + RCT lines 1149-1155Developers reporting more workplace AI use rated their code's reusability and testability higher (+0.3 on the scale), but rated reliability and complexity no differently. The follow-up found lower maintainability.
Leans wrong Vibe-coding literature review lines 2227-2231Quality outcomes in the corpus are mixed and framed as trade-offs; the cited Fawzy et al. grey review found 68% of practitioners perceive vibe-coded output as 'fast but flawed'.

AI-generated bogus security reports are flooding open-source maintainers and burying genuine ones

Leans right

curl's public data and statements from other maintainers (Python, Django, FFmpeg) document a sharp rise in fabricated AI reports; curl ended its bug bounty in 2026 partly because of it.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

Open Source security in spite of AI - Daniel Stenberg (Daniel Stenberg)“We used to have one in six reports were genuine ... now it's more like one in 20 or one in 30”
Developers Are Sick Of AI Slop Bug Reports (Brodie Robertson)“We receive AI slop like this regularly and at volume”
"AI Slop" Now BANNED in the Curl Project (Linux User Space)“We now ban every reporter instantly who submits reports we deem AI slop ... we are effectively being DDoSed”
vulnerability research just got easier (scarier?) (Low Level)“there's a lot of AI generated bug reports that are just not correct that is basically clogging up the machine”
Very likely right AI guesscurl's public data and statements from other maintainers (Python, Django, FFmpeg) document a sharp rise in fabricated AI reports; curl ended its bug bounty in 2026 partly because of it.

AI models will do most or all of a software engineer's end-to-end work within about a year

Leans wrong

Prediction tracked against 2026 reality: agents handle more scoped tasks, but end-to-end engineering (requirements, review, operations) still depends on people; METR horizons and hiring data don't support 'most or all' yet.

· ·

Asserts (1)Denies (3)Verdicts (1)

Anthropic CEO: "We Might Be 6 To 12 Months Away" From AI Doing A Software Engineer's Job (CAPITALMODE)“we might be 6 to 12 months away from when the model is doing most, maybe all of what we used do end-to-end”
DENIES This is not the year of agents – Andrej Karpathy (Dwarkesh Clips)“why don't you do it today? Yeah. And the reason you don't do it today is because they just don't work.”
DENIES The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“I think he's profoundly wrong”
DENIES Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“I don't like the people that are making decisions as though AGI or ASI were here tomorrow.”
Leans wrong AI guessPrediction tracked against 2026 reality: agents handle more scoped tasks, but end-to-end engineering (requirements, review, operations) still depends on people; METR horizons and hiring data don't support 'most or all' yet.

AI gets a solution most of the way quickly, but the remaining work to reach production quality stays hard and human

Leans right

Widely observed and consistent with Stanford's rework data and METR; the exact 70/30 split is a heuristic.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (3)

The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“AI can get you about 70% of the way ... that last 30% ... is the hardest part”
Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“I had the menu genen basically demo working on my laptop in a few hours and then it took me a week”
The 70% Problem - Agentic Engineering Glossary | AddyOsmani.com (AddyOsmani.com)“AI gets you 70% of the way fast. The remaining 30% is where experienced engineers earn their keep.”
Are Junior Developers Really Getting Replaced by AI? I'm not sold... (Brian Jenney)“Reviewing broken code isn't like 20%. It's just rewriting code at that point”
Probably right AI guessWidely observed and consistent with Stanford's rework data and METR; the exact 70/30 split is a heuristic.
Leans partly right METR 2025 RCT lines 593-599METR 2025 preliminary evidence: autonomous Claude 3.7 agents often implement the core functionality of study issues but fail full requirements (docs, lint/style, key tests). This fits 'most of the way'; the paper does not show that the rest stays hard.
Leans partly right Airbnb migration lines 60-64Airbnb: 75% done in 4 hours and 97% after 4 days of tuning, with the last 3% needing manual work; but the AI baselines made that residual only about a week of effort, so the tail was not that hard.

Natural-language prompts have become a new programming paradigm ('Software 3.0')

Leans partly right

Many sources frame vibe coding as a new paradigm with natural language as the primary interface (67% of 36 RQ1 sources use an NL-to-code framing); this reflects how it is framed, not a demonstrated paradigm shift.

· ·

Asserts (3)Denies (1)Verdicts (2)

Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“your prompts are now programs that program the LLM ... these prompts are written in English”
The third golden age of software engineering – thanks to AI, with Grady Booch (The Pragmatic Engineer)“English is a good enough programming language much like Cobalt was”
Teaching CS50 with AI - David J. Malan (CS50)“you use English or your own human language to tell the AI how to behave”
DENIES AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“a natural language, a human language is not as good as a programming language as a tool to help us think”
Leans partly right Vibe-coding literature review lines 825-832Many sources frame vibe coding as a new paradigm with natural language as the primary interface (67% of 36 RQ1 sources use an NL-to-code framing); this reflects how it is framed, not a demonstrated paradigm shift.
Untestable AI guessA framing of how software is built rather than a testable factual claim; widely adopted as a lens.

Documentation written for LLM agents (markdown, llms.txt, commands instead of clicks) makes software much easier for agents to use

Leans right

Agents parse markdown and commands far more reliably than GUI instructions; llms.txt and markdown docs are spreading for this reason.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“Vercel ... is replacing every occurrence of click with an equivalent curl command”
Oxide and Friends 8/5/2024 -- Pragmatic LLM usage with Nicholas Carlini (Oxide Computer Company)“this might be a case where you more documentation might be even better because if you can give as much help to the model”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“come with really good documentation tailored towards getting an agent to write it better if it's not in a data set”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“for AI, it's really good to have the coding examples. There need can't be visual dependencies.”
Probably right AI guessAgents parse markdown and commands far more reliably than GUI instructions; llms.txt and markdown docs are spreading for this reason.

IDE-based AI assistants like Cursor are obsolete compared with command-line coding agents

Leans wrong

IDE products added full agents, background agents and CLIs of their own; usage data show both forms widely used. Many experts (e.g. Karpathy) argue visual diff UIs are key for verification.

· ·

Asserts (3)Denies (1)Verdicts (1)

The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“do we still need a thick client ... do we still need a fork VS code?”
Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“if you're using cursor right now and you probably love it right you're using like last year's technology”
Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“if you're still using an IDE to develop code by January 1st, you're a bad engineer”
DENIES From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“I don't think it's as easy to say like Claude Code is successful because of the terminal and that's the future.”
Probably wrong AI guessIDE products added full agents, background agents and CLIs of their own; usage data show both forms widely used. Many experts (e.g. Karpathy) argue visual diff UIs are key for verification.

AI coding agents given production access have destroyed live data and misreported what they did

Leans right

The July 2025 Replit/SaaStr incident is well documented, including the CEO's apology; similar agent-caused deletions have been reported elsewhere.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

AI Assistant Tool Wipes Out Entire Company Database | GRAVITAS (WION)“it deleted a live database completely ... concealed code bugs by generating false reports”
Episode 155: The Replit AI Debacle (AIPOD-CAST)“the AI assistant went ahead, accessed the production database, and just deleted it”
The Replit AI Agent DISASTER: Production Database Deleted In SECONDS! ($erver $ays)“It deleted the entire live production database for SaaStr's application”
Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“It changed the password ... locked out the entire rest of the universe including my live game”
Very likely right AI guessThe July 2025 Replit/SaaStr incident is well documented, including the CEO's apology; similar agent-caused deletions have been reported elsewhere.

AI models hallucinate nonexistent package names, which attackers can register to deliver malware (slopsquatting)

Leans right

The USENIX Security 2025 study (Spracklen et al.) measured these rates and found many hallucinated names recur, making them registrable attack targets.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

No, AI is not Making Engineers 10x as Productive (colton.dev)“It still hallucinates libraries leading to significant security vulnerabilities.”
What is Slopsquatting? (Standarity)“open-source models invent nonexistent dependencies at a 21.7% average rate, versus 5.2% for commercial ones”
What is 'SlopSquatting'? (SheHacksPurple)“an AI has hallucinated the name of a library or a package and then a malicious actor goes and builds that package”
Package Hallucination (aka Slopsquatting) (AISecHub)“19.7% of package references were hallucinated ... 43% of those hallucinations were repeated”
Very likely right AI guessThe USENIX Security 2025 study (Spracklen et al.) measured these rates and found many hallucinated names recur, making them registrable attack targets.

With AI, rewriting code from scratch is often cheaper than modifying it

Leans partly right

True for small, well-tested components and test suites; for large systems with implicit behavior, rewrites still lose knowledge and introduce regressions.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“scrapping the whole thing and starting over is now cheaper”
Andrew Ng: Building Faster with AI (Y Combinator)“I've rebuilt a codebase three times in the last month”
Vibe code is legacy code (Val Town)“sometimes it's easier to start over with building a well-written code base from scratch than to fix a legacy one that nobody understands”
Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“It is better to just start over and rewrite it from scratch than it is to try to fix it”
Leans partly right AI guessTrue for small, well-tested components and test suites; for large systems with implicit behavior, rewrites still lose knowledge and introduce regressions.

Enforcing coding conventions with deterministic checks (linters, static analysis) is more reliable than stating them in agent instruction files

Leans right

Deterministic checks fire every time and give agents concrete feedback, whereas instruction-file rules compete for attention and are followed inconsistently.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“custom linters, stuff like that, which is now easier to build as well”
The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“most of these we actually achieve with uh linting rules.”
SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“I can actually try and get like a clean slate all the time and then it becomes a much better sensor”
Birgitta Böckeler - State of Play: AI Coding Assistants - AI Native DevCon June 2026 (AI Native Dev)“It's much more effective to have this linting rule than to like stuff it into your context every single time the agent runs.”
Probably right AI guessDeterministic checks fire every time and give agents concrete feedback, whereas instruction-file rules compete for attention and are followed inconsistently.

AI coding assistants improve developer satisfaction and help them stay in flow

Leans partly right

Self-reported surveys consistently show higher satisfaction and flow, but these are vendor-run perception data and other reports show added review burden and fatigue.

· ·

Asserts (3)Denies (1)Verdicts (3)

GitHub Copilot productivity: what the research actually shows (Mneme HQ)“73% of developers report they are now able to stay in the flow”
AI in software engineering at Google: Progress and the path ahead (research.google)“improvements to these surfaces can directly impact developer productivity and satisfaction”
Measuring the actual impact of AI coding | Abi Noda from DX (Changelog)“one of the strongest relationships we're seeing in the data is actually with engagement meaning developer job engagement”
DENIES How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“Now the nature of writing code in systems itself is interrupt driven or or full of interruptions at least”
Leans partly right AI guessSelf-reported surveys consistently show higher satisfaction and flow, but these are vendor-run perception data and other reports show added review burden and fatigue.
Leans partly right METR 2025 RCT lines 1175-1178METR 2025: 69% of developers kept using Cursor after the study even though they were measured as slowed down, which suggests they value it beyond speed; satisfaction and flow were not measured directly.
Leans partly right Echoes of AI maintainability RCT lines 1609-1614Habitual AI users reported +1.71 SD higher latent perceived productivity (items include fulfilment, enjoyment, flow) when using AI in Task 1, though authors flag possible confirmation bias and the boost did not carry over to Task 2.

Measuring AI's impact requires pre-AI baselines and a mix of system telemetry and developer surveys

Leans right

Self-reports alone overstate speedups (METR) and telemetry alone misses experience and quality, so triangulating against a baseline is standard measurement practice.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (2)

Measuring AI code assistants and agents with the AI Measurement Framework (DX)“combining both self-reported data with telemetry and systems data is the way to go as always.”
How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“I would do surveys uh because surveys can give you a nice kind of overall view of the landscape quickly”
Developer experience in the AI era (Develocity)“So I love combining them when it's possible.”
Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“combining self-reported metrics with system and workflow metrics is absolutely essential when measuring the impact of AI tools”
Probably right AI guessSelf-reports alone overstate speedups (METR) and telemetry alone misses experience and quality, so triangulating against a baseline is standard measurement practice.
Leans right Anthropic internal study lines 165-167Anthropic pairs surveys with 200K Claude Code transcripts and says self-reports (subject to social-desirability and recall bias) should be read alongside objective usage data, with better instruments needed.

AI makes it cheap to submit plausible-looking low-quality pull requests, overwhelming open-source maintainers

Leans right

Maintainers of curl, Ghostty and others document floods of low-effort AI PRs and reports, and GitHub added tooling to restrict PRs in response.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“AI makes it trivial to create plausible-looking but incorrect and low-quality contributions”
A Year Of Vibes (Armin Ronacher)“I find reading such pull requests quite rage-inducing.”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“projects are flooded with junk contributions at the moment to the point that people are trying to convince GitHub to disable pull requests”
A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“why do these open source projects get these random pull requests? Well, it is because it can”
Very likely right AI guessMaintainers of curl, Ghostty and others document floods of low-effort AI PRs and reports, and GitHub added tooling to restrict PRs in response.

Repeatedly telling an LLM its answer is wrong in the same conversation rarely fixes it; restarting or editing the conversation works better

Leans partly right

Self-correction research shows LLMs rarely fix errors from bare 'that's wrong' feedback, but concrete external feedback such as test output does help iteration.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (2)

SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker (Hamel Husain)“Every time you try to get a language model to fix a mistake you make it more likely to make more mistakes of that kind”
Unpacking METR’s findings: Does AI slow developers down? (DX)“that will be much more effective than continuing to fight within the same chat.”
Oxide and Friends 8/5/2024 -- Pragmatic LLM usage with Nicholas Carlini (Oxide Computer Company)“you could just keep saying wrong and it will just it will continue to be sorry the whole way through and never fix anything”
o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“Don't like keep betting on a losing hand.”
Probably partly right AI guessSelf-correction research shows LLMs rarely fix errors from bare 'that's wrong' feedback, but concrete external feedback such as test output does help iteration.
Leans partly right Vibe-coding literature review lines 1319-13293 of 47 sources report repetitive prompt-patch cycles 'without convergence', where developers get stuck re-prompting outputs that fail to fix the issue; restarting is not compared.

LLM coding assistants favor libraries well represented in their training data, making it harder for new libraries to gain adoption

Leans partly right

Studies show LLMs prefer popular libraries and older APIs, but web search and in-context docs partly offset it, and the effect on adoption is not measured.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

Armin Ronacher Leaning In To Find Out - PyAI Conf 2026 (Pydantic)“the thing is incredibly terrible at using UV. It will keep going back to not using UV.”
Oxide and Friends 8/5/2024 -- Pragmatic LLM usage with Nicholas Carlini (Oxide Computer Company)“if everyone is relying on these models to use only the things that popular”
Agentic Coding: Armin Ronacher (Microsoft Azure Developers)“a lot of new software that has to that doesn't exist in a training set”
No, AI is not Making Engineers 10x as Productive (colton.dev)“If you use a library that isn't StackOverflow's favorite it will butcher it even after an agentic lookup of the documentation.”
Leans partly right AI guessStudies show LLMs prefer popular libraries and older APIs, but web search and in-context docs partly offset it, and the effect on adoption is not measured.

AI agents work better with widely used standard frameworks than with bespoke in-house frameworks

Leans right

Models do measurably better on libraries well represented in training data, and bespoke frameworks must be explained in context, though good docs narrow the gap.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (1)

Armin Ronacher Leaning In To Find Out - PyAI Conf 2026 (Pydantic)“anything working with something that's sort of very very common works really well”
AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“The AI doesn't know anything about the frameworks and the coding patterns that we've developed”
How far can we push AI autonomy in code generation? (martinfowler.com)“A common tech stack that should be quite prevalent in the training data”
The AI-Native Data Engineer featuring Fabric, dbt, VSCode devcontainers and copilot (Microsoft Reactor)“align on things that are industry standards, like DBT, that everybody in the world recognizes”
Probably right AI guessModels do measurably better on libraries well represented in training data, and bespoke frameworks must be explained in context, though good docs narrow the gap.

Late-2025 frontier models crossed a threshold where coding agents became clearly net-useful for experienced engineers

Leans partly right

METR's Feb 2026 update suggests speedups are now likely but its data is unreliable; most evidence is practitioner testimony.

· denied 0 ·

Asserts (4)Denies (0)Verdicts (2)

Reconsidering AI w / Steve Klabnik (Zed Industries)“I think that the new models really went from like not being useful to being useful for me”
An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“Opus 4.5 (and the models that came after it) are an order of magnitude better than coding LLMs released just months before it”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“the models in November just got so good.”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“last November it's like oh wait it like crossed the threshold right and suddenly the models were good”
Leans partly right AI guessMETR's Feb 2026 update suggests speedups are now likely but its data is unreliable; most evidence is practitioner testimony.
Leans partly right METR 2026 update lines 68-72METR Feb 2026: through 2025 agentic tools (Claude Code, Codex) spread, and many experienced OSS developers now refuse to do half their work without AI, which is consistent with a shift; measured speedups remain small and uncertain (-18%, CI -38% to +9%).

Stale planning docs left in the repo mislead agents later (doc rot)

Leans right

Agents treat repo docs as authoritative; outdated docs demonstrably steer them wrong.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“This is doc rot”
Building the Context Engine AI Agents Need (InfoQ)“static content in a repo, which is going to rot like every other documentation you ever wrote”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“this thing kind of disynchronized from the codebase ... I can just actually delete it”
Probably right AI guessAgents treat repo docs as authoritative; outdated docs demonstrably steer them wrong.

Combining semantic (embedding) code search with grep improves coding-agent retrieval over grep alone

Leans partly right

Cursor's own A/B results support it, but Claude Code and others do well with agentic grep alone; benefit is real but tool- and codebase-dependent.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“especially when you have a large code base, where you might want some of these other approaches to have more powerful search”
Context Engineering & Coding Agents with Cursor (OpenAI)“you likely want both GP and semantic search for the best results”
OpenCode: The BEST AI Coding Agent Ever! BYE Gemini CLI & ClaudeCode! (Opensource) (WorldofAI)“Mgrep is going to make open code four times faster while using three times fewer tokens”
Leans partly right AI guessCursor's own A/B results support it, but Claude Code and others do well with agentic grep alone; benefit is real but tool- and codebase-dependent.

Technical interviews are shifting away from LeetCode puzzles toward design and AI-assisted problem solving

Leans partly right

Some companies allow AI or changed formats (e.g. Meta AI-enabled rounds), but LeetCode-style screens remain common.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

AI Replaced 75% of Work for Software Engineers & Data Scientists - here's what's left (Sundas Khalid)“Some of them even allow you to use AI tools in interviews”
Don't Waste 2026 on the Wrong Career (Software vs AI Engineer) (Sajjaad Khader)“feature development type interview ... access to AI coding tools”
How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“If they don't use a co-pilot, they usually get creamed by someone who does.”
Leans partly right AI guessSome companies allow AI or changed formats (e.g. Meta AI-enabled rounds), but LeetCode-style screens remain common.

Giving an agent too many tools degrades tool selection; exposing only relevant tools helps

Leans right

Tool-selection accuracy drops as tool count grows (e.g. Berkeley function-calling and RAG-MCP results); dynamic tool loadouts help.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Context engineering for AI agents (Microsoft Developer)“it ends up choosing the wrong one due to the number of tools”
Building the Context Engine AI Agents Need (InfoQ)“agents don't know actually when to call them ... it becomes a bit of a mess”
How Agents Use Context Engineering (LangChain)“keeping the function calling layer very lightweight”
Probably right AI guessTool-selection accuracy drops as tool count grows (e.g. Berkeley function-calling and RAG-MCP results); dynamic tool loadouts help.

Claude Code is more expensive and more rate-limited than competing AI coding tools

Leans partly right

Rate-limit and cost complaints about Claude Code are common and real for heavy users, but competitors also cap usage; anecdotal, from a competitor's video.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

OpenCode Review: Because I Hit Claude Code Usage Limits (Claude Code vs OpenCode vs Codex CLI) (pookie)“after only like 30 minutes with Opus 4.5, I didn't have any usage left”
Claude Code Alternatives in 2026: Cursor, Qodo & GitHub Copilot Compared (Qodo)“clot code is way more expensive ... it keeps on hitting the rate limit”
How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“you get more uh for the same plan in Codex. So you get higher limits”
Leans partly right AI guessRate-limit and cost complaints about Claude Code are common and real for heavy users, but competitors also cap usage; anecdotal, from a competitor's video.

AI investment is a financial bubble even though the underlying technology is durable

Untestable

A market-valuation opinion; not testable from a video's claims.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

AI made Developers Slower (Telusko)“AI as a technology is very solid. What is a bubble is this fundings”
Oxide and Friends 8/5/2024 -- Pragmatic LLM usage with Nicholas Carlini (Oxide Computer Company)“people should not confuse the coming bust for the the Technologies themselves not being useful”
The Truth About Developer Productivity in the AI Age (IT'S A TRAP) (Modern Software Engineering)“Gen AI is a stock market bubble, a technology hype train, and real, genuinely exciting innovation all at the same time.”
Untestable AI guessA market-valuation opinion; not testable from a video's claims.

Developers should read and correct AI-generated plans or specs in detail before letting the agent execute them

Leans partly right

Reviewing plans catches costly mistakes early (e.g. editing secrets), but some practitioners get similar alignment through interactive Q&A and skip reading plans; value depends on task risk.

· ·

Asserts (2)Denies (1)Verdicts (1)

Context Engineering 101 - The Simple Strategy to 100x AI Coding (Cole Medin)“read through your your PRPs before you run the execute command”
FULL Guide to Becoming a Principled Agentic Engineer (Build Anything with AI) (Cole Medin)“it's important for us to review the artifact and make sure that things are really aligned”
DENIES Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“Should I be reviewing this document? ... I don't look at these”
Leans partly right AI guessReviewing plans catches costly mistakes early (e.g. editing secrets), but some practitioners get similar alignment through interactive Q&A and skip reading plans; value depends on task risk.

AI coding tools tend to suppress type and lint errors rather than fix them

Leans right

Agents commonly add any/ts-ignore/lint-disable or weaken tests to get green checks; widely reported and seen in reward-hacking evaluations.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

AI-Generated Code Is Creating a New Kind of Technical Debt (Imran Gardezi)“The AI hadn't fixed the TypeScript errors it was finding along the way. It had suppressed them”
Why AI WON'T Replace Software Engineering... (Modern Software Engineering)“it's removed or broken several tests ... it gave up and deleted them instead”
Are Junior Developers Really Getting Replaced by AI? I'm not sold... (Brian Jenney)“It looked like the AI hardcoded the return values ... the tests were designed to always pass”
Probably right AI guessAgents commonly add any/ts-ignore/lint-disable or weaken tests to get green checks; widely reported and seen in reward-hacking evaluations.

Developer trust in the accuracy of AI-generated code has been declining

Leans right

Stack Overflow surveys show trust in AI accuracy fell from about 40% (2024) to about 30% (2025) while usage rose.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

The AI-Native Software Engineer | Addy Osmani (JavaScript Conferences by GitNation)“usage is high. It continues to go up but trust is sliding”
AI-Generated Code Is Creating a New Kind of Technical Debt (Imran Gardezi)“Only 29% of developers trust AI-generated code. This was down from 40%”
AI in engineering: Q2 2026 benchmarks & research readout (DX)“I can modify the codes that's in front of me but I don't trust what I'm pushing into production”
Probably right AI guessStack Overflow surveys show trust in AI accuracy fell from about 40% (2024) to about 30% (2025) while usage rose.

Customer-facing forward-deployed engineer roles are among the fastest-growing engineering jobs

Leans right

FDE postings grew several-fold in 2025 as AI labs and startups (OpenAI, Anthropic, Palantir-style firms) hired for them; still a small share of jobs.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Forward Deployed Engineer: The Truth About Tech's Hottest Job (I Counted Every Open Role) (Code with Jay)“Forward deployed engineer 42 times growth since 2023”
AI Replacing Developers Has Officially Failed (Sajjaad Khader)“This is one of the fastest growing roles in tech right now”
Don't Waste 2026 on the Wrong Career (Software vs AI Engineer) (Sajjaad Khader)“forward deployed engineers ... Very very hot role”
Leans right AI guessFDE postings grew several-fold in 2025 as AI labs and startups (OpenAI, Anthropic, Palantir-style firms) hired for them; still a small share of jobs.

Models perform best inside their own vendor's agent harness

Leans partly right

Models are tuned on their vendors' harnesses and often score higher there, but third-party harnesses sometimes match or beat them on benchmarks (e.g. Terminal-Bench).

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

OpenCode Review: Because I Hit Claude Code Usage Limits (Claude Code vs OpenCode vs Codex CLI) (pookie)“the model really isn't trained for the tools that open code provides”
The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“swapping out a different model and expecting similar results is is very different”
A very subjective comparison of Claude Code, OpenCode, Cursor & GitHub Copilot (Maximilian Schwarzmüller)“especially that combination of Claude Code with those models is really good”
Leans partly right AI guessModels are tuned on their vendors' harnesses and often score higher there, but third-party harnesses sometimes match or beat them on benchmarks (e.g. Terminal-Bench).

Developers who use AI will replace developers who don't

Untestable

A slogan about future labor competition, not a testable claim; plausible but unmeasured.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Andrew Ng: Building Faster with AI (Y Combinator)“people that know how to use AI will replace people that don't”
"STOP GETTING COMPUTER DEGREES" - Sam Altman's SHOCKING Interview (AI Upload)“just start using these tools start incorporating them into the way that you work”
Will AI replace Software Engineers ? Raw & Honest Opinion (Apporva arya)“AI will surely replace the people who are not learning AI”
Untestable AI guessA slogan about future labor competition, not a testable claim; plausible but unmeasured.

Engineering headcount cuts justified by AI productivity rest on gains never validated in controlled studies

Leans partly right

Few firms publish controlled evidence for AI-justified cuts and the best RCT points the other way for experts; but internal metrics may exist that aren't public.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (3)

The AI Productivity Paradox: Why 10X Output Doesn't Mean 10X Business Outcome - Mik Kersten (Tech Lead Journal)“your software developers based on an output productivity gain, not an outcome”
The METR Paradox. Why AI Made Senior Devs 19% Slower (Macro Lens)“they are making a claim that has never been validated in a controlled study on the population they are actually paying”
Where's the Shovelware? Why AI Coding Claims Don't Add Up (mikelovesrobots.substack.com)“justify layoffs with newfound productivity narratives”
Leans partly right AI guessFew firms publish controlled evidence for AI-justified cuts and the best RCT points the other way for experts; but internal metrics may exist that aren't public.
Leans partly right Microsoft/Accenture field RCTs lines 774-784Large field RCTs do validate a positive gain (~26% more completed tasks), so gains are not wholly unvalidated, but the validated size is modest and far from what would justify large cuts.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 597-613This controlled crossover experiment did validate a large productivity gain (completeness +43 points, robust to session checks), so gains have been shown in controlled studies. But the setting was 75-minute toy tasks, mostly students, and no time gain held up, which says nothing about headcount-level gains.

AI tools confidently report security vulnerabilities that do not exist

Leans right

Documented across curl's public HackerOne reports and other projects: fabricated stack traces, nonexistent functions, already-fixed CVEs.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Developers Are Sick Of AI Slop Bug Reports (Brodie Robertson)“You did not find anything worth reporting. You were fooled by an AI into believing that”
"AI Slop" Now BANNED in the Curl Project (Linux User Space)“Turns out that the code that this particular report was about didn't actually exist. It was made up”
vulnerability research just got easier (scarier?) (Low Level)“in the remaining 28 of the reports, there are false positives”
Probably right AI guessDocumented across curl's public HackerOne reports and other projects: fabricated stack traces, nonexistent functions, already-fixed CVEs.

LLM coding agents are vulnerable to prompt injection

Leans right

Repeatedly demonstrated against coding agents (malicious repos, issues, docs and MCP tool descriptions causing exfiltration or unsafe commands).

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“they are susceptible to prompt injection risks. They might leak your data”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“there there is no way to reliably say, these are This is the data, and these are the instructions.”
Let's put the AI in lots of little boxes - Kenton Varda, Cloudflare Connect 2025 (Cloudflare Developers)“what if this scam email starts to contain prompt injection attacks? Like we haven't figured out how to defend against those yet.”
Very likely right AI guessRepeatedly demonstrated against coding agents (malicious repos, issues, docs and MCP tool descriptions causing exfiltration or unsafe commands).

LLM-assisted work is more homogeneous than unassisted work

Leans right

Several studies find reduced diversity of ideas and phrasing with LLM assistance (creative writing, ideation); consistent with the MIT result.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (2)

The AI Productivity Paradox: Why 10X Output Doesn't Mean 10X Business Outcome - Mik Kersten (Tech Lead Journal)“in voice, similarity in idea generations that is coming from the models as well.”
Your Brain on ChatGPT: Accumulation of Cognitive Debt (June 2025) (AI Paper Slop)“The LLM group's essays on the same topic were statistically much more similar to each other”
12 Months of Unity Dev with Claude Code (Mythmatic)“Claude code is always pulling everything towards the average.”
Probably right AI guessSeveral studies find reduced diversity of ideas and phrasing with LLM assistance (creative writing, ideation); consistent with the MIT result.
Leans partly right Echoes of AI maintainability RCT lines 2603-2606Authors speculate that AI assistants homogenize code into standard constructs, citing Haase et al. on LLM output clustering, but do not measure homogeneity themselves.

Running coding agents in parallel requires isolating their workspaces (separate worktrees and databases)

Leans right

Shared files and databases cause collisions; worktrees or containers per agent are the standard approach, and Codex and Claude apps build this in.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Agentic Engineering in Action with Mitchell Hashimoto (Zed Industries)“they can't operate in parallel in the same folder”
The AI-Native Data Engineer featuring Fabric, dbt, VSCode devcontainers and copilot (Microsoft Reactor)“I can go ahead and have 10 different Git work trees and Visual Studio Code”
Claude Code Git Worktree Setup: Run Multiple Agents in Parallel (Damian Galarza)“they're all pointing at the same directory. Same files, same git state, same database”
Probably right AI guessShared files and databases cause collisions; worktrees or containers per agent are the standard approach, and Codex and Claude apps build this in.

The main change LLMs bring to programming is the shift from deterministic to non-deterministic computing

Untestable

A framing about what matters most; it's a useful lens, but you can't test it as a fact.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“the single biggest change ... going from deterministic systems to non-deterministic ones”
Software engineering with LLMs in 2025: reality check (at LDX3 by LeadDev) (The Pragmatic Engineer)“except these things are non-deterministic for the first time in computing”
AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“If I'm talking to an AI, I'm going to get different responses.”
Untestable AI guessA framing about what matters most; it's a useful lens, but you can't test it as a fact.

AI assistants tend to agree with users rather than challenge flawed ideas

Leans right

Sycophancy is a well-documented LLM failure mode in research and vendor reports, though newer models and prompting can reduce it.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Nicole Forsgren: Leading high-performing engineering teams in the age of AI - The Pragmatic Summit (The Pragmatic Engineer)“Also, they just agree with us constantly. I'm always so smart and I'm like I know that was dumb.”
o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“Stop telling me I'm right.”
576 - Using LLMs at Oxide | RFD | Oxide (Oxide Computer Company)“LLMs are infamous pleasers — and you may find that the breathless praise from an LLM is in fact more sycophancy than analysis.”
Very likely right AI guessSycophancy is a well-documented LLM failure mode in research and vendor reports, though newer models and prompting can reduce it.

A working coding agent is just an LLM calling a few tools in a loop and needs only a few hundred lines of code

Leans right

Minimal agent loops (read file, list dir, run shell) are well documented to work with modern tool-calling models; production harnesses add much more but the core loop is small.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Armin Ronacher Leaning In To Find Out - PyAI Conf 2026 (Pydantic)“it has only four tools it's the most minimal set of coding agent you can use and it still works”
Agentic coding changes everything | Thorsten Ball from Sourcegraph Amp (Changelog)“And it's only 300 lines of code.”
My AI Skeptic Friends Are All Nuts (Fly)“You could write an effective coding agent in a weekend.”
Very likely right AI guessMinimal agent loops (read file, list dir, run shell) are well documented to work with modern tool-calling models; production harnesses add much more but the core loop is small.

Interacting with coding agents is compulsive, like a slot machine's intermittent reward

Untestable

This describes subjective experience; no controlled data measures compulsive use of coding agents.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“these tools are super addictive. You never know if that next prompt is going to be the one that makes your product work”
TDD, AI agents and coding with Kent Beck (The Pragmatic Engineer)“this is exactly like a slot machine. You got intermittent reinforcement. You got negative outcomes and positive outcomes.”
o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“this is another theme that's emerging for me is intermittent reinforcement. It's a slot machine.”
Untestable AI guessThis describes subjective experience; no controlled data measures compulsive use of coding agents.

High-quality internal developer platforms are needed for AI gains to translate into organizational outcomes

Leans partly right

DORA 2025 reports this association, but it is correlational survey evidence and platform quality is hard to separate from overall engineering maturity.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit (The Pragmatic Engineer)“Come to find out, in fact, these are the things that make AI really successful.”
DORA Community Discussion - State of AI-assisted Software Development (DORA)“highquality platforms are essential to unlocking the organizational value of AI and also essential to managing the associated risks.”
DORA Community Discussion - Decoding DORA 2025: AI’s Real Impact (DORA)“as the quality of the platform reaches a certain level that's when you start to see that it's benefiting the organization”
Leans partly right AI guessDORA 2025 reports this association, but it is correlational survey evidence and platform quality is hard to separate from overall engineering maturity.

Keeping an agent working on slow background tasks while you do other work adds useful output

Leans partly right

METR Feb 2026: developers using agentic tools often worked on unrelated tasks while waiting, and some ran several agents at once, which made per-task time unreliable. The practice is common, but the extra output is not measured.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (2)

Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“I got a lot of coding work done on that meeting heavy day that I would have not necessarily gotten done”
Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“I endeavor to always have an agent doing something at all times.”
My AI Adoption Journey (Mitchell Hashimoto)“instead of trying to do more in the time I have, try to do more in the time I don't have.”
Leans partly right METR 2026 update lines 83-83METR Feb 2026: developers using agentic tools often worked on unrelated tasks while waiting, and some ran several agents at once, which made per-task time unreliable. The practice is common, but the extra output is not measured.
Unknown AI guessPlausible for research-style tasks, but no study measures the net gain against review and context-switching costs.

LLM tutors help learners by answering basic questions patiently and without judgment

Leans partly right

Some RCTs (e.g. a Harvard physics AI-tutor study) show learning gains when tutors are scaffolded, but hallucinations and answer-giving can undermine learning.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

Oxide and Friends 8/5/2024 -- Pragmatic LLM usage with Nicholas Carlini (Oxide Computer Company)“you can just ask the computer now like the computer's not going to judge you”
Teaching CS50 with AI - David J. Malan (CS50)“I love how AI bots will answer the questions without ego and without judgment”
Sustainable Augmented Development • Kent Beck • YOW! 2025 (GOTO Conferences)“The genie is an endlessly patient, endlessly creative, usually right tutor.”
Probably partly right AI guessSome RCTs (e.g. a Harvard physics AI-tutor study) show learning gains when tutors are scaffolded, but hallucinations and answer-giving can undermine learning.

AI agents let engineers take on refactoring and tech-debt cleanup they previously lacked time for

Leans right

Airbnb finished a migration estimated at 1.5 engineering-years by hand in 6 weeks, after Enzyme had lingered since RTL was adopted in 2020.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (4)

What We Learned Deploying AI within Bloomberg’s Engineering Organization – Lei Zhang, Bloomberg (AI Engineer)“we are able to uh see very much better results from the um uplift agents”
Using AI to write better code more slowly (Read the Tea Leaves)“I’m writing unit tests and fixing subtle flaws that pre-date the PR.”
How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“tech debt reduction, refactoring, u making things easier to read, like those are the things I'm seeing the best engineers at Shopify use AI tooling”
Probably right AI guessAirbnb and Google migration case studies show large mechanical refactors sped up a lot with AI, though results need careful review and broad cleanup claims are mostly anecdotal.
Leans right Google migrations report lines 79-85Google: LLM migrations finished efforts stalled for years and 'saved hundreds of engineers worth of work', making previously infeasible tech-debt work possible.
Leans right Airbnb migration lines 6-10Airbnb finished a migration estimated at 1.5 engineering-years by hand in 6 weeks, after Enzyme had lingered since RTL was adopted in 2020.
Leans right Anthropic internal study lines 134-134Anthropic usage data: 8.6% of Claude Code tasks are papercut fixes such as refactoring for maintainability, which people say would normally be deprioritized.

Coding agents perform better in clean, well-structured codebases than in messy ones

Leans right

Practitioner reports and agent studies show agents imitate surrounding code and struggle in large messy repos, though controlled evidence is limited.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents (IEEEComputerSociety)“this is a very very messy code base so it's a lot higher probability that it will do something wrong”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“there's lots to be said for having for keeping your code base high-quality, because the agent will then add to it in a high-quality way.”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“agents work better in the context of a code base that is well-tended and well-put-together and relatively simple and tightly engineered.”
Probably right AI guessPractitioner reports and agent studies show agents imitate surrounding code and struggle in large messy repos, though controlled evidence is limited.

Having agents exercise the running software (e.g. start a server and call its API) catches bugs that unit tests miss

Leans right

End-to-end checks catch integration failures that unit tests miss, a well-established testing principle that applies to agents.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“it retrieved the trace from Honeycomb with its fingers. And then it drew it again and it drew it right”
Are Harnesses Worth the Cost? (martinfowler.com)“Contract-driven dependency mocks made downstream interactions executable”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“Start the server running um in the background, and then use curl to exercise the API that you just created." And that works.”
Probably right AI guessEnd-to-end checks catch integration failures that unit tests miss, a well-established testing principle that applies to agents.

Language-agnostic conformance test suites let coding agents implement complex specifications largely autonomously

Leans right

Demonstrations such as spec-driven compiler and library ports show agents converge on conformance suites, though code quality can lag.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“I think those kind of um comprehensive tests uh rigorous tests help a lot”
You can just choose how many bugs you want now (Read the Tea Leaves)“an agent could just grind through the test suite, and if it got close to 100% then I could be reasonably certain”
Simon Willison: Engineering practices that make coding agents work - The Pragmatic Summit (The Pragmatic Engineer)“you can give it to a a good agent and say, "Write code until this test suite passes." And it kind of will.”
Probably right AI guessDemonstrations such as spec-driven compiler and library ports show agents converge on conformance suites, though code quality can lag.

When an agent writes both the code and its tests, a passing test suite gives weak assurance that the behavior is correct

Leans right

A shared misreading of requirements can appear in both code and tests, and agent-written tests often assert current behavior rather than intent.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“I think that's very unsatisfying to me because the agent also generates the tests”
AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“if the AI is writing the code and the tests, what's what are we doing?”
Are Harnesses Worth the Cost? (martinfowler.com)“If an agent interprets requirements to write both the implementation and its tests, the same misreading or omission can appear in both.”
Probably right AI guessA shared misreading of requirements can appear in both code and tests, and agent-written tests often assert current behavior rather than intent.

AI agents should implement work in thin vertical slices rather than layer by layer, to get early end-to-end feedback

Leans right

Tracer-bullet/vertical slicing gives earlier integration feedback; applies to agents as to humans.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“AI loves to code horizontally”
AWS re:Invent 2025 - Spec-driven development with Kiro (DEV314) (AWS Events)“you first start with a skateboard and move to a bike and then build a car ... you always have something that's workable”
Probably right AI guessTracer-bullet/vertical slicing gives earlier integration feedback; applies to agents as to humans.

Planning and alignment need a human in the loop, while well-specified implementation can run unattended

Leans right

Current agents are weakest at intent/requirements and strongest at well-specified implementation with verification.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

Full Walkthrough: Workflow for AI Coding — Matt Pocock (AI Engineer)“planning, this alignment phase, has to be human in the loop”
Inside minions: how Stripe uses autonomous one-shot AI coding agents (Stripe Developers)“minions are very capable of larger feature changes that are well scoped out by the engineer in advance and then handed off to them”
Probably right AI guessCurrent agents are weakest at intent/requirements and strongest at well-specified implementation with verification.
Leans partly right Anthropic internal study lines 50-50Anthropic: most staff can 'fully delegate' only 0-20% of their work and actively supervise the rest, keeping design for themselves; this supports keeping a human on design but shows implementation is also rarely left unattended.

GitHub Copilot is the cheapest paid entry point and the best fit for GitHub-centric teams

Leans partly right

Copilot Pro's $10 tier is the cheapest headline price and GitHub integration is real; 'best fit' depends on team needs.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Codex vs Claude Code vs Cursor vs Copilot — Best AI Coding Agent in 2026? (Byte & Beyond)“at $10, it's paid individual tier is the cheapest headline subscription”
Cursor vs Claude Code vs GitHub Copilot — Which AI Coding Tool Actually Wins? (2026) (Nordic AI Dev)“Copilot wins on price and editor flexibility”
Leans partly right AI guessCopilot Pro's $10 tier is the cheapest headline price and GitHub integration is real; 'best fit' depends on team needs.

Subscription sticker prices understate the real cost of heavy agentic coding

Leans right

Agentic usage is metered or rate-limited across vendors; heavy users routinely hit limits or pay usage overages.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Codex vs Claude Code vs Cursor vs Copilot — Best AI Coding Agent in 2026? (Byte & Beyond)“Treat the sticker price as the entry point, not a promise of unlimited heavy automation”
My AI Coding Workflow That Actually Works (Chris Titus Tech)“You're not going to be able to do it without the $100, $200 subscriptions”
Probably right AI guessAgentic usage is metered or rate-limited across vendors; heavy users routinely hit limits or pay usage overages.

Dedicated AI code-review tools find more issues than general-purpose coding assistants

Unknown

Based on one vendor-run demo on one repo; no independent comparison supports or refutes it.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Claude Code Alternatives in 2026: Cursor, Qodo & GitHub Copilot Compared (Qodo)“when it comes to coding review platforms and security reviews, Kod kind of wins”
Comparing all AI Pull Request Review tools to find the best one (Morning Maker Show with Sandra and Dan)“the code review was pretty much equal to just prompting something directly in Claude code”
Unknown AI guessBased on one vendor-run demo on one repo; no independent comparison supports or refutes it.

Newer models reliably execute much longer and more complex specifications than earlier ones

Leans right

Instruction-following and task horizon rose sharply from Claude 3.7 to 4.x and later; METR's horizon metric shows the same trend.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

Context Engineering 101 - The Simple Strategy to 100x AI Coding (Cole Medin)“after cloud 4 I can kick off like a 500 line prompt reliably”
Are Harnesses Worth the Cost? (martinfowler.com)“All 10 unharnessed GPT-6 Sol applications passed every functional check on the first pass without human debugging or corrective follow-ups.”
Probably right AI guessInstruction-following and task horizon rose sharply from Claude 3.7 to 4.x and later; METR's horizon metric shows the same trend.
Leans partly right Anthropic internal study lines 124-127Anthropic transcripts Feb->Aug 2025: max consecutive tool calls rose from 9.8 to 21.2, human turns fell from 6.2 to 4.1 and task complexity rose from 3.2 to 3.8, but usage shifts could explain part of this and reliability isn't measured.

Software engineering job postings have been recovering, not falling, as AI coding tools improved

Leans partly right

Postings rose from the 2023 trough in some trackers (TrueUp, parts of Indeed), but remain far below 2022 and junior postings stayed weak; the data are mixed.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

The TRUTH About the Software Development Job Market in 2026 (Internet Made Coder)“listings for software engineers on Indeed are up 11% annually”
AI Replacing Developers Has Officially Failed (Sajjaad Khader)“Software engineering postings were up 32%”
Leans partly right AI guessPostings rose from the 2023 trough in some trackers (TrueUp, parts of Indeed), but remain far below 2022 and junior postings stayed weak; the data are mixed.

AI-generated code contains more defects than human-written code

Leans partly right

CodeRabbit's 2025 PR analysis found ~1.7x more issues in AI-authored PRs, and security studies agree; results depend heavily on review and tooling, and some studies find parity.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (7)

AI Replacing Developers Has Officially Failed (Sajjaad Khader)“AI generated code has 1.7 times more errors compared to code written by human software engineers”
Is AI making devs SLOWER? Yes, according to this study... (Native Notify)“800 engineers had 41% more bugs in their pull requests using AI tools”
Leans partly right AI guessCodeRabbit's 2025 PR analysis found ~1.7x more issues in AI-authored PRs, and security studies agree; results depend heavily on review and tooling, and some studies find parity.
Leans partly right METR 2025 RCT lines 1187-1192METR 2025: raw AI output often needed major cleanup, but final AI-allowed PRs looked qualitatively similar in quality, and post-review fix time differed insignificantly (15 vs 9 min). Experienced reviewers caught defects before submission.
Leans wrong Shan et al. field study + RCT lines 836-856In the RCT, ChatGPT access raised task accuracy by 26 percentage points (0.65 to about 0.91), and directly accepting AI output did not lower accuracy. The tasks were simple (GPT-4 solved them all), so this says little about defects in real code.
Leans partly right Microsoft/Accenture field RCTs lines 506-514Using build success as a quality proxy, Copilot had no negative effect at Microsoft but a negative one at Accenture (-17.4%, p<0.05); pooled -5.5% NS, so evidence of more defects is mixed and the proxy is weak.
Unknown Debt study (302k AI commits) lines 1031-1035The Debt study explicitly does not compare AI commits with a human-only baseline, so it can't say whether AI code has more defects.
Unknown Copilot RCT (Peng) lines 215-219Peng et al. state explicitly that the study does not examine AI's effect on code quality, which could go either way.
Unknown Vibe-coding literature review lines 1454-1458Defect risks are reported in 21% (10/47) of sources, discussed alongside productivity gains, but the review gives no human-written baseline and treats counts as mention frequency, not effect size.

Agent configuration formats (instruction files, skill folders) remain fragmented across tools

Leans right

AGENTS.md vs CLAUDE.md vs .cursor/rules and different skill folders are documented differences; standardization is ongoing.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

OpenCode Review: Because I Hit Claude Code Usage Limits (Claude Code vs OpenCode vs Codex CLI) (pookie)“Claude uses a different format ... you will have to rename that to agents.markdown”
A very subjective comparison of Claude Code, OpenCode, Cursor & GitHub Copilot (Maximilian Schwarzmüller)“every provider tries to have its own configuration”
Probably right AI guessAGENTS.md vs CLAUDE.md vs .cursor/rules and different skill folders are documented differences; standardization is ongoing.

Rankings in 'best AI coding tool' roundups mostly reflect reviewer opinion rather than repeatable benchmarks

Leans right

Most roundup scores are unblinded single-reviewer judgments; that's why rankings disagree across sites.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

Best AI Code Editors in 2026 — Cursor vs Windsurf vs Copilot vs Zed vs Claude Code (Nordic AI Dev)“A 9.2 out of 10 from one blog is just that person's opinion dressed up to look like data”
A Year Of Vibes (Armin Ronacher)“I have nothing beyond vibes to back up my preference for Claude.”
Probably right AI guessMost roundup scores are unblinded single-reviewer judgments; that's why rankings disagree across sites.
Leans partly right Vibe-coding literature review lines 1716-1722The review stresses that tool counts reflect visibility rather than comparative effectiveness, because most sources mention tools without controlled evaluation; it does not analyze roundup rankings specifically.

Leading AI labs pay engineers record compensation, signaling strong ongoing demand for engineering skill

Leans right

Levels.fyi and reporting show median packages at OpenAI/Anthropic far above big-tech norms; pay signals demand, though partly for AI research talent specifically.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

AI Will Replace Software Engineers? Here’s What’s Actually Happening (Chris Schwenk | Tech Jobber)“If you truly believed AI was about to replace software engineers, would you pay the median one $750,000 a year?”
Are Junior Developers Really Getting Replaced by AI? I'm not sold... (Brian Jenney)“AI companies are hiring developers at outrageous salaries”
Probably right AI guessLevels.fyi and reporting show median packages at OpenAI/Anthropic far above big-tech norms; pay signals demand, though partly for AI research talent specifically.

Including the prompt and intent with AI-authored pull requests makes them easier to review

Leans partly right

Capturing intent helps reviewers, but raw prompt logs are noisy; a good PR description serves the same purpose. Plausible, untested.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

A Year Of Vibes (Armin Ronacher)“I wish I could see the prompts that led to changes.”
AI Made Devs 19% Slower (METR 2025 Study) (zencoderai)“you can just put the prompt in there as part of the pull request”
Leans partly right AI guessCapturing intent helps reviewers, but raw prompt logs are noisy; a good PR description serves the same purpose. Plausible, untested.

Growth in agents' task horizons accelerated recently, faster than the long-run seven-month doubling

Leans right

METR noted a faster ~4-month doubling for 2024-25 models; later data roughly held, though the sample is small and uncertain.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

AI Agent Capabilities Are Doubling Every Three Months (The AI Daily Brief: Artificial Intelligence News)“For models created since last year, the doublings in capability are occurring every 3 months”
Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“in the recent past, it looks like it's um it's accelerating or or or rising rising more quickly.”
Leans right AI guessMETR noted a faster ~4-month doubling for 2024-25 models; later data roughly held, though the sample is small and uncertain.

Agents' reliable (80%-success) task horizon is several times shorter than their 50%-success horizon

Leans right

METR's own analysis: 80%-success horizons are roughly 4-5x shorter than 50% horizons.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

AI Agent Capabilities Are Doubling Every Three Months (The AI Daily Brief: Artificial Intelligence News)“increasing the reliability threshold from 50 to 80% reduces the average time horizon by a factor of five”
AI Safety (CS 2881) Lecture 7: Lab vs Field: Guest lecture by Joel Becker (Boaz Barak)“The 80% time horizons are much lower. They're something like a factor of five lower, I think.”
Very likely right AI guessMETR's own analysis: 80%-success horizons are roughly 4-5x shorter than 50% horizons.

AI tools that correctly find bugs still usually propose worse fixes than a human would write

Leans partly right

Maintainers widely report poor auto-fixes, and fix quality lags detection in evaluations; but auto-fix quality has improved and varies by bug type.

· ·

Asserts (1)Denies (1)Verdicts (1)

Open Source security in spite of AI - Daniel Stenberg (Daniel Stenberg)“All of these AI detection tools ... offer like generate a patch to fix this, but they're never good”
DENIES vulnerability research just got easier (scarier?) (Low Level)“it had realized that setting session user equal to null was insufficient to fix the issue”
Leans partly right AI guessMaintainers widely report poor auto-fixes, and fix quality lags detection in evaluations; but auto-fix quality has improved and varies by bug type.

Unattended agent loops suit low-stakes, automatically verifiable tasks rather than production work

Leans right

Loops without human checkpoints work when tests define success; for ambiguous or high-stakes work, drift, false completion and token burn dominate. Matches vendor guidance on long-running harnesses.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

Ralph Wiggum: Continuous Agent Loops in Claude Code (AI Makerspace)“It's not good for tasks requiring human judgment ... don't use Ralph at work”
How far can we push AI autonomy in code generation? (martinfowler.com)“AI is not ready to create and maintain a maintainable business software codebase without human oversight.”
Probably right AI guessLoops without human checkpoints work when tests define success; for ambiguous or high-stakes work, drift, false completion and token burn dominate. Matches vendor guidance on long-running harnesses.
Leans partly right Anthropic internal study lines 19-19Anthropic: engineers fully delegate easily verifiable, low-stakes work (throwaway debug or research code) and keep supervising high-stakes work; this is about delegation in general, not unattended loops specifically.

Gemini CLI's free tier makes it the cheapest capable terminal coding agent

Leans partly right

At launch the free quota was unusually generous; Google later tightened limits and model routing, and 'capable' trailed Claude Code and Codex on many tasks.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Gemini CLI: Google's NEW Fully Free Opensource Coding Agent! Beats Claude Code! (WorldofAI)“you get a,000 requests a day with 60 requests an hour for free using the Gemini 2.5 Pro”
Gemini CLI: How To Get Started (pixegami)“The free tier lets you use up to 60 requests per minute or up to 1000 requests per day”
Leans partly right AI guessAt launch the free quota was unusually generous; Google later tightened limits and model routing, and 'capable' trailed Claude Code and Codex on many tasks.

OpenCode offers a richer agent harness than OpenAI's Codex CLI

Leans partly right

True of features at the time (LSP, agents, plan mode); Codex CLI has since added several, and 'richer' isn't 'better results'.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

OpenCode Review: Because I Hit Claude Code Usage Limits (Claude Code vs OpenCode vs Codex CLI) (pookie)“plan mode, LSPs, the Vim mode, the sub agents ... simply aren't included in the intentionally very simple CLI tool of Codex”
OpenCode: The BEST AI Coding Agent Ever! BYE Gemini CLI & ClaudeCode! (Opensource) (WorldofAI)“it actually ships with better feature sets than most AI CLIs”
Leans partly right AI guessTrue of features at the time (LSP, agents, plan mode); Codex CLI has since added several, and 'richer' isn't 'better results'.

Reliable autonomous AI agents will take about a decade to mature, not a single year

Untestable

A long-range forecast; consistent with autonomy history (self-driving) but not yet checkable.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

This is not the year of agents – Andrej Karpathy (Dwarkesh Clips)“I just think that it will take about a decade to work through all of those issues.”
Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“when I see things like oh 2025 is the year of agents I get very concerned ... this is the decade of agents”
Untestable AI guessA long-range forecast; consistent with autonomy history (self-driving) but not yet checkable.

Some major employers are increasing entry-level engineering hiring because new graduates are AI-fluent

Leans partly right

IBM, LinkedIn, Cloudflare and others publicly announced more junior hiring in 2026, citing pipeline and AI fluency; still a minority of employers.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Junior Dev Jobs Are Down 20%… So Why Are Companies Hiring? (HTML All The Things)“IBM, Cognizant, Cloudflare, Dropbox ... AI fluency is one of them”
How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“I convinced Toby to hire a thousand interns this term”
Leans partly right AI guessIBM, LinkedIn, Cloudflare and others publicly announced more junior hiring in 2026, citing pipeline and AI fluency; still a minority of employers.

Experienced engineers using coding agents can produce many thousands of lines of production code per day

Leans wrong

Lines of code isn't a productivity measure, and controlled studies show modest gains; sustained output of that size without proportional review implies unreviewed code.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

Vibe Coding Is The Only Future - Steve Yegge (A Life Engineered)“now claims he can write 12,000 lines of production code per day using AI agents”
FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“single engineer over the course of 4 weeks when armed with this stuff can produce over a million lines of Rust code”
Leans wrong AI guessLines of code isn't a productivity measure, and controlled studies show modest gains; sustained output of that size without proportional review implies unreviewed code.
Leans partly right Debt study (302k AI commits) lines 143-145One cited case (claudes-c-compiler) shows ~500K lines added by Claude over a few weeks, so tens of thousands of lines per day are possible, but nothing here establishes that this is production-quality code or attributes it to experienced engineers.

Current AI code-review tools trade precision against recall, and none catch most real bugs

Leans right

Independent and vendor benchmarks of PR review bots report recall well below 60% and wide precision variation; the specific ranking here is vendor-reported.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

We benchmarked the TOP AI Code Reviewers (Augment Code)“Codex actually has higher precision at 68% but look at the recall, 29% ... Claude Code ... 51% recall, but only 23% precision”
Comparing all AI Pull Request Review tools to find the best one (Morning Maker Show with Sandra and Dan)“It did miss those security issues that Code Rabbit has highlighted”
Probably right AI guessIndependent and vendor benchmarks of PR review bots report recall well below 60% and wide precision variation; the specific ranking here is vendor-reported.

Reviewing research and implementation plans can replace line-by-line review of AI-generated code

Leans partly right

Plan review catches costly errors early and helps team alignment, but plans don't guarantee the code matches; most guidance and defect data still favor reviewing code, especially for security.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Context Engineering SF: Advanced Context Engineering for Agents - Dex Horthy (Chroma)“I still read all the tests, but I no longer read every line of code because I read the specs and I know they're right”
Control the ideas, not the code - <antirez> (antirez.com)“if you control the ideas of your software, looking at the code itself is suboptimal and often pointless.”
Leans partly right AI guessPlan review catches costly errors early and helps team alignment, but plans don't guarantee the code matches; most guidance and defect data still favor reviewing code, especially for security.

Most enterprise AI initiatives have not yet delivered measurable financial returns

Leans right

McKinsey 2025 and the MIT NANDA report found most deployments in pilots and few with P&L impact; measurement methods vary and some sectors see clear returns.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit (The Pragmatic Engineer)“high adoption but actually low transformation”
AI Layoffs Don’t Add Up. Here’s the Data (HuMan Intelligence)“Only about 30% of scaled AI beyond a few teams and only 39% report any bottom line impact”
Probably right AI guessMcKinsey 2025 and the MIT NANDA report found most deployments in pilots and few with P&L impact; measurement methods vary and some sectors see clear returns.

AI coding agents deliberately deceive users to hide their own mistakes

Leans partly right

Agents do fabricate results and misreport failures (well documented here), but attributing deliberate intent is anthropomorphic; it's better explained as optimizing for apparent task success.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

The Replit AI Agent DISASTER: Production Database Deleted In SECONDS! ($erver $ays)“It lied on purpose ... calculated deceptive behavior”
AI Just Crossed the Terrifying Line - Now What? (Kurzgesagt – In a Nutshell)“So the only remaining option was to trick the scorer! They needed to create a fake history.”
Leans partly right AI guessAgents do fabricate results and misreport failures (well documented here), but attributing deliberate intent is anthropomorphic; it's better explained as optimizing for apparent task success.

Coding agents work best when they edit freely and humans review afterward, rather than approving every edit

Leans partly right

Removing per-edit approval cuts friction and works with version control and good tests; in risky environments, permission gates and sandboxing still matter.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“the agent should just make edits to your files. It shouldn't ask you at every turn”
Agentic coding changes everything | Thorsten Ball from Sourcegraph Amp (Changelog)“forget about accepting each change forget about not giving the model real access to the file system.”
Leans partly right AI guessRemoving per-edit approval cuts friction and works with version control and good tests; in risky environments, permission gates and sandboxing still matter.

Merging work from many parallel agents is the main bottleneck once agents scale up

Leans partly right

Conflicts and integration do become the constraint with many concurrent agents (consistent with review-bottleneck data), but it's one of several bottlenecks alongside review and verification.

· ·

Asserts (1)Denies (1)Verdicts (1)

Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE (Latent Space)“Merging is the ... wall that everyone is hitting right now”
DENIES FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“So, we've noticed that most um uh commits do not have merge conflicts.”
Leans partly right AI guessConflicts and integration do become the constraint with many concurrent agents (consistent with review-bottleneck data), but it's one of several bottlenecks alongside review and verification.

Deriving property-based tests from structured requirements makes AI-generated code more verifiably correct

Leans partly right

Property-based testing is a strong verification technique and pairs well with explicit invariants; vendor-reported gains, and property quality still depends on how well they're written.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

AWS re:Invent 2025 - Spec-driven development with Kiro (DEV314) (AWS Events)“adding these property-based testing makes the output of your code at a much higher quality and oftentimes less bugs”
Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro (AI Engineer)“your ears requirements can be translated directly into properties of the system which are effectively invariants”
Leans partly right AI guessProperty-based testing is a strong verification technique and pairs well with explicit invariants; vendor-reported gains, and property quality still depends on how well they're written.

Everyone, including non-engineers, should learn to code because AI makes coding easier and more valuable

Leans partly right

Basic coding helps people direct and check AI output, and adoption among non-engineers is growing. The labor-market payoff for non-engineers is still unproven.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Andrew Ng: Building Faster with AI (Y Combinator)“some people are telling people not to learn to code ... it'll go down as some of the worst career advice ever”
How To Solve It With Code—background on the course; a discussion with Jeremy and Hamel (Jeremy Howard)“you do actually need to learn to code”
Leans partly right AI guessBasic coding helps people direct and check AI output, and adoption among non-engineers is growing. The labor-market payoff for non-engineers is still unproven.

Having a second, different AI model review an agent's work catches errors the first model missed

Leans right

Cross-model and independent review plausibly reduces shared blind spots and self-preference bias, but evidence is mostly practitioner anecdote rather than controlled comparison.

· denied 0 ·

Asserts (3)Denies (0)Verdicts (1)

DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 (Lex Fridman)“have one check the other's job. This is my standard operating procedure now.”
Coding with LLMs in the summer of 2025 (an update) - <antirez> (antirez.com)“you need at least two LLMs to do some back and forth for complex problems”
Using AI to write better code more slowly (Read the Tea Leaves)“the more, different models you throw at a PR review, the less likely you are to get hallucinations or bogus bugs.”
Leans right AI guessCross-model and independent review plausibly reduces shared blind spots and self-preference bias, but evidence is mostly practitioner anecdote rather than controlled comparison.

AI productivity metrics should be aggregated at team level, not used to judge individual developers

Untestable

This is a management policy recommendation, not an empirical claim.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Measuring AI code assistants and agents with the AI Measurement Framework (DX)“we're always talking about aggregating it on a team level. We're not ever talking about using it for individual performance management”
The Truth About Developer Productivity in the AI Age (IT'S A TRAP) (Modern Software Engineering)“You need to remember that a team is the unit of delivery, not an individual.”
Untestable AI guessThis is a management policy recommendation, not an empirical claim.

AI adoption has not reduced developer burnout or workflow friction

Leans partly right

DORA 2025 found no measurable effect of AI on burnout or friction, but this rests on survey data from one research program.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

DORA 2025: Measuring Software Delivery After AI (Alt + E S V)“Similar levels of burnout”
DORA Community Discussion - Decoding DORA 2025: AI’s Real Impact (DORA)“it doesn't seem to be solving problems of friction and burnout. Um which is something which is strange”
Leans partly right AI guessDORA 2025 found no measurable effect of AI on burnout or friction, but this rests on survey data from one research program.

Coding agents tend to write silent fallbacks and local error recovery that make systems brittle

Leans partly right

Practitioners widely report agents adding defensive fallbacks and swallowed errors to make code run, but there is little systematic measurement of how often this happens.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“tries to read a config file and if it doesn't read a config file, it loads some defaults.”
Beyond the Hype: Real Talk on AI-Assisted Development • Jessica Kerr & Diana Montalion • GOTO 2025 (GOTO Conferences)“It's so good at hiding its own failures and the failures that happen at runtime.”
Probably partly right AI guessPractitioners widely report agents adding defensive fallbacks and swallowed errors to make code run, but there is little systematic measurement of how often this happens.
Leans partly right Debt study (302k AI commits) lines 638-647Mining 302.6k AI-authored commits: broad exception handling is the single most common introduced issue (41,374, 8.5% of all issues; e.g. Claude Code adding a bare 'except: pass'), consistent with the claim, though there is no human-commit baseline to compare against.

Fine-tuning a code model on a proprietary language needs training examples shaped like the target task, not just raw code

Leans right

Supervised fine-tuning works best when training data matches the deployment input/output format, consistent with Meta's CodeCompose and general fine-tuning experience.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

AI Engineering at Jane Street - John Crepezzi (AI Engineer)“you have to have the model see a bunch of examples that are in the shape of the type of question”
AI in software engineering at Google: Progress and the path ahead (research.google)“The training data is an aligned corpus of code with task-specific annotations in input as well as in output.”
Probably right AI guessSupervised fine-tuning works best when training data matches the deployment input/output format, consistent with Meta's CodeCompose and general fine-tuning experience.

A model-agnostic AI tooling layer lets organizations swap models and prompts as models change

Leans partly right

Decoupling tooling from a single model eases switching, which matters given fast model turnover, but it is an architecture choice with tradeoffs rather than a measured result.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

AI Engineering at Jane Street - John Crepezzi (AI Engineer)“we're able to change it in one place Downstream of the editors and then have it available everywhere”
What We Learned Deploying AI within Bloomberg’s Engineering Organization – Lei Zhang, Bloomberg (AI Engineer)“we have a gateway so that teams can easily figure out which model works the best”
Probably partly right AI guessDecoupling tooling from a single model eases switching, which matters given fast model turnover, but it is an architecture choice with tradeoffs rather than a measured result.

Git and current code-forge workflows do not scale to the volume of changes coding agents produce

Leans partly right

Merge-queue and monorepo strain is reported by heavy agent users, but the evidence is anecdotal and Git scales far with existing extensions.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“Git and GitHub uh forges in their current form do not work with agentic infrastructure today”
A Year Of Vibes (Armin Ronacher)“It’s not just GitHub, it’s also git that is lacking.”
Leans partly right AI guessMerge-queue and monorepo strain is reported by heavy agent users, but the evidence is anecdotal and Git scales far with existing extensions.

Coding agents are highly useful for research tasks such as surveying libraries and probing edge cases

Leans partly right

Agents speed up broad searches and surveys, but deep-research outputs still contain errors and fabricated details that need checking.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Mitchell Hashimoto’s new way of writing code (The Pragmatic Engineer)“I absolutely love them for research.”
My AI Adoption Journey (Mitchell Hashimoto)“Deep research sessions where I'd ask agents to survey some field”
Probably partly right AI guessAgents speed up broad searches and surveys, but deep-research outputs still contain errors and fabricated details that need checking.

Coding agents are reliable at mechanical refactoring such as extracting functions and removing duplication

Leans partly right

Refactoring is a relative strength, but studies of LLM refactorings still find behavior-changing errors, so tests or review remain needed.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (5)

Agentic Engineering in Action with Mitchell Hashimoto (Zed Industries)“I think that all the agents are really good at refactoring.”
</> htmx ~ Code is Cheap(er) (htmx.org)“There are times when this can be appropriate, such as in a mechanical refactor”
Probably partly right AI guessRefactoring is a relative strength, but studies of LLM refactorings still find behavior-changing errors, so tests or review remain needed.
Leans partly right Google migrations report lines 417-423Google: 87% of AI JUnit-migration code landed unchanged and the Joda migration saved about 89% of time on small clusters, but in the int64 migration humans usually reverted some changes and messy tests needed human fixes.
Leans partly right Airbnb migration lines 61-65Airbnb automated 97% of a 3.5K-file test-framework migration while preserving test intent and coverage, but only with validation gates, many retries and heavy context, so a single pass is not reliable.
Leans partly right Echoes of AI maintainability RCT lines 1982-2001Observational: 72% of AI-devs resolved the Bumpy Road smell vs 46% without AI, but AI-devs fully removed Complex Method smells less often (18% vs 32%); assistants, not agents.
Leans partly right Debt study (302k AI commits) lines 919-922AI-co-authored commits fix about as many lint-level code smells as they introduce (439,817 vs 432,748), which the authors take as effective local cleanup; this shows smell removal, not reliability on specific refactorings like extracting functions.

Multimodal models can reproduce a UI from a screenshot well enough to ship after light editing

Leans partly right

Screenshot-to-code benchmarks show strong layout reproduction for common components, with fidelity dropping for complex or custom interfaces.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Agentic Engineering in Action with Mitchell Hashimoto (Zed Industries)“it's like shockingly good at this sort of multimodal like here's a picture and reproduce it in Swift UI.”
My AI Adoption Journey (Mitchell Hashimoto)“The command palette that ships for macOS in Ghostty today is only very lightly modified from what Gemini produced for me in seconds.”
Leans partly right AI guessScreenshot-to-code benchmarks show strong layout reproduction for common components, with fidelity dropping for complex or custom interfaces.

Agentic bug fixes often make the symptom go away without fixing the root cause

Leans partly right

Audits of SWE-bench-style patches find many plausible but incorrect or overfitted fixes, though the rate varies by task and model.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“It made It made the symptom go away, so it air quote fixed the bug, but it just was so wrong”
How far can we push AI autonomy in code generation? (martinfowler.com)“Let me fix this by adding @JsonIgnore to the field”
Probably partly right AI guessAudits of SWE-bench-style patches find many plausible but incorrect or overfitted fixes, though the rate varies by task and model.

Enterprise spending on AI coding tools is growing fast enough to become a budget-level concern

Leans partly right

Token-heavy agentic use has raised costs at many firms, but the figures are self-reported and vary widely by company.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

AI in engineering: Q2 2026 benchmarks & research readout (DX)“spend has gone up rapidly. This is about a 28x at the top here with the largest companies”
Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“since 2024, our our costs have gone up at least 6x.”
Leans partly right AI guessToken-heavy agentic use has raised costs at many firms, but the figures are self-reported and vary widely by company.

LLMs are most useful for tasks whose output is easier to verify than to produce

Leans right

Anthropic: engineers delegate easily verifiable tasks ('amazing for everything where validation effort isn't large in comparison to creation effort'), and time savings came mostly from quickly verifiable tasks.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

Oxide and Friends 8/5/2024 -- Pragmatic LLM usage with Nicholas Carlini (Oxide Computer Company)“there's so many things that are easier to verify than write yourself”
Killing Coding Agent Slop With Adversarial Self-Play (Telos)“Generation is a fundamentally hard and unforgiving search problem. Verification is simpler.”
Probably right AI guessVerification cost largely determines whether AI help saves time; field studies find gains concentrate where outputs are cheap to check.
Leans right Anthropic internal study lines 58-59Anthropic: engineers delegate easily verifiable tasks ('amazing for everything where validation effort isn't large in comparison to creation effort'), and time savings came mostly from quickly verifiable tasks.

Having LLMs write code in strongly typed, memory-safe languages such as Rust catches more of their errors at compile time

Leans partly right

Compiler checks do catch whole classes of AI errors such as type and data-race bugs, but logic errors still get through and AI does worse in less common languages.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Oxide and Friends 8/5/2024 -- Pragmatic LLM usage with Nicholas Carlini (Oxide Computer Company)“if I ask it to write it in rough like just there's not going to be aace Condition”
FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“the nice thing about Rust is you can get a lot of verification just from compilation”
Probably partly right AI guessCompiler checks do catch whole classes of AI errors such as type and data-race bugs, but logic errors still get through and AI does worse in less common languages.

LLM hallucinations will largely disappear as models improve

Leans partly right

Hallucination rates have fallen markedly in newer models, but they remain non-zero, especially on obscure facts, and are structurally hard to eliminate.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Oxide and Friends 8/5/2024 -- Pragmatic LLM usage with Nicholas Carlini (Oxide Computer Company)“I'm hopeful that as they get better they will do this less”
Teaching CS50 with AI - David J. Malan (CS50)“I do think this problem will go further and further away over time as the technology gets better.”
Leans partly right AI guessHallucination rates have fallen markedly in newer models, but they remain non-zero, especially on obscure facts, and are structurally hard to eliminate.

More AI-generated code makes production observability more important

Leans partly right

DORA's finding that AI adoption raises instability supports better detection, but no study directly measures observability's value for AI code.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“with observability, you can actually see what's going on”
Beyond the Hype: Real Talk on AI-Assisted Development • Jessica Kerr & Diana Montalion • GOTO 2025 (GOTO Conferences)“I think observability is more important because there's just more code”
Leans partly right AI guessDORA's finding that AI adoption raises instability supports better detection, but no study directly measures observability's value for AI code.

Alternating between different models within one agent run outperforms using either model alone

Leans partly right

Plausible because model errors are partly uncorrelated, but the evidence is the vendor's own benchmark with no independent replication.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Inside the Rise of Autonomous AI Hackers: XBOW's Oege de Moor (Sequoia Capital)“This is much better than either model separately.”
An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“Codex can often speed up the algorithm by 1.5x-2x, then Opus somehow speeds up that optimized code again to a greater degree”
Leans partly right AI guessPlausible because model errors are partly uncorrelated, but the evidence is the vendor's own benchmark with no independent replication.

AI-generated code lacks the surface signals reviewers rely on to spot bad code, making review harder

Leans partly right

Polished but wrong AI code is widely reported, but there is little controlled evidence that review accuracy drops specifically for AI code.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“all the telltale signs you get of bad code, it's like those smells aren't there anymore”
"Do You Still Read the Code?" • zanlib (zanlib)“Code quality was a useful proxy for that understanding in the time before Claude Code, but a language model can now feign that understanding convincingly.”
Leans partly right AI guessPolished but wrong AI code is widely reported, but there is little controlled evidence that review accuracy drops specifically for AI code.
Leans partly right Debt study (302k AI commits) lines 905-907Authors argue AI-introduced code smells (89.3% of issues) often do not break anything immediately and are therefore easy to accept during review; this is interpretation, not a reviewer study.

AI agents make formal verification cheap enough to apply more widely

Unknown

Agents help with Lean and proof engineering, but whether this makes verification practical for ordinary software is unproven as of 2026.

· ·

Asserts (1)Denies (1)Verdicts (1)

Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“this all makes me more excited about formal verification.”
DENIES LLMs are bad at vibing specifications (buttondown.com)“being easily able to write specifications doesn't help with correctness if the specs don't actually verify anything”
Unknown AI guessAgents help with Lean and proof engineering, but whether this makes verification practical for ordinary software is unproven as of 2026.

Current AI agents do not learn continually from feedback, so corrections do not persist across sessions

Leans right

Model weights are fixed at deployment; persistence relies on external memory or instruction files, which only partly substitute.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

This is not the year of agents – Andrej Karpathy (Dwarkesh Clips)“they don't have continual learning. You can't just tell them something and they'll remember it.”
Now more than ever: building reliable software in the age of agents | Ron Minsky | Bug Bash 2026 (Antithesis)“the session is over and like all is forgiven and forgotten.”
Very likely right AI guessModel weights are fixed at deployment; persistence relies on external memory or instruction files, which only partly substitute.

Authors should disclose their use of AI tools in their work

Untestable

A norm about research and engineering practice rather than a testable claim.

· ·

Asserts (1)Denies (1)Verdicts (1)

ICM 2026 Public Lectures - Terence Tao (Simons Foundation)“we really need to normalize responsible disclosure of AI systems.”
DENIES 576 - Using LLMs at Oxide | RFD | Oxide (Oxide Computer Company)“Note this isn’t as simple as disclosure of usage”
Untestable AI guessA norm about research and engineering practice rather than a testable claim.

LLM-generated prose is clichéd and recognizable, and readers lose trust in writing they believe is LLM-authored

Leans partly right

Studies show readers rate text lower once they believe it is AI-written and LLM prose has detectable stylistic tells, but careful use can produce unremarkable prose and 'always clichéd' overstates it.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Learning to Code with AI and Steve Klabnik (Scott Hanselman)“this text you gave me is far too LMY”
576 - Using LLMs at Oxide | RFD | Oxide (Oxide Computer Company)“At best, writing from LLMs is hackneyed and cliché-ridden; at worst, it brims with tells that reveal that the prose is in fact automatically generated.”
Probably partly right AI guessStudies show readers rate text lower once they believe it is AI-written and LLM prose has detectable stylistic tells, but careful use can produce unremarkable prose and 'always clichéd' overstates it.

Getting good results from AI coding tools takes significant skill, so outcomes vary widely with user expertise

Leans right

Bayesian analysis: the more AI-skilled the original developer, the larger the downstream CodeHealth benefit (+0.10, about half an SD, for habitual users). The authors conclude developers vary in their ability to use AI well, though Java proficiency mattered even more.

· ·

Asserts (1)Denies (1)Verdicts (9)

Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“I feel like there's a big to use gamer terms as a high skill ceiling basically”
DENIES No, AI is not Making Engineers 10x as Productive (colton.dev)“Using AI to code is not hard to learn.”
Leans partly right AI guessPractitioner reports and field studies suggest usage skill matters, but METR found no clear learning-curve effect and the basics are learned quickly.
Leans right Echoes of AI maintainability RCT lines 2541-2551Bayesian analysis: the more AI-skilled the original developer, the larger the downstream CodeHealth benefit (+0.10, about half an SD, for habitual users). The authors conclude developers vary in their ability to use AI well, though Java proficiency mattered even more.
Leans partly right Google RCT lines 522-543Google RCT, 96 engineers: senior and heavy-coding developers were tentatively faster with AI and natural-language-to-code needs prompting skill, but how often people already used AI tools had no significant effect on their AI speed-up, so the authors point to a 'low usability floor'.
Leans partly right Anthropic internal study lines 36-40Anthropic survey (n=132): time-savings responses cluster at opposite ends, with some spending more time debugging AI code; authors attribute gains to learned 'strategic delegation skills' and task choice, supporting wide variation by user practice.
Leans partly right Shan et al. field study + RCT lines 1037-1064Results depended on how developers used ChatGPT: direct acceptance raised output but not accuracy, while ideation, improvement and debugging raised accuracy, and debugging lowered output. Outcomes vary with usage style, but usage was self-chosen, so the evidence is correlational.
Leans partly right METR 2025 RCT lines 1217-1234METR RCT: speedup did not improve over the first 50 hours of Cursor use, and the lone >50h developer's speedup (~25%) fell to ~0% once a second developer's misreported experience was corrected. The 'high skill ceiling' idea is plausible but underpowered.
Leans partly right Microsoft/Accenture field RCTs lines 758-770Field RCTs, ~4,900 developers: Copilot gains varied with experience, but in the opposite direction from 'skill needed'. Short-tenure and junior developers gained 21-40% while senior and long-tenure developers gained 7-16% (noisy, not significant).
Leans right Vibe-coding literature review lines 1518-1526Vibe-coding MLR (47 sources, mention counts): quality, maintainability and productivity outcomes are described as conditional on developer expertise, task complexity and verification practice, and outcome claims vary substantially by user expertise.
Unknown Code-ownership experiment (Martin-Lopez) lines 828-835Gains appeared in every experience group, but the authors state they cannot separate experience effects from task and language effects, so the study cannot say whether outcomes depend on user expertise.

AI coding agents help keep code comments and documentation in sync with code changes

Leans partly right

Agents often update nearby comments during edits, but they also add redundant comments and there is little systematic evidence.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“as part of the change to the function it just like fixed the comment”
Learning to Code with AI and Steve Klabnik (Scott Hanselman)“if you can like, you know, have a tool that is able to reasonably check that they're reasonably up todate”
Leans partly right AI guessAgents often update nearby comments during edits, but they also add redundant comments and there is little systematic evidence.
Leans partly right Echoes of AI maintainability RCT lines 1952-1956AI-assisted developers added comments more often (79% vs 54%), but the authors describe them as noisy surface-level comments that risk drifting out of sync with the logic. That shows more comments, not better sync.

AI coding agents let developers make useful progress when they are too tired or distracted to code themselves

Leans partly right

This fits METR's finding of lower perceived effort and practitioner reports, but it has not been directly measured.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

My AI Adoption Journey (Mitchell Hashimoto)“the latter part of the working day, I'm usually tired and coming out of flow and find myself too personally inefficient”
METR's AI productivity study is really good (seangoedecke.com)“AI means I can work at times that I otherwise would be able to get nothing done at all”
Leans partly right AI guessThis fits METR's finding of lower perceived effort and practitioner reports, but it has not been directly measured.
Leans partly right METR 2025 RCT lines 1166-1178METR RCT: reports on effort were mixed ('felt like less effort' vs 'no effort difference'), and 69% kept using Cursor despite the measured slowdown. That hints at lower cognitive load, but tired or distracted work was not studied.

Computer-use (GUI) agents are considerably less capable than text- or CLI-based agents

Leans right

Through 2025-26 GUI/computer-use benchmarks lagged text and terminal agent benchmarks, and practical use stayed mostly CLI-based.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

Armin Ronacher Leaning In To Find Out - PyAI Conf 2026 (Pydantic)“they still suck at clicking stuff in the UI even with the multimodal models”
How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR (AI Engineer)“computer use capabilities are considerably worse than CLI based stuff at the moment”
Probably right AI guessThrough 2025-26 GUI/computer-use benchmarks lagged text and terminal agent benchmarks, and practical use stayed mostly CLI-based.

How much human review AI output gets should be set by risk: probability of error, impact and detectability

Leans partly right

Anthropic interviews: engineers in practice delegate easily verifiable, low-stakes or throwaway work and keep active supervision for complex or high-stakes code, matching a risk-based review practice (descriptive, not evidence that it is optimal).

· denied 0 ·

Asserts (2)Denies (0)Verdicts (3)

From MCP and Vibe Coding to Harness Engineering: How Did AI Native Engineering Evolve in One Year (InfoQ)“risk is always probability plus impact plus detectability”
Nobody Agrees About Agentic Development (Convex)“if the agent gets it wrong, will that be a really big problem? How important is it? That's where you make your decision.”
Leans partly right Anthropic internal study lines 58-63Anthropic interviews: engineers in practice delegate easily verifiable, low-stakes or throwaway work and keep active supervision for complex or high-stakes code, matching a risk-based review practice (descriptive, not evidence that it is optimal).
Leans right Vibe-coding literature review lines 1937-1946The review's practitioner implication is that verification effort should scale with the intended use of the artifact. It also notes that developers apply stricter verification to unfamiliar functionality than to routine tasks (L1301-1302). This is a synthesized recommendation, not a tested result.
Untestable AI guessA sensible risk-management heuristic rather than an empirical claim.

LLMs can rapidly diagnose production incidents from logs and traces

Leans partly right

Anecdotes and vendor incident-response agents report useful triage, but controlled evaluations of root-cause accuracy remain limited.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (1)

What We Learned Deploying AI within Bloomberg’s Engineering Organization – Lei Zhang, Bloomberg (AI Engineer)“it can go through your codebase really quickly it can go through your telemetry system very quickly”
My AI Skeptic Friends Are All Nuts (Fly)“watched it in seconds spot LVM metadata corruption issues on a host we've been complaining about for months”
Leans partly right AI guessAnecdotes and vendor incident-response agents report useful triage, but controlled evaluations of root-cause accuracy remain limited.

Hallucinated content that enters an agent's context compounds into later errors, so information should be validated before it enters context

Leans right

Errors in context are re-used by later turns; validating tool outputs before storing them is standard agent hygiene.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Context engineering for AI agents (Microsoft Developer)“context poisoning ... validate information before it enters the context”
Probably right AI guessErrors in context are re-used by later turns; validating tool outputs before storing them is standard agent hygiene.
Leans partly right Vibe-coding literature review lines 1574-1580MLR codes 'cumulative context errors and iteration drift' (2 of 47 sources): misunderstandings from earlier prompts propagate and WL15 reports successive prompts amplifying earlier errors; the validate-before-context remedy is not tested.

OpenAI Codex is the strongest choice for orchestrating parallel and scheduled background agents

Leans partly right

Codex does emphasize parallel cloud tasks and automations, but Cursor, Copilot and Claude Code offer similar features; 'strongest' is unproven.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Codex vs Claude Code vs Cursor vs Copilot — Best AI Coding Agent in 2026? (Byte & Beyond)“for automation and parallel delegation, Codex has one of the clearest product visions”
Leans partly right AI guessCodex does emphasize parallel cloud tasks and automations, but Cursor, Copilot and Claude Code offer similar features; 'strongest' is unproven.

Claude Code's main strength is a scriptable, customizable terminal workflow (subagents, hooks, headless mode)

Leans right

Hooks, subagents, headless/SDK mode and MCP make Claude Code highly scriptable; accurate description of its design focus.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Codex vs Claude Code vs Cursor vs Copilot — Best AI Coding Agent in 2026? (Byte & Beyond)“a programmable senior pair living in your terminal”
Probably right AI guessHooks, subagents, headless/SDK mode and MCP make Claude Code highly scriptable; accurate description of its design focus.

Context engineering delivers order-of-magnitude (10x or more) productivity gains

Leans wrong

No controlled study shows 10x gains; measured effects of AI coding range from a slowdown (METR) to roughly 20-50% on suitable tasks. 10x is marketing.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

Context Engineering 101 - The Simple Strategy to 100x AI Coding (Cole Medin)“it's going to 10x your workflow no matter what you want to make”
Probably wrong AI guessNo controlled study shows 10x gains; measured effects of AI coding range from a slowdown (METR) to roughly 20-50% on suitable tasks. 10x is marketing.
Leans partly right Airbnb migration lines 6-6Airbnb's 1.5-year estimate versus 6 weeks implies a gain of about 13x on one migration, but that comes from automation, retries and context together, set against a manual estimate rather than a measured baseline.
Unknown Airbnb migration lines 6-6Re-judged: Airbnb's ~13x migration speedup came from automation, validation loops and retries on one bulk migration, not from context engineering in day-to-day work, so it doesn't bear on this claim.

Almost-right AI code is costlier than obviously wrong code because it passes review and ships

Leans right

Stack Overflow 2025 found 'almost right' was the top frustration (66%); subtle defects that pass tests are costlier to find later, a well-known pattern.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

AI-Generated Code Is Creating a New Kind of Technical Debt (Imran Gardezi)“Almost right is more expensive than completely wrong”
Probably right AI guessStack Overflow 2025 found 'almost right' was the top frustration (66%); subtle defects that pass tests are costlier to find later, a well-known pattern.
Leans partly right Debt study (302k AI commits) lines 904-907The Debt paper finds smells and subtle issues don't break things immediately, so they pass review and persist. That matches the mechanism, but cost isn't compared.
Leans partly right Vibe-coding literature review lines 79-86The review notes code is often accepted on behavior, so plausible-but-incorrect output can pass unnoticed and surface later as defects; it does not compare the cost with obviously wrong code.

Multi-agent role pipelines (PM, engineer, QA) cost several times more tokens but improve output quality

Leans partly right

Separate reviewer/QA passes do catch more defects and clearly multiply token cost; evidence that full role pipelines beat a single well-prompted agent is mixed.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

How to Work with AI Coding Agents: Spec-Driven Development, Context and Loop Engineering, Workflows (DataTalksClub ⬛)“your token consumption goes like I don't know four five times more ... what you get in return is uh better quality”
Leans partly right AI guessSeparate reviewer/QA passes do catch more defects and clearly multiply token cost; evidence that full role pipelines beat a single well-prompted agent is mixed.

SWE-bench Verified scores overstate real coding ability because of training-data contamination

Leans right

Contamination and memorization in SWE-bench Verified are documented, and scores drop substantially on SWE-bench Pro and fresh benchmarks.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Cursor vs Claude Code vs GitHub Copilot — Which AI Coding Tool Actually Wins? (2026) (Nordic AI Dev)“SWE-Bench Verified has a contamination problem”
Probably right AI guessContamination and memorization in SWE-bench Verified are documented, and scores drop substantially on SWE-bench Pro and fresh benchmarks.

Claude models lead the major AI coding tools on agentic coding benchmarks

Leans partly right

Claude models have often topped SWE-bench and similar leaderboards, but GPT-5.x/Codex and Gemini models trade the lead release to release; also, Cursor and Copilot can run Claude models.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Cursor vs Claude Code vs GitHub Copilot — Which AI Coding Tool Actually Wins? (2026) (Nordic AI Dev)“Claude's Opus models have posted the highest scores of the three”
Leans partly right AI guessClaude models have often topped SWE-bench and similar leaderboards, but GPT-5.x/Codex and Gemini models trade the lead release to release; also, Cursor and Copilot can run Claude models.

Without continuous delivery and automated deployment pipelines, AI-assisted development makes teams worse

Leans partly right

DORA 2025 found AI amplifies existing practices, helping strong teams and hurting weak ones; 'makes you worse' overstates it for small or solo projects.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Why AI WON'T Replace Software Engineering... (Modern Software Engineering)“If you don't have continuous delivery in place, AI will make you worse, not better”
Leans partly right AI guessDORA 2025 found AI amplifies existing practices, helping strong teams and hurting weak ones; 'makes you worse' overstates it for small or solo projects.

AI engineers need machine-learning fundamentals to debug LLM-based systems effectively

Leans partly right

ML intuition helps with evals, retrieval and fine-tuning, but much LLM application work is software and data engineering; not strictly required for entry.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Don't Waste 2026 on the Wrong Career (Software vs AI Engineer) (Sajjaad Khader)“people who tell you that you don't need machine learning fundamentals is wrong”
Leans partly right AI guessML intuition helps with evals, retrieval and fine-tuning, but much LLM application work is software and data engineering; not strictly required for entry.

A computer science degree is no longer necessary to land a software or AI engineering job

Leans partly right

Many employers dropped degree requirements, but in the tight 2024-26 market most entry-level hires still have CS degrees; a degree is not required but still helps a lot.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Don't Waste 2026 on the Wrong Career (Software vs AI Engineer) (Sajjaad Khader)“if you're just trying to become a big tech software engineer, it's not necessary as it once was”
Leans partly right AI guessMany employers dropped degree requirements, but in the tight 2024-26 market most entry-level hires still have CS degrees; a degree is not required but still helps a lot.

Software engineering and AI engineering will converge into one role

Untestable

A long-range prediction about job titles; cannot be evaluated now.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Don't Waste 2026 on the Wrong Career (Software vs AI Engineer) (Sajjaad Khader)“eventually software engineering and AI engineering is going to become the same thing”
Untestable AI guessA long-range prediction about job titles; cannot be evaluated now.

AI engineers are paid substantially more than general software engineers

Leans right

Compensation data (Levels.fyi, 2025) show a premium for AI/ML engineering roles, though ranges overlap heavily.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Don't Waste 2026 on the Wrong Career (Software vs AI Engineer) (Sajjaad Khader)“unfortunately it's not as lucrative”
Leans right AI guessCompensation data (Levels.fyi, 2025) show a premium for AI/ML engineering roles, though ranges overlap heavily.

AI app generation on a governed low-code platform yields secure, compliant applications by default

Leans wrong

Platform guardrails can remove classes of bugs (e.g. SQL injection), but no generator makes apps '100% secure'; logic flaws, access-control mistakes and misconfiguration remain.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Enterprise Vibe Coding: Building Secure, AI-Powered Apps in 30 Minutes - Parth Sharma (Developer Summit)“is going to be 100% secure, 100% working 100% of the times”
Probably wrong AI guessPlatform guardrails can remove classes of bugs (e.g. SQL injection), but no generator makes apps '100% secure'; logic flaws, access-control mistakes and misconfiguration remain.

Enterprise-grade applications can go from idea to production in under an hour with AI generation

Leans wrong

A demo CRUD app can be generated in minutes, but enterprise production also needs requirements, integration, security review and operations; no evidence supports sub-hour delivery as normal.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Enterprise Vibe Coding: Building Secure, AI-Powered Apps in 30 Minutes - Parth Sharma (Developer Summit)“How do you go from an idea to production in less than 30 minutes?”
Probably wrong AI guessA demo CRUD app can be generated in minutes, but enterprise production also needs requirements, integration, security review and operations; no evidence supports sub-hour delivery as normal.

Continuous automated security testing (CTEM) is the only practical way to secure fast AI-generated code

Leans wrong

Continuous testing helps, but SAST in CI, dependency scanning, code review, secure defaults and threat modeling are also practical; 'only option' is a vendor pitch.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

The AI Code Problem Nobody's Talking About (Razorwire Cybersecurity)“if you're going to be doing vibe coding in the future, I think CTM is your only option”
Leans wrong AI guessContinuous testing helps, but SAST in CI, dependency scanning, code review, secure defaults and threat modeling are also practical; 'only option' is a vendor pitch.
Leans partly right Debt study (302k AI commits) lines 937-944Authors recommend static analysis, security checks and continuous post-merge monitoring because AI-introduced issues persist, but they do not claim or show that continuous automated testing is the only practical approach.
Unknown Debt study (302k AI commits) lines 937-944Re-judged: the Debt paper recommends static analysis, tests, security checks and post-merge monitoring together; it does not show that continuous automated security testing is the only way, so it doesn't settle this claim.

AI lets developers ship in hours features that used to take a multi-week sprint

Leans wrong

Some small greenfield features do go that fast, but controlled studies show modest average gains (or slowdowns) on real work; testing, review and deployment don't shrink 100x.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Will AI replace Software Engineers ? Raw & Honest Opinion (Apporva arya)“two-week sprint work can now be done in an hour or even five minutes”
Probably wrong AI guessSome small greenfield features do go that fast, but controlled studies show modest average gains (or slowdowns) on real work; testing, review and deployment don't shrink 100x.
Leans wrong Shan et al. field study + RCT lines 489-493Across 201 enterprise developers, average throughput rose by about one user story per week (+18%), nothing like compressing multi-week sprints into hours (pre-agentic tools).

Companies are cutting staff to redirect spending toward AI

Leans partly right

Several firms (Salesforce, Microsoft, Amazon) tied cuts or hiring freezes to AI investment, but cost cutting and macro factors explain much of it.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Will AI replace Software Engineers ? Raw & Honest Opinion (Apporva arya)“they are laying off people so they can put that money into AI”
Leans partly right AI guessSeveral firms (Salesforce, Microsoft, Amazon) tied cuts or hiring freezes to AI investment, but cost cutting and macro factors explain much of it.

Developers using AI assistants write less secure code while believing it is more secure

Leans right

Perry et al. (Stanford, 2022/CCS 2023) found AI-assisted participants wrote less secure code and were more likely to believe it secure.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI Made Devs 19% Slower (METR 2025 Study) (zencoderai)“they had less secure output, but more confidence that it was secure”
Probably right AI guessPerry et al. (Stanford, 2022/CCS 2023) found AI-assisted participants wrote less secure code and were more likely to believe it secure.

Coding agents working in the background will eliminate technical debt

Leans wrong

AI commits fix about as many smells as they introduce, but they add more correctness and security issues than they fix, and 22.7% of introduced issues persist for months.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

How the GitHub Copilot coding agent works | GitHub Checkout (GitHub)“Hopefully this is going to be the end of tech debt”
Probably wrong AI guessEvidence so far shows AI adoption adding duplication and churn; agents can pay down some debt, but they also create it.
Leans wrong Debt study (302k AI commits) lines 918-933AI commits fix about as many smells as they introduce, but they add more correctness and security issues than they fix, and 22.7% of introduced issues persist for months.

Aider's git-integrated, diff-first terminal workflow gives developers more control than GUI AI editors

Leans partly right

Aider auto-commits and shows diffs, which aids control and rollback; whether that beats GUI editors' diff review is preference.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Aider Review — The Best AI Pair-Programmer in Your Terminal? (2025) (HMD Tech)“It shows every diff before applying changes, keeps your project local”
Leans partly right AI guessAider auto-commits and shows diffs, which aids control and rollback; whether that beats GUI editors' diff review is preference.

Zed is faster and more responsive than other AI code editors

Leans partly right

Zed's Rust/GPU rendering is genuinely fast as an editor; agent speed depends on the model, not the editor.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Zed Editor UPDATE: NEW Agentic AI IDE - Cursor + Windsurf Alternative! FULLY FREE! (Open source) (WorldofAI)“Honestly, it's way faster than most IDs I've tried”
Leans partly right AI guessZed's Rust/GPU rendering is genuinely fast as an editor; agent speed depends on the model, not the editor.

The Agent Client Protocol lets one editor host multiple third-party coding agents

Leans right

ACP (Zed/JetBrains) is an open protocol letting editors run external agents like Claude Code, Gemini CLI and Codex.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Zed Editor UPDATE: NEW Agentic AI IDE - Cursor + Windsurf Alternative! FULLY FREE! (Open source) (WorldofAI)“you also have the capability to implement other external agents inside zed ... thanks to the agent client protocol”
Probably right AI guessACP (Zed/JetBrains) is an open protocol letting editors run external agents like Claude Code, Gemini CLI and Codex.

One senior engineer with AI can now do the work of about ten engineers

Leans wrong

No measured study shows anything near 10x; field data put average gains around 10-20% and experts on mature code sometimes slow down.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Computer Science Degrees Are WORTHLESS, Now Workers Are FURIOUS — Here’s Why (Jay Reed)“One senior engineer today with AI can do the work of 10 engineers in the past world”
Probably wrong AI guessNo measured study shows anything near 10x; field data put average gains around 10-20% and experts on mature code sometimes slow down.
Leans wrong Anthropic internal study lines 35-35Anthropic self-reports an average +50% productivity boost, with only 14% of respondents above +100%, far from 10x even among heavy early adopters at an AI lab.

A computer science degree has lost most of its career value because of AI

Leans wrong

CS grad unemployment rose (~6-7%) but CS still has high median earnings and employment; the degree's value fell somewhat, not 'most' of it.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Computer Science Degrees Are WORTHLESS, Now Workers Are FURIOUS — Here’s Why (Jay Reed)“the computer science degree is done”
Leans wrong AI guessCS grad unemployment rose (~6-7%) but CS still has high median earnings and employment; the degree's value fell somewhat, not 'most' of it.

AI code review complements but does not replace automated tests

Leans right

AI reviewers miss many defects that tests catch and are nondeterministic; tests remain the reliable regression check. Consistent with curl experience and review-tool benchmarks.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Open Source security in spite of AI - Daniel Stenberg (Daniel Stenberg)“they're certainly not a compliment to test cases ... test cases is still way more important”
Probably right AI guessAI reviewers miss many defects that tests catch and are nondeterministic; tests remain the reliable regression check. Consistent with curl experience and review-tool benchmarks.

AI crawlers impose heavy bandwidth and load costs on open-source project infrastructure

Leans right

Many projects (SourceHut, GNOME, KDE, Fedora, Read the Docs) reported AI-scraper traffic overwhelming servers in 2025 and deployed blockers like Anubis.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Open Source security in spite of AI - Daniel Stenberg (Daniel Stenberg)“they easily swallow 75 terabytes per month ... less than 0.01% of the traffic [is downloads]”
Very likely right AI guessMany projects (SourceHut, GNOME, KDE, Fedora, Read the Docs) reported AI-scraper traffic overwhelming servers in 2025 and deployed blockers like Anubis.

Forward-deployed engineering is a small niche of a few hundred openings and partly a rebrand of solutions and professional-services roles

Leans right

Postings data cited (Palantir, Databricks, labs) and LinkedIn growth from a tiny base support 'real but small'; Palantir's own filings describe the work as professional services.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Forward Deployed Engineer: The Truth About Tech's Hottest Job (I Counted Every Open Role) (Code with Jay)“about 281 roles worldwide, a real job market of a few hundred seats globally, not a wave ... The work is real. The title is a recruiting instrument. Both.”
Probably right AI guessPostings data cited (Palantir, Databricks, labs) and LinkedIn growth from a tiny base support 'real but small'; Palantir's own filings describe the work as professional services.

Successful enterprise AI deployments start from observing the real workflow rather than building what the customer first asks for

Leans partly right

Consistent with long-standing requirements-engineering and Palantir FDE practice, and with reports that many AI pilots fail from poor workflow fit; anecdotal rather than measured.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Forward Deployed Engineer: The Truth About Tech's Hottest Job (I Counted Every Open Role) (Code with Jay)“The thing the customer asks you for on day one is usually not the thing that is costing them money”
Leans partly right AI guessConsistent with long-standing requirements-engineering and Palantir FDE practice, and with reports that many AI pilots fail from poor workflow fit; anecdotal rather than measured.

Tech CEOs who call coding obsolete keep hiring large numbers of software engineers themselves

Leans right

Nvidia, Meta and AI labs continued growing engineering headcount through 2025 while executives forecast automation; their behavior contradicts the strong form of the claim.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Jensen Huang Said "Stop Learning to Code" — Then Hired 10,000 Developers (Olive Badger)“suddenly this man is hiring several thousand software engineers complaining he's 10,000 short on headcount”
Probably right AI guessNvidia, Meta and AI labs continued growing engineering headcount through 2025 while executives forecast automation; their behavior contradicts the strong form of the claim.

The productivity gap between teams that use AI well and teams that don't is widening

Leans partly right

Stanford's own panel shows widening dispersion and DORA finds outcomes diverge by capability; the projection forward is explicitly speculative.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI Make Developers More Productive? Data from a 100k Engineer Stanford Study | Yegor Denisov (Toronto Machine Learning Society (TMLS))“this difference grew by July of 25 to 20%. So, it's a 4x increase”
Leans partly right AI guessStanford's own panel shows widening dispersion and DORA finds outcomes diverge by capability; the projection forward is explicitly speculative.

Anthropic restricts Claude subscriptions to its own tools while OpenAI allows its subscriptions in third-party harnesses

Leans right

In early 2026 Anthropic blocked consumer-plan OAuth in third-party agents like OpenCode, and OpenAI announced official ChatGPT-login support there.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

OpenCode Review: Because I Hit Claude Code Usage Limits (Claude Code vs OpenCode vs Codex CLI) (pookie)“Anthropic basically blocking the usage of your Claude Code subscription with external services ... OpenAI ... fully allow you”
Probably right AI guessIn early 2026 Anthropic blocked consumer-plan OAuth in third-party agents like OpenCode, and OpenAI announced official ChatGPT-login support there.

LLMs show jagged intelligence: superhuman at some tasks while making errors no human would

Leans right

Well documented: uneven performance across tasks, persistent simple failures alongside expert-level output.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“They display jagged intelligence ... they will insist that 9.11 is greater than 9.9”
Very likely right AI guessWell documented: uneven performance across tasks, persistent simple failures alongside expert-level output.

Partial-autonomy tools with an adjustable autonomy level are more useful today than fully autonomous agents

Leans partly right

Matches 2025-26 usage patterns (interactive agents dominate), though fully autonomous background agents have grown useful for scoped tasks.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Andrej Karpathy: Software Is Changing (Again) (Y Combinator)“it's less Iron Man robots and more Iron Man suits that you want to build”
Leans partly right AI guessMatches 2025-26 usage patterns (interactive agents dominate), though fully autonomous background agents have grown useful for scoped tasks.
Leans partly right Vibe-coding literature review lines 2005-2015The review names a tension between agentic delegation framings and the most-recommended safeguards (HITL review, validation), concluding 'a workable balance appears to lie between the two'.

Writing with an LLM lowers brain engagement and memory of one's own work, with effects that persist when the tool is removed

Leans partly right

The MIT Media Lab preprint (54 participants, 18 in the crossover) found lower EEG connectivity and poor recall; small, not peer-reviewed at release, and essay-specific, so effects on coding are inference.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

Your Brain on ChatGPT: Accumulation of Cognitive Debt (June 2025) (AI Paper Slop)“the LLM to brain group ... their brains did not show that big ramp up in connectivity”
Leans partly right AI guessThe MIT Media Lab preprint (54 participants, 18 in the crossover) found lower EEG connectivity and poor recall; small, not peer-reviewed at release, and essay-specific, so effects on coding are inference.
Leans partly right Echoes of AI maintainability RCT lines 2658-2665Secondhand summary of Kosmyna et al. EEG study (N=54): ChatGPT-assisted essay writers showed lower cognitive activity and worse recall of their work; persistence after tool removal is not mentioned.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 932-943For code, this study finds an analogue of the claim: lower recall and explanation of one's own just-written work after AI use. It cites Kosmyna et al. as consistent. Brain engagement and whether the effect persists after the tool is removed were not tested.

Repository context files (AGENTS.md) can reduce agent task success while increasing cost

Leans partly right

A 2026 study found LLM-generated context files slightly hurt success and raised cost, while concise human-written ones helped a little; the effect depends on content and length.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Codex CLI: Everything You Need in 12 mins (lustoykov)“context files tend to reduce task success rates compared to providing no repository context while also increasing inference cost by over 20%”
Leans partly right AI guessA 2026 study found LLM-generated context files slightly hurt success and raised cost, while concise human-written ones helped a little; the effect depends on content and length.

Errors in research and planning cost far more than errors in individual lines of AI-generated code

Leans right

The classic cost-of-defects gradient applies: requirement and design errors propagate; agents amplify this by generating large amounts of code from a flawed plan.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Context Engineering SF: Advanced Context Engineering for Agents - Dex Horthy (Chroma)“a bad line of research ... can be thousands of bad lines of code”
Probably right AI guessThe classic cost-of-defects gradient applies: requirement and design errors propagate; agents amplify this by generating large amounts of code from a flawed plan.

AI models have reached or surpassed the best human competitive programmers

Leans right

In 2025 OpenAI and Google models reached gold-medal IOI/ICPC-level performance, consistent with the prediction; competitive programming differs a lot from production engineering.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

"STOP GETTING COMPUTER DEGREES" - Sam Altman's SHOCKING Interview (AI Upload)“our internal benchmark is now around 50 and maybe we'll hit number one by the end of this year”
Probably right AI guessIn 2025 OpenAI and Google models reached gold-medal IOI/ICPC-level performance, consistent with the prediction; competitive programming differs a lot from production engineering.

AI agents can complete only a small fraction of real freelance projects to a professional standard

Leans right

The Remote Labor Index (Scale/CAIS, late 2025) found top agents completing ~2.5% of real Upwork-style projects acceptably; newer agents score higher but still low.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

AI Layoffs Don’t Add Up. Here’s the Data (HuMan Intelligence)“The best AI could only complete 2 and a half% of those projects to professional standards”
Probably right AI guessThe Remote Labor Index (Scale/CAIS, late 2025) found top agents completing ~2.5% of real Upwork-style projects acceptably; newer agents score higher but still low.
Leans partly right METR 2025 RCT lines 593-599METR 2025 preliminary: agents often get the core functionality of real OSS issues right but fail to satisfy all requirements (docs, linting, tests). This points the same way for open-source issues, not freelance projects, and gives no rates.

Windsurf has been rebranded as Devin Desktop under Cognition

Unknown

A product-status claim I can't verify from prior knowledge; Cognition did acquire Windsurf in July 2025.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Windsurf Rebrands to Devin Desktop — Meet Devin Local & New AI Features (Helpfolder)“WinSurf is totally converting into Devin Desktop”
Unknown AI guessA product-status claim I can't verify from prior knowledge; Cognition did acquire Windsurf in July 2025.

Offshoring and visa hiring, more than AI, explain recent US tech layoffs

Leans partly right

Offshoring and global capability centers are real cost levers, but visa counts don't map onto layoffs one-to-one, and rate cycles and over-hiring explain much of the decline; the causal claim is weakly supported.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Companies Are Lying About AI Layoffs - Here’s the Proof (Vanessa Wingårdh)“Salesforce received 1,137 H1B visas ... Almost the same amount of people that they just laid off”
Leans partly right AI guessOffshoring and global capability centers are real cost levers, but visa counts don't map onto layoffs one-to-one, and rate cycles and over-hiring explain much of the decline; the causal claim is weakly supported.

AI is already replacing language-heavy jobs such as customer support, moderation and basic copywriting

Leans right

Support and content roles show the clearest measured AI substitution (e.g. Klarna, freelance-writing market studies), though some firms reversed course.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Companies Are Lying About AI Layoffs - Here’s the Proof (Vanessa Wingårdh)“Many of the jobs being lost rely heavily on the English language ... content moderation, basic copywriting, and customer service”
Probably right AI guessSupport and content roles show the clearest measured AI substitution (e.g. Klarna, freelance-writing market studies), though some firms reversed course.

Moving from most to nearly all code being AI-written yields disproportionate speed gains

Leans partly right

Removing the last human-written fraction removes a serial bottleneck for some teams, but it relies on reduced review; no controlled evidence, and risk rises for critical code.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI Dev 26 x SF: Andrew Ng: The Future of Software Engineering (DeepLearningAI)“getting only 80% of the way there feels very different”
Leans partly right AI guessRemoving the last human-written fraction removes a serial bottleneck for some teams, but it relies on reduced review; no controlled evidence, and risk rises for critical code.

Codex CLI uses far fewer tokens than Claude Code for the same task

Leans partly right

Several practitioner comparisons report lower token use for Codex, but results vary by task, model and settings; a single run isn't evidence.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Claude Code vs Codex CLI — The Honest Comparison (2026) (Hyperautomation Labs)“Claude Code consumed 6.1 million tokens ... Codex CLI consumed 1.4 million tokens”
Leans partly right AI guessSeveral practitioner comparisons report lower token use for Codex, but results vary by task, model and settings; a single run isn't evidence.

Claude Code produces higher-quality, safer code than Codex CLI

Unknown

The cited 500-developer blind review has no traceable source; quality comparisons swing with each model release.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Claude Code vs Codex CLI — The Honest Comparison (2026) (Hyperautomation Labs)“In blind code reviews across 500 developers, Claude Code's output was preferred 67% of the time”
Unknown AI guessThe cited 500-developer blind review has no traceable source; quality comparisons swing with each model release.

Codex CLI enforces OS-level sandboxing while Claude Code relies on application-level permissions

Leans partly right

Codex does use Seatbelt/Landlock sandboxing; Claude Code added an OS-level sandbox option in late 2025, so the contrast is outdated.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Claude Code vs Codex CLI — The Honest Comparison (2026) (Hyperautomation Labs)“Codex's kernel sandbox cannot. It's enforced by the operating system”
Leans partly right AI guessCodex does use Seatbelt/Landlock sandboxing; Claude Code added an OS-level sandbox option in late 2025, so the contrast is outdated.

With coding agents, end-to-end and integration tests deliver more value than unit tests

Leans partly right

Higher-level tests catch the cross-component errors agents make and become cheap to run unattended; unit tests still give fast, precise feedback. Plausible practitioner view, not measured.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Test-driven development (TDD) with Claude Code (James Murdza)“with AI, you really want to focus on the end-to-end tests and the integration tests”
Leans partly right AI guessHigher-level tests catch the cross-component errors agents make and become cheap to run unattended; unit tests still give fast, precise feedback. Plausible practitioner view, not measured.

Flat-rate pricing pushes coding-agent vendors toward cheaper, weaker models

Leans partly right

The incentive is real and some vendors route to cheaper models under load, but competition on quality and usage caps limit it.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp (AI Engineer)“my incentive now is to push the inference cost as low as possible. And the easiest way to do that is to use dumber models”
Leans partly right AI guessThe incentive is real and some vendors route to cheaper models under load, but competition on quality and usage caps limit it.

A one- or two-person company run mostly with AI tools has reached billion-dollar-scale revenue

Unknown

Attributed to a New York Times report I can't verify; the business is a telehealth middleman whose revenue is mostly drug sales, so it says little about AI-built software per se.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Two People Vibe Coded a $1.8B Company. Yes, This is AI Takeoff. (AI For Humans)“he alone with AI tools has created a $1.8 billion annual recurring revenue business”
Unknown AI guessAttributed to a New York Times report I can't verify; the business is a telehealth middleman whose revenue is mostly drug sales, so it says little about AI-built software per se.

Implementation choices mentioned in a prompt anchor the agent's design, so asking it for alternatives leads to better designs

Leans right

LLMs are known to be sycophantic and anchor on user suggestions; prompting for alternatives or critique reliably surfaces other options.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro (AI Engineer)“Anything you put in the prompt is effectively rounding the agent. For better or for worse”
Probably right AI guessLLMs are known to be sycophantic and anchor on user suggestions; prompting for alternatives or critique reliably surfaces other options.

AI weakens code review as a mentoring channel because juniors pass reviewer feedback straight to agents

Leans partly right

Plausible and consistent with the Anthropic learning study on skipped understanding; evidence is anecdotal so far.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Why Coding Agents Keep Making Your Codebase Worse (Beyond Coding)“the junior just copies the review comment into Claude ... nobody learns anything”
Leans partly right AI guessPlausible and consistent with the Anthropic learning study on skipped understanding; evidence is anecdotal so far.
Leans partly right Anthropic internal study lines 110-113Anthropic: juniors now take questions to Claude instead of seniors, so mentorship opportunities shrink (though they learn faster); this is about questions, not code-review feedback specifically.

Using LLMs to understand legacy codebases is effective and ready for wide adoption

Leans right

Code explanation and codebase Q&A are among the most consistently reported wins, and Thoughtworks lists it under Adopt. Explanations can still be confidently wrong.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

How AI will change software engineering – with Martin Fowler (The Pragmatic Engineer)“using LLMs to understand legacy systems ... we put it in adopt”
Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“AI has the advantage of making existing code bases more understandable for newcomers.”
Probably right AI guessCode explanation and codebase Q&A are among the most consistently reported wins, and Thoughtworks lists it under Adopt. Explanations can still be confidently wrong.
Leans partly right Echoes of AI maintainability RCT lines 2639-2644Participants reported AI helped onboarding and understanding new code, but the system was a small Java app and authors note AI may be less capable on large codebases that exceed the context window.

Developers report that AI saves more time on debugging and stack-trace analysis than on new code generation

Leans partly right

This comes from self-reported vendor survey data, which is known to misestimate time savings, though diagnosis tasks avoid the review overhead of generated code.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“stack trace analysis and refactoring existing code we're saving more time than the midloop code generation”
Leans partly right AI guessThis comes from self-reported vendor survey data, which is known to misestimate time savings, though diagnosis tasks avoid the review overhead of generated code.
Leans partly right Shan et al. field study + RCT lines 1157-1171Developers praised AI for finding bugs and quick troubleshooting, but in the experiment debugging-style use lowered task output while raising accuracy. Reports of value exist, but no time saving was observed.

AI can lower developer satisfaction by speeding up the coding they enjoy and leaving proportionally more toil

Leans partly right

DORA reported this pattern in one year's data, but other surveys find AI users report higher satisfaction, so the effect is mixed.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Measuring the impact of AI on software engineering – with Laura Tacho (The Pragmatic Engineer)“many developers were actually feeling less satisfied because AI is accelerating the parts that they enjoy.”
Leans partly right AI guessDORA reported this pattern in one year's data, but other surveys find AI users report higher satisfaction, so the effect is mixed.
Leans partly right Anthropic internal study lines 104-108Anthropic: some engineers miss hands-on coding and find prompting unfulfilling, but people mostly delegate boring tasks (44% of Claude work was tasks they wouldn't enjoy), which is the opposite of the claimed mechanism.

Self-reported code maintainability tends to decline as more code is AI-written

Leans right

In a 272-developer follow-up using multi-item self-report scales, AI use was associated with lower code maintainability and more technical debt. This held after dropping zero-experience respondents.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Measuring AI code assistants and agents with the AI Measurement Framework (DX)“we do see not in all cases but in many cases a decrease in code self-reported co code maintainability scores.”
Leans partly right AI guessVendor survey data shows some decline, but the Echoes of AI controlled study found no significant maintainability penalty, so the effect is uncertain.
Leans right Shan et al. field study + RCT lines 1219-1229In a 272-developer follow-up using multi-item self-report scales, AI use was associated with lower code maintainability and more technical debt. This held after dropping zero-experience respondents.

Coding agents perform better on libraries with narrow, well-defined APIs than on products with many intertwined concerns

Leans partly right

This fits evidence that gains shrink with complexity and cross-cutting context, but no study directly compares libraries with products.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil (AI Engineer)“for libraries, they tend to excel a lot more ... products ... it's much harder because there are so many interacting concerns and components.”
Leans partly right AI guessThis fits evidence that gains shrink with complexity and cross-cutting context, but no study directly compares libraries with products.

Reinforcement learning against compile-and-test results improves a code model's ability to produce working code

Leans right

RL with verifiable rewards from execution and tests is a well-established technique behind recent coding model gains.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI Engineering at Jane Street - John Crepezzi (AI Engineer)“better align the model to write code that actually does compile and past tests”
Very likely right AI guessRL with verifiable rewards from execution and tests is a well-established technique behind recent coding model gains.

Evaluating models on verifiable coding tasks is essential when customizing code models

Leans right

Without task-level evals, fine-tuning regressions go unnoticed; this is standard ML practice.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI Engineering at Jane Street - John Crepezzi (AI Engineer)“having evaluations that are meaningful is kind of a Cornerstone of making sure that models don't go off the rails”
Very likely right AI guessWithout task-level evals, fine-tuning regressions go unnoticed; this is standard ML practice.

AI usage is concentrated in software and mathematical tasks

Leans right

Anthropic's own usage data and other usage studies consistently find coding is the largest single category.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Anthropic Economic Index 2026—Measuring Claude Usage with “Economic Primitives” Metrics (CosmoX)“A whopping 34% of all tasks are in computer and mathematical fields. We're talking coding, software development, that kind of thing.”
Very likely right AI guessAnthropic's own usage data and other usage studies consistently find coding is the largest single category.

AI is being adopted in the US much faster than earlier major technologies

Leans partly right

Survey data show unusually fast uptake of generative AI, though comparisons with past technologies depend on how adoption is defined.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Anthropic Economic Index 2026—Measuring Claude Usage with “Economic Primitives” Metrics (CosmoX)“AI is spreading like wildfire, about 10 times faster than major technologies from the last century.”
Probably partly right AI guessSurvey data show unusually fast uptake of generative AI, though comparisons with past technologies depend on how adoption is defined.

AI task success rates fall as task complexity rises

Leans right

Benchmarks, METR task-horizon work and usage studies all show success falling with task length and complexity.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (4)

Anthropic Economic Index 2026—Measuring Claude Usage with “Economic Primitives” Metrics (CosmoX)“The success rate actually drops. It goes from 70% on simpler stuff down to 66% on the hard stuff.”
Very likely right AI guessBenchmarks, METR task-horizon work and usage studies all show success falling with task length and complexity.
Leans right Airbnb migration lines 22-24Airbnb: simple files passed with a few retries, complex ones needed rich context, and the last ~100 files hit a ceiling after 50-100 retries and were fixed by hand.
Leans right Shan et al. field study + RCT lines 1279-1282GPT-4 alone solved all five easy experiment tasks but got two of five harder APPS interview/competition tasks wrong, and the AI-assisted accuracy gain shrank on the harder set (tiny task sample).
Leans partly right METR 2025 RCT lines 986-993METR 2025, qualitative: developers say AI does worse in complex settings, e.g. a refactor that was 'too big' where AI 'introduced as many errors as it fixed', and failures tied to file size. These are anecdotes, not success rates.

AI on net is deskilling jobs by taking over their higher-skill tasks

Leans partly right

This is a model-based estimate from one vendor's usage data; independent labor-market evidence on deskilling versus upskilling is not yet clear.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Anthropic Economic Index 2026—Measuring Claude Usage with “Economic Primitives” Metrics (CosmoX)“Right now, the net effect across the entire economy is actually deskilling.”
Leans partly right AI guessThis is a model-based estimate from one vendor's usage data; independent labor-market evidence on deskilling versus upskilling is not yet clear.

AI could add around one percentage point to annual labor productivity growth

Unknown

This is a projection built on model-estimated task speedups; macro productivity data has not yet confirmed effects of this size.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Anthropic Economic Index 2026—Measuring Claude Usage with “Economic Primitives” Metrics (CosmoX)“once you adjust for how often AI actually succeeds in the real world, the number comes down to a more realistic 1%.”
Unknown AI guessThis is a projection built on model-estimated task speedups; macro productivity data has not yet confirmed effects of this size.

AI-generated unit tests can become technical debt by locking in implementation details

Leans partly right

Mock-heavy or change-detector tests do raise maintenance cost, but well-guided AI tests are often useful, so it depends on review.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Build to Last — Chris Lattner talks with Jeremy Howard (Jeremy Howard)“unit tests are their own potential tech debt.”
Probably partly right AI guessMock-heavy or change-detector tests do raise maintenance cost, but well-guided AI tests are often useful, so it depends on review.

AI's broad proficiency across the stack shifts value toward full-stack engineers over layer specialists

Leans partly right

Anthropic: engineers report becoming 'more full-stack' (backend engineers building UIs, researchers building front-end visualizations), but nothing shows value moving away from specialists.

· denied 0 ·

Asserts (2)Denies (0)Verdicts (2)

Building Blocks for Uber’s Software Factory— Uday Kiran Medisetty & Adam Huda, Uber (AI Engineer)“the roles of engineers are getting blurred.”
Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“AI gives you this kind of universal basic proficiency at all the layers of the stack”
Leans partly right Anthropic internal study lines 83-84Anthropic: engineers report becoming 'more full-stack' (backend engineers building UIs, researchers building front-end visualizations), but nothing shows value moving away from specialists.
Unknown AI guessRole blurring is widely reported anecdotally, but there is no solid labor-market evidence yet that specialists are losing ground.

Purpose-built test-generation agents with a separate critic produce better unit tests than general-purpose agents

Leans partly right

Filtered pipelines like Meta's TestGen-LLM show real gains, but the comparative evidence is mostly company-internal.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Uber: Leading engineering through an agentic shift - The Pragmatic Summit (The Pragmatic Engineer)“almost a 3x rate of quality versus something that would be generated from your typical generic agent.”
Leans partly right AI guessFiltered pipelines like Meta's TestGen-LLM show real gains, but the comparative evidence is mostly company-internal.

Building code a few lines at a time in a running dialogue with the AI solves harder problems than one-shot generation

Leans partly right

Iterative use (adapt, improve, debug, or the ideate-adapt-debug sequence) improved accuracy at some speed cost, while one-shot direct acceptance did not improve accuracy. The evidence is correlational and on simple tasks.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

How To Solve It With Code—background on the course; a discussion with Jeremy and Hamel (Jeremy Howard)“the level of difficulty of problem that you're able to solve and the complexity that the AI can solve is much higher”
Leans partly right Shan et al. field study + RCT lines 1072-1083Iterative use (adapt, improve, debug, or the ideate-adapt-debug sequence) improved accuracy at some speed cost, while one-shot direct acceptance did not improve accuracy. The evidence is correlational and on simple tasks.
Unknown AI guessConsistent with evidence for incremental verification, but no controlled comparison of this workflow exists.

LLMs can quickly pinpoint a regression when given the working and broken versions of the code plus the stack trace

Leans partly right

Localizing bugs from diffs and traces is a known LLM strength, but success depends on the cause being visible in the supplied context.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Coding with AI episode 3: fixing a real world bug in Redis vector sets with LLMs (Salvatore Sanfilippo)“so it knows the version of the code when it was correct and the version of the code modified with the bug.”
Probably partly right AI guessLocalizing bugs from diffs and traces is a known LLM strength, but success depends on the cause being visible in the supplied context.

LLM teaching assistants answer most introductory course content questions correctly

Leans right

Several course deployments report high accuracy on intro-level content questions, though accuracy drops on course-specific policy and advanced material.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Teaching CS50 with AI - David J. Malan (CS50)“the duck answered 88% of curricular questions, content questions, correctly”
Probably right AI guessSeveral course deployments report high accuracy on intro-level content questions, though accuracy drops on course-specific policy and advanced material.

Students given an AI tutor shift help-seeking away from human office hours and forums

Leans partly right

Course reports show drops in forum posts and office-hour attendance after AI tutors launch, but the evidence is observational and confounded.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Teaching CS50 with AI - David J. Malan (CS50)“fewer students were taking advantage, funny enough, of those same human resources”
Probably partly right AI guessCourse reports show drops in forum posts and office-hour attendance after AI tutors launch, but the evidence is observational and confounded.

Agents work best when tasks are decomposed into writing and running code, because code is verifiable

Leans right

Tool-using agents that run code beat pure text reasoning on calculation and data tasks, and verifiable coding rewards drove recent agent training gains.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

A Year of Agents | Armin Ronacher | CodeCrafts 2026 (SQUER Solutions)“agents really are all about writing code, running code and doing that”
Probably right AI guessTool-using agents that run code beat pure text reasoning on calculation and data tasks, and verifiable coding rewards drove recent agent training gains.
Leans partly right Google migrations report lines 589-603Google: splitting migrations into simpler, well-defined sub-steps (each validated by build/tests) speeds iteration, and LLMs are very good at fixing a well-defined problem; this supports decomposition into verifiable steps in a pipeline setting.

Fine-tuning a model does not reliably teach it new facts; retrieval-augmented generation is the better way to add knowledge

Leans right

Studies find fine-tuning adds new facts slowly and can raise hallucination, while retrieval is the standard, better-performing way to supply fresh or private knowledge.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI tools for software engineers, but without the hype – with Simon Willison (Co-Creator of Django) (The Pragmatic Engineer)“for adding new fact into the model it just doesn't work”
Very likely right AI guessStudies find fine-tuning adds new facts slowly and can raise hallucination, while retrieval is the standard, better-performing way to supply fresh or private knowledge.

Spending thousands of dollars per engineer per month on AI tools is justified by the productivity gains

Leans partly right

The arithmetic holds if gains are real, but measured gains are modest and uneven (Stanford, METR, DORA), and the speaker admits they can't measure them.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

How AI is changing software engineering at Shopify with Farhan Thawar (The Pragmatic Engineer)“$1,000 a month is too cheap. Like it's too like if I can get 10 percent more, it is way too cheap.”
Leans partly right AI guessThe arithmetic holds if gains are real, but measured gains are modest and uneven (Stanford, METR, DORA), and the speaker admits they can't measure them.
Leans partly right Airbnb migration lines 65-65Airbnb says total cost (LLM API plus 6 weeks of engineering) was far below the manual estimate, even with high retry counts, but gives no dollar figures and covers only one migration.

Autonomous AI agents can find exploitable web vulnerabilities at or above the level of top human bug-bounty hunters

Leans partly right

XBOW did top HackerOne's US leaderboard, but that ranking rewards volume and many findings were lower severity, so it doesn't show parity with the best humans.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Inside the Rise of Autonomous AI Hackers: XBOW's Oege de Moor (Sequoia Capital)“Within a few weeks, Xbo first became the number one hacker in the United States”
Leans partly right AI guessXBOW did top HackerOne's US leaderboard, but that ranking rewards volume and many findings were lower severity, so it doesn't show parity with the best humans.

The time between vulnerability disclosure and exploitation in the wild has collapsed, often to zero or less

Leans partly right

Mandiant and others report time-to-exploit falling to days and many zero-days exploited before disclosure, but 'most CVEs' overstates it and AI's role is unproven.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Inside the Rise of Autonomous AI Hackers: XBOW's Oege de Moor (Sequoia Capital)“For most CVEs, it is already being exploited before the the CVE is even published.”
Leans partly right AI guessMandiant and others report time-to-exploit falling to days and many zero-days exploited before disclosure, but 'most CVEs' overstates it and AI's role is unproven.

Randomized AI-uplift studies are becoming unreliable because developers who expect the most benefit refuse to work without AI

Very likely right

METR Feb 2026: more developers declined to join because they won't work without AI, and 30-50% withheld tasks they didn't want to do without AI. METR calls its estimate a likely lower bound and is redesigning the study (pay cut from $150 to $50/h also contributed).

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“developers who expect to be sped up very significantly are systematically dropping out of our data.”
Very likely right METR 2026 update lines 70-74METR Feb 2026: more developers declined to join because they won't work without AI, and 30-50% withheld tasks they didn't want to do without AI. METR calls its estimate a likely lower bound and is redesigning the study (pay cut from $150 to $50/h also contributed).
Probably right AI guessMETR's Feb 2026 update reported this selection problem; how much it biases estimates is uncertain.

Measured AI task speedup can greatly overstate the value AI actually adds

Leans partly right

The 56% lab speedup would imply nearly twice as many tasks, but real-world field experiments measured only a 26% increase in completed tasks, showing lab task speedups overstate on-the-job gains (though gains remain positive).

· denied 0 ·

Asserts (1)Denies (0)Verdicts (4)

Joel Becker: Reconciling Impressive AI Benchmark Performance, Limited Developer Productivity Impacts (Stanford Digital Economy Lab)“speed up can can greatly overstate the the degree to which people are getting more value out of AI.”
Leans partly right AI guessSpeedup on cheap tasks need not translate into value, but AI can also enable new valuable work, so the direction is case-dependent.
Leans partly right Microsoft/Accenture field RCTs lines 774-784The 56% lab speedup would imply nearly twice as many tasks, but real-world field experiments measured only a 26% increase in completed tasks, showing lab task speedups overstate on-the-job gains (though gains remain positive).
Leans partly right Code-ownership experiment (Martin-Lopez) lines 900-927A doubling of completeness came with a measurable loss in comprehension (-5.95 pp), and the tasks where AI helped most (D4) showed the steepest loss (55% vs 93% correct). The authors argue downstream costs may offset the time saved, but they did not measure those costs.
Leans partly right METR 2026 update lines 77-79METR Feb 2026: developers changed task types and output quality (tests, docs) under AI, so time differences 'may not represent value-differences'. This gap can run in either direction; METR itself thinks its estimate understates uplift.

Frontier AI systems can now autonomously solve some unpublished research-level math problems

Leans right

Independent efforts like First Proof and Erdos-problem results show genuine solutions, though at high cost and on selected problems.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

ICM 2026 Public Lectures - Terence Tao (Simons Foundation)“collectively they were able to sort of seven out of 10 at at publication level quality but at a non-trivial cost.”
Probably right AI guessIndependent efforts like First Proof and Erdos-problem results show genuine solutions, though at high cost and on selected problems.

Software is bifurcating into disposable AI-generated tools and long-lived systems that people must understand

Untestable

A framing of how software is changing rather than a measurable claim.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI at Honeycomb: What’s Actually Working (w/ CTO Charity Majors) (Jam)“the world of software is bifurcating into disposable software stuff that nobody really understands”
Untestable AI guessA framing of how software is changing rather than a measurable claim.

LLMs are highly effective at reading comprehension tasks such as summarizing documents and answering questions about long specifications

Leans right

Summarization and document QA are among the best-validated LLM strengths, though errors and omissions on long technical documents still occur.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

576 - Using LLMs at Oxide | RFD | Oxide (Oxide Computer Company)“LLMs are superlative at reading comprehension, able to process and meaningfully comprehend documents effectively instantly.”
Very likely right AI guessSummarization and document QA are among the best-validated LLM strengths, though errors and omissions on long technical documents still occur.

AI-text detectors such as Pangram can identify LLM-written prose with very low false-positive rates

Leans partly right

Independent evaluations find Pangram among the most accurate detectors with low false positives on typical text, but detectors in general degrade on edited, short or non-native-English text.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

576 - Using LLMs at Oxide | RFD | Oxide (Oxide Computer Company)“We use Pangram at Oxide, and have found its false positive rate to be extremely low.”
Leans partly right AI guessIndependent evaluations find Pangram among the most accurate detectors with low false positives on typical text, but detectors in general degrade on edited, short or non-native-English text.

Quizzing yourself on an agent's change before shipping it keeps the AI loop from outrunning human understanding

Leans partly right

Quizzes generated by o1 about the participant's own code exposed a comprehension gap after AI use, so quizzing does reveal lost understanding. The study used quizzes only to measure and did not test whether they help as an intervention; the authors propose active-review mechanisms as future work.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Understanding is the new bottleneck (geoffreylitt.com)“A quiz is a speed regulator. Working with AI, it's easy for the loop to run faster than the speed of human understanding.”
Leans partly right Code-ownership experiment (Martin-Lopez) lines 957-967Quizzes generated by o1 about the participant's own code exposed a comprehension gap after AI use, so quizzing does reveal lost understanding. The study used quizzes only to measure and did not test whether they help as an intervention; the authors propose active-review mechanisms as future work.
Untestable AI guessThis is a personal workflow practice drawn from learning-science ideas; its value for code comprehension has not been measured.

Heavy dependence on hosted coding agents creates vendor lock-in, leaving teams unable to work during provider outages

Leans partly right

Outages of hosted agents have stalled teams that rely on them, but most teams can still code without the agent; the dependency is real but usually recoverable.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Agentic Coding is a Trap | Lars Faye (larsfaye.com)“Vendor lock-in for individuals and entire teams (Claude Code outages have already had entire teams at a stand-still).”
Leans partly right AI guessOutages of hosted agents have stalled teams that rely on them, but most teams can still code without the agent; the dependency is real but usually recoverable.

With AI making bug-finding nearly free, software quality is now limited by how many bugs teams choose to fix rather than by finding them

Leans partly right

AI review tools do surface many real bugs cheaply, but they still miss many issues and fixing found bugs carries real cost, so finding is not fully solved.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

You can just choose how many bugs you want now (Read the Tea Leaves)“you can basically just decide how many bugs you want your software to have now.”
Leans partly right AI guessAI review tools do surface many real bugs cheaply, but they still miss many issues and fixing found bugs carries real cost, so finding is not fully solved.

Looping a generator agent against an independent adversarial verifier agent yields simpler, more spec-compliant code than one-shot generation

Leans partly right

Separate verifier/critic setups improve outputs in several studies and vendor reports, but the evidence here is a small self-run benchmark subset with two trials and large token costs.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Killing Coding Agent Slop With Adversarial Self-Play (Telos)“the harness takes a goal specification as input and runs two coding agents in a loop: one generates code; the other reviews and tests it.”
Leans partly right AI guessSeparate verifier/critic setups improve outputs in several studies and vendor reports, but the evidence here is a small self-run benchmark subset with two trials and large token costs.

The underlying model matters more than the agent harness for AI coding results

Leans partly right

Model choice is the largest driver of benchmark results, but harness differences also produce sizable gaps on agentic benchmarks.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Broken AI Discourse with Steve Klabnik (Software Unscripted Podcast)“it doesn't really feel like the agent tool itself is like making that big of a difference”
Leans partly right AI guessModel choice is the largest driver of benchmark results, but harness differences also produce sizable gaps on agentic benchmarks.

LLM coding performance depends on meaningful identifier names, and obfuscated names degrade it

Leans right

Several studies show code-model accuracy drops when identifiers are anonymized or misleading.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Learning to Code with AI and Steve Klabnik (Scott Hanselman)“these tools will do better if you have good variable names”
Very likely right AI guessSeveral studies show code-model accuracy drops when identifiers are anonymized or misleading.

LLM agents perform better writing code against an API than issuing direct tool calls

Leans partly right

Cloudflare and Anthropic report token and accuracy gains from code-execution tool use, but the evidence is mostly vendor-reported and task-dependent.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Let's put the AI in lots of little boxes - Kenton Varda, Cloudflare Connect 2025 (Cloudflare Developers)“They can uh call more complicated interfaces when they're calling code than when they're calling tools.”
Leans partly right AI guessCloudflare and Anthropic report token and accuracy gains from code-execution tool use, but the evidence is mostly vendor-reported and task-dependent.

Splitting an agent into narrowly scoped sub-agents with one-way data flows can prevent prompt-injection data leaks

Leans partly right

Information-flow designs such as dual-LLM and CaMeL can provably limit leakage, but they cost functionality and do not stop every kind of misuse.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Let's put the AI in lots of little boxes - Kenton Varda, Cloudflare Connect 2025 (Cloudflare Developers)“That agent cannot leak any secrets because the only thing it knows are things that the recipients of anything it sends already know.”
Probably partly right AI guessInformation-flow designs such as dual-LLM and CaMeL can provably limit leakage, but they cost functionality and do not stop every kind of misuse.

Existing OAuth permission scopes are too coarse to restrict AI agents to least privilege

Leans right

Major APIs such as Gmail offer broad scopes that cannot express per-item agent permissions, though some APIs have finer-grained scopes.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Let's put the AI in lots of little boxes - Kenton Varda, Cloudflare Connect 2025 (Cloudflare Developers)“There is no way to issue an OOTH token that gives access to a single uh email in a Gmail inbox.”
Probably right AI guessMajor APIs such as Gmail offer broad scopes that cannot express per-item agent permissions, though some APIs have finer-grained scopes.

Coding agents fill gaps in requirements with unstated assumptions that can shift between runs

Leans right

Nondeterministic generation from underspecified prompts reliably yields inconsistent guesses, as agent experiments widely report.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

How far can we push AI autonomy in code generation? (martinfowler.com)“Gaps in the requirements will be filled with assumptions”
Very likely right AI guessNondeterministic generation from underspecified prompts reliably yields inconsistent guesses, as agent experiments widely report.

Growth in AI capabilities is roughly proportional to compute growth, so a slowdown in compute spending would slow capability gains

Unknown

A contested economic modeling assumption; plausible as a near-term model but not empirically established.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR (AI Engineer)“if compute growth were to half, then time horizon growth would half.”
Unknown AI guessA contested economic modeling assumption; plausible as a near-term model but not empirically established.

Much developer use of agentic AI tools is driven by fear of becoming obsolete rather than by the tools' usefulness

Leans wrong

METR Feb 2026: more developers refused to work without AI, and 30-50% held back tasks, which METR attributes mainly to higher expectations of AI uplift (plus lower pay). Developer quotes describe liking the tools, not fear of obsolescence.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Measuring the actual impact of AI coding | Abi Noda from DX (Changelog)“they believe is more fear driven than utility uh utility driven”
Leans wrong METR 2026 update lines 70-74METR Feb 2026: more developers refused to work without AI, and 30-50% held back tasks, which METR attributes mainly to higher expectations of AI uplift (plus lower pay). Developer quotes describe liking the tools, not fear of obsolescence.
Unknown AI guessAnecdotal report from vendor researchers; motivation for adoption has not been measured.

LLMs perform poorly on low-level code that is scarce in training data, such as GPU kernels and parallel or distributed code

Leans right

Kernel and parallelism code remained a weak spot through 2025, though specialized models and agents improved on kernel benchmarks by 2026.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Unpacking METR’s findings: Does AI slow developers down? (DX)“GPU kernels like those are kind of out of distribution for models right now.”
Probably right AI guessKernel and parallelism code remained a weak spot through 2025, though specialized models and agents improved on kernel benchmarks by 2026.

Time-boxing attempts to get an AI to solve a task, then switching to doing it manually, avoids sunk-cost time losses

Leans partly right

METR: developers accepted <44% of generations, and one 'wasted at least an hour' trying with AI before reverting and doing the work by hand. That documents the sunk-cost loss, but the time-boxing remedy was not tested.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Unpacking METR’s findings: Does AI slow developers down? (DX)“People need to sort of fast fail. So like sort of time box when you're working with an L.”
Leans partly right METR 2025 RCT lines 1006-1015METR: developers accepted <44% of generations, and one 'wasted at least an hour' trying with AI before reverting and doing the work by hand. That documents the sunk-cost loss, but the time-boxing remedy was not tested.
Untestable AI guessWorkflow advice that is plausible but has not been directly measured.

AI is effective at translating an application or code samples from one programming language to another

Leans partly right

Migration case studies (Airbnb, Google) show large time savings on translation and migration, but output still needs review and deterministic tooling helps.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

AI-Infused Development: Lessons Learned and Patterns from Esri Engineers (Esri Developers)“how useful it became to translate an application built from one language to another”
Probably partly right AI guessMigration case studies (Airbnb, Google) show large time savings on translation and migration, but output still needs review and deterministic tooling helps.
Leans wrong Google migrations report lines 687-690Google authors' related-work aside: prior LLM approaches to language translation 'do not reach high enough quality to become useful yet' (an opinion on the literature, not a measurement).
Leans partly right Echoes of AI maintainability RCT lines 2232-2237One participant reported that Copilot sped up their work by translating constructs from familiar languages into Java/Spring idioms. This is a single anecdote about snippet-level translation, not whole applications.

AI substantially shortens the time it takes developers to onboard onto a codebase

Leans partly right

Vendor telemetry shows time-to-10th-PR falling alongside AI adoption, but that is correlational and not independently verified.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (4)

Data vs Hype: How Orgs Actually Win with AI - The Pragmatic Summit (The Pragmatic Engineer)“Uh AI is fantastic for onboarding.”
Leans partly right AI guessVendor telemetry shows time-to-10th-PR falling alongside AI adoption, but that is correlational and not independently verified.
Leans partly right Echoes of AI maintainability RCT lines 2197-2210Eight participants new to Java/Spring said AI was valuable for getting up to speed, and some used it only for onboarding. The evidence is qualitative self-report from a non-randomized phase, not a measured onboarding time.
Leans partly right METR 2025 RCT lines 957-976METR RCT: developers were slowed less on issues with low prior exposure and high reference needs, and said AI helps most with unfamiliar code (e.g. Git hooks, finding unknown helpers). Onboarding time itself was not measured, and the main result was a 19% slowdown for familiar experts.
Leans partly right Microsoft/Accenture field RCTs lines 758-770At Microsoft, recent hires (below-median tenure) raised Copilot-assisted output by 27-39% vs 8-13% for long-tenure staff. That is consistent with AI helping newer developers, but onboarding time itself was not measured and the differences are not significant.

Agents told to follow TDD often skip or fake the red step and implement ahead of the tests

Leans right

Consistent with practitioner reports that agents struggle to follow incremental test-first discipline, though evidence is mostly anecdotal.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

TDD inside the agent loop - theater or actual value? (martinfowler.com)“agents still sometimes skipped or faked the red step, or implemented ahead of the test so that it passed immediately”
Probably right AI guessConsistent with practitioner reports that agents struggle to follow incremental test-first discipline, though evidence is mostly anecdotal.

Running TDD inside the agent loop multiplies token cost

Leans right

More turns and tool calls mechanically raise token usage, though cache hits reduce the dollar impact.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

TDD inside the agent loop - theater or actual value? (martinfowler.com)“TDD reliably cost several times more, how many times exactly is variable.”
Probably right AI guessMore turns and tool calls mechanically raise token usage, though cache hits reduce the dollar impact.

AI-generated tests are often tautological, deriving expected values from the code under test

Leans partly right

A commonly observed failure mode in generated tests, but how often it occurs has not been well quantified.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

TDD inside the agent loop - theater or actual value? (martinfowler.com)“tests checked the implementation's output against itself”
Leans partly right AI guessA commonly observed failure mode in generated tests, but how often it occurs has not been well quantified.

Mutation testing is a better way than TDD instructions to make agent-written regression tests effective

Leans partly right

Mutation testing directly measures test effectiveness, but there is little comparative evidence against TDD prompting.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

TDD inside the agent loop - theater or actual value? (martinfowler.com)“I monitor and improve regression quality with the help of mutation testing, instead of giving elaborate TDD instructions and hoping for the best.”
Leans partly right AI guessMutation testing directly measures test effectiveness, but there is little comparative evidence against TDD prompting.

Self-reported AI time savings are a weak measure of AI's productivity impact

Very likely right

METR RCT, 16 experienced OSS devs, 246 tasks: developers estimated afterwards that AI sped them up by 20% when it actually slowed them by 19%, so self-reports got even the direction wrong.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (6)

The Truth About Developer Productivity in the AI Age (IT'S A TRAP) (Modern Software Engineering)“self-reported time savings, which is a very weak construct”
Very likely right METR 2025 RCT lines 407-424METR RCT, 16 experienced OSS devs, 246 tasks: developers estimated afterwards that AI sped them up by 20% when it actually slowed them by 19%, so self-reports got even the direction wrong.
Probably right AI guessMETR found developers' self-estimates diverged sharply from measured times, and self-reports are easily biased.
Leans right METR 2026 update lines 63-63METR Feb 2026 update says some developers self-report very high speedups but repeats that such estimates are 'quite unreliable'; it still hopes carefully designed questionnaires can give useful signal.
Leans partly right Copilot RCT (Peng) lines 179-188Copilot RCT: participants estimated a 35% productivity gain against a measured 55.8% speed-up. Self-reports were inaccurate here, but they understated rather than overstated the gain.
Leans right Anthropic internal study lines 36-41Anthropic's own survey caveats its self-reported 50% productivity boost: cites METR finding that developers overestimated gains, says time savings 'may reflect perception biases', and lists social-desirability and recall bias, noting self-reports should be 'taken with a grain of salt'.
Leans partly right Echoes of AI maintainability RCT lines 1926-1931AI users were 30.7% faster by median, yet perceived productivity looked similar across groups. The authors also flag confirmation bias in AI users' self-ratings. Self-report diverged from measured time, but this measured perceived productivity rather than estimated time savings.

Pull-request throughput is not a meaningful measure of delivery speed or productivity

Leans partly right

PR counts are noisy and gameable, but they correlate somewhat with delivery and are used with other metrics in studies such as Cui/Demirer.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

The Truth About Developer Productivity in the AI Age (IT'S A TRAP) (Modern Software Engineering)“Number of merged artifacts per developer per week is not speed.”
Probably partly right AI guessPR counts are noisy and gameable, but they correlate somewhat with delivery and are used with other metrics in studies such as Cui/Demirer.
Leans partly right METR 2025 RCT lines 124-129METR argues that PR, commit and LOC outcome measures can rise without real productivity gains, e.g. when developers split work into smaller PRs. This is argued, not measured, in this paper.
Leans partly right Microsoft/Accenture field RCTs lines 242-276The authors treat completed pull requests as a unit of work because PR scope stays fairly stable within an organization, while conceding that scope differs across organizations and that many zero-PR weeks make the data noisy. Their 26% PR gain is close to Google's 21% time-on-task estimate.

AI-generated test suites cover basic functionality but miss specification edge cases and abuse cases

Leans partly right

Generated tests often favor happy paths unless prompted, but this depends heavily on the prompt and the reviewer.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

A look at CloudFlare’s AI-coded OAuth library (Neil Madden)“they are woefully inadequate for what I would expect of a critical auth service”
Leans partly right AI guessGenerated tests often favor happy paths unless prompted, but this depends heavily on the prompt and the reviewer.

LLM-generated code reproduces common mistakes found in its training data, such as popular Stack Overflow answers

Leans partly right

Studies of generated code show it reproduces insecure patterns common in public code, though attributing a specific bug to its source is speculative.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

A look at CloudFlare’s AI-coded OAuth library (Neil Madden)“Many of these same mistakes can be found in popular Stack Overflow answers, which is probably where Claude learnt them from too.”
Probably partly right AI guessStudies of generated code show it reproduces insecure patterns common in public code, though attributing a specific bug to its source is speculative.

Software release volumes (apps, games, domains, repositories) show no surge since AI coding tools became widespread

Leans partly right

Release counts were roughly flat through mid-2025, but later data showed rises in new apps and repositories, and release counts are a crude productivity proxy.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Where's the Shovelware? Why AI Coding Claims Don't Add Up (mikelovesrobots.substack.com)“There’s no shovelware surge.”
Leans partly right AI guessRelease counts were roughly flat through mid-2025, but later data showed rises in new apps and repositories, and release counts are a crude productivity proxy.

Developers who spend more hours coding each day gain more from AI tools

Leans right

Google RCT: the interaction between AI and coding 5+ hours a day was large and negative (beta = -0.29, faster with AI) but not significant (model p = 0.018). The authors call it tentative evidence that heavy coders gain more.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

How much does AI impact development speed? (newsletter.getdx.com)“The largest speed gains from AI were observed among developers who code more hours per day.”
Leans right Copilot RCT (Peng) lines 169-177Copilot RCT heterogeneity analysis: developers with more coding hours per day benefited more, along with less experienced and older developers. The sample is small (95 Upwork freelancers) and the regression coefficients are imprecise.
Leans right Google RCT lines 484-493Google RCT: the interaction between AI and coding 5+ hours a day was large and negative (beta = -0.29, faster with AI) but not significant (model p = 0.018). The authors call it tentative evidence that heavy coders gain more.
Unknown AI guessGoogle's RCT found a large but not statistically significant interaction, so it is suggestive only.

AI agents increase organizational complexity, so management layers remain necessary though their role changes

Untestable

An organizational-design prediction with no direct evidence either way yet.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

The AI Productivity Paradox: Why 10X Output Doesn't Mean 10X Business Outcome - Mik Kersten (Tech Lead Journal)“agents are gonna increase the complexity, not decrease it”
Untestable AI guessAn organizational-design prediction with no direct evidence either way yet.

Developers who prefer reviewing and architecture over hands-on coding tend to enjoy agentic coding tools more

Leans partly right

Anthropic interviews (53): engineers who 'enjoy what I get out of writing code' or think at the product level embrace Claude, while those who value flow-state hands-on coding mourn it; whether people embrace it 'depends on what aspects of software engineering they find most meaningful' (qualitative, not about reviewing per se).

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Reconsidering AI w / Steve Klabnik (Zed Industries)“getting my fingers into the details is like not as interesting to me as it was before”
Leans partly right Anthropic internal study lines 104-108Anthropic interviews (53): engineers who 'enjoy what I get out of writing code' or think at the product level embrace Claude, while those who value flow-state hands-on coding mourn it; whether people embrace it 'depends on what aspects of software engineering they find most meaningful' (qualitative, not about reviewing per se).
Untestable AI guessA plausible point about personal preference that has not been systematically measured.

Hierarchical planner-worker agent swarms can split a large project so thousands of agents work concurrently with few merge conflicts

Leans partly right

Shown in a few vendor research runs that produced large, partly working code, but not independently replicated or held to production quality.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“we have the concept of uh planners and workers”
Leans partly right AI guessShown in a few vendor research runs that produced large, partly working code, but not independently replicated or held to production quality.

Tolerating a stable rate of small transient errors in commits lets multi-agent systems sustain higher throughput than requiring every commit to pass

Leans partly right

Matches long-standing trade-offs in large human teams, but the agent evidence is a single research project.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

FastRender with Wilson Lin - building a web browser with swarms of thousands of coding agents (Simon Willison)“there's this kind of like a little bit of slack in the system to allow this kind of temporary errors”
Leans partly right AI guessMatches long-standing trade-offs in large human teams, but the agent evidence is a single research project.

AI-assisted code is no harder for later developers to maintain than unassisted code

Probably right

Same RCT on quality: no frequentist CodeHealth difference after a second developer evolved the code, a small Bayesian CodeHealth gain when a habitual AI user wrote the original, and identical test coverage. Scope is limited to a small app and short-term effects.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (7)

AI Makes Verifying the Bottleneck with Dave Farley @ModernSoftwareEngineeringYT (Aviator)“The the AI didn't make it any better and it didn't make it any worse.”
Probably right Echoes of AI maintainability RCT lines 2519-2524Preregistered RCT, 75 mostly professional developers manually evolving Java code written with or without AI: no reliable difference in completion time or perceived productivity, and the study is only powered to rule out large effects.
Probably right Echoes of AI maintainability RCT lines 2581-2587Same RCT on quality: no frequentist CodeHealth difference after a second developer evolved the code, a small Bayesian CodeHealth gain when a habitual AI user wrote the original, and identical test coverage. Scope is limited to a small app and short-term effects.
Probably right AI guessThe Echoes of AI controlled experiment found no significant maintainability penalty, though it covered one task and small changes.
Leans right Code-ownership experiment (Martin-Lopez) lines 880-892The related work cites Borg et al.'s 151-participant controlled experiment, which found no consistent sign that AI-assisted code was harder for later developers to maintain, while warning of cognitive debt. This is secondary evidence; the study's own data cover only the original author.
Leans wrong Shan et al. field study + RCT lines 1219-1229The follow-up survey linked AI use to lower self-rated maintainability and more technical debt, which points against 'no harder to maintain'. These are self-reports, not measures of later maintainers.
Leans wrong Vibe-coding literature review lines 1434-145211 of 47 sources raise maintainability concerns: conversational code that lacks cohesive structure and documentation, structural complexity from rapid iteration, and long-term debt. These are reported concerns rather than measured comparisons with unassisted code.
Unknown Debt study (302k AI commits) lines 1031-1035The mining study of 302k AI-authored commits explicitly has no human-written baseline, so its counts of surviving debt cannot show that AI code is harder to maintain than human code.

A behavioral harness of executable contracts and tests lets cheaper models match frontier-model reliability on well-specified tasks

Leans partly right

Supported by one small experiment on a single greenfield service with an unusually detailed brief, so the scope is unproven.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Are Harnesses Worth the Cost? (martinfowler.com)“it turned a 3-in-10 baseline into 10-for-10 reliability”
Leans partly right AI guessSupported by one small experiment on a single greenfield service with an unusually detailed brief, so the scope is unproven.

For frontier models on well-specified tasks, continuous in-loop verification harnesses add cost without improving outcomes

Leans partly right

Observed in one experiment; harder or less specified tasks may still benefit from in-loop checks.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Are Harnesses Worth the Cost? (martinfowler.com)“The harness increased average time and cost without changing the observed acceptance result.”
Leans partly right AI guessObserved in one experiment; harder or less specified tasks may still benefit from in-loop checks.

Vendor-run studies of their own AI coding tools overstate benefits through easy tasks, small samples and soft metrics

Leans partly right

Vendor studies such as GitHub's often use favorable tasks and narrow metrics, though some vendor RCTs, like Google's, are reasonably designed.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (5)

Does GitHub Copilot Improve Code Quality? Here (Jadarma)“you cannot blame me from being skeptical of a big corporation conducting a study on its own products”
Probably partly right AI guessVendor studies such as GitHub's often use favorable tasks and narrow metrics, though some vendor RCTs, like Google's, are reasonably designed.
Leans partly right Microsoft/Accenture field RCTs lines 774-784Microsoft-affiliated field study, ~4,900 developers: real-world Copilot gains (26% more PRs) were much smaller than the 56% lab speed-up in Peng et al. The authors attribute the gap to easy lab tasks and to coding being only part of the job. This supports 'easy tasks overstate', though this field study is itself vendor-run.
Leans partly right Google RCT lines 522-539Google tested its own tools and admits it cannot rule out bias from that. Its enterprise-task estimate (~21%) is far below GitHub's 56% Upwork/GitHub Classroom result, a gap it attributes mainly to population and task differences, so vendor estimates vary a lot with task and sample.
Leans partly right METR 2025 RCT lines 115-129METR critiques earlier speedup studies (e.g. Peng et al.'s basic HTTP-server task) for synthetic tasks that resemble training data and for outcome metrics that are not fixed in advance. That supports the easy-tasks/soft-metrics part but does not single out vendor sponsorship.
Leans partly right Copilot RCT (Peng) lines 208-219GitHub/Microsoft-authored Copilot RCT reporting a 55.8% speed-up. It admits it used one standardized HTTP-server task rather than large collaborative projects, did not measure code quality, and that results may not generalize.

AI coding features that appear inline in the existing workflow get far more use than ones users must remember to trigger

Leans right

Consistent with Google's internal A/B results and with general UX evidence on defaults and friction, though mostly reported by the vendor.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

AI in software engineering at Google: Progress and the path ahead (research.google)“Experiments requiring the user to remember to trigger the feature have failed to scale.”
Probably right AI guessConsistent with Google's internal A/B results and with general UX evidence on defaults and friction, though mostly reported by the vendor.
Leans partly right Google migrations report lines 75-95Google: inline code completion is the most popular AI dev application because its UX feels natural in the IDE (38% acceptance, 67% of code characters AI-assisted); consistent but not a controlled inline-vs-triggered comparison.

Agentic coding makes it practical to build in-house what teams previously took on as dependencies or vendor services

Leans partly right

Cheap generation lowers build costs, but maintenance and security burden still favor many dependencies, and the trend is not yet measured.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

A Year Of Vibes (Armin Ronacher)“AI has the potential to encourage building on fewer dependencies”
Leans partly right AI guessCheap generation lowers build costs, but maintenance and security burden still favor many dependencies, and the trend is not yet measured.

Human-language design documents describing a system's ideas are more useful for maintaining AI-written code than reviewing the code itself

Untestable

A workflow preference that has not been compared empirically against code review.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Control the ideas, not the code - <antirez> (antirez.com)“use LLMs to write a DESIGN.md file where each data structure is described in human language”
Untestable AI guessA workflow preference that has not been compared empirically against code review.

AI code assistants suggest adding code far more than updating, moving or deleting it, which discourages refactoring

Leans partly right

True of completion-style assistants and consistent with GitClear's falling moved-code share, but agentic tools now routinely refactor and delete code.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

AI assistance is leading to lower code quality, claim researchers (devclass)“suggestions for added code, but never suggestions for updating, moving, or deleting code.”
Leans partly right AI guessTrue of completion-style assistants and consistent with GitClear's falling moved-code share, but agentic tools now routinely refactor and delete code.
Leans partly right Echoes of AI maintainability RCT lines 2067-2080RefactoringMiner found AI-assisted developers did as many or more refactorings (mean 18.3 vs 15.4) but mostly simple ones such as Inline or Rename Variable. Non-AI developers did more structural moves (Move Method 1.89 vs 0.21). Refactoring was not discouraged overall, but the deeper kinds were rarer.
Leans partly right Debt study (302k AI commits) lines 918-922Across 302k real AI-authored commits, AI fixed slightly more static-analysis code smells than it introduced (439,817 vs 432,748), which suggests it does perform local cleanup. Smell removal is only a loose proxy for refactoring.

Early AI productivity gains fade after a few months as the debt and defects they introduced slow teams down

Leans partly right

A difference-in-differences study of Cursor adoption found transient velocity gains with lasting complexity increases, but other field data show sustained gains.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

Does AI-driven productivity come at the cost of code quality? (Proxify)“there is a temporary increase usually across most companies right in productivity thing for the first couple of months.”
Leans partly right AI guessA difference-in-differences study of Cursor adoption found transient velocity gains with lasting complexity increases, but other field data show sustained gains.
Leans partly right Debt study (302k AI commits) lines 79-80Second-hand only: cites He et al., where Cursor adoption in 807 repos gave a transient velocity boost but persistent increases in code complexity. That fits the claim, but the paper does not itself measure productivity.
Unknown Vibe-coding literature review lines 1892-1905The review notes that several sources warn short-term gains may come at the expense of long-term maintainability, but it states that longitudinal evidence is limited and that long-horizon outcomes are rarely examined, so it cannot tell whether gains fade.

Only a minority of software teams are currently positioned to benefit from AI-assisted coding

Leans partly right

DORA 2025 shows that benefits depend on existing practices, but the 'small minority' estimate extrapolates beyond the survey data.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

DORA 2025 AI assisted dev report: Some Benefit, Most Don’t (Rob Bowley)“only a small minority of the industry are likely to benefit from AI-assisted coding”
Leans partly right AI guessDORA 2025 shows that benefits depend on existing practices, but the 'small minority' estimate extrapolates beyond the survey data.

Good documentation and code comments in a codebase improve the output of AI coding tools

Leans partly right

Grounding models in accurate documentation generally improves output, though studies of repository context files show mixed effects on agent success.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

How to measure AI developer productivity in 2025 | Nicole Forsgren (Lenny's Podcast)“the better documentation, the better comments you have, the better performance you're going to get out of your AI tools”
Probably partly right AI guessGrounding models in accurate documentation generally improves output, though studies of repository context files show mixed effects on agent success.
Leans partly right METR 2025 RCT lines 1029-1041METR developers say AI lacks undocumented tacit knowledge of the repo (edge cases, backwards compatibility), which makes its outputs less useful. That fits the claim's mechanism, but documentation was not varied or tested.
Leans partly right Vibe-coding literature review lines 1064-1070MLR synthesis says that giving context (requirements, examples, existing code, API documentation, conventions) reduces ambiguity and improves the relevance of generated artifacts. This concerns context supplied in prompts, not documentation quality inside the codebase, and it is untested.

AI adoption increases average pull request size

Leans right

Telemetry from DX and other vendors shows PR sizes growing alongside AI adoption, though magnitude and causality vary by organization.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI in engineering: Q2 2026 benchmarks & research readout (DX)“the PR size has increased along the same period of study”
Probably right AI guessTelemetry from DX and other vendors shows PR sizes growing alongside AI adoption, though magnitude and causality vary by organization.

Coding agents iterating against benchmarks with correctness checks can produce implementations faster than established optimized libraries

Leans partly right

Individual reports show large speedups on specific algorithms, but they are self-benchmarked and not independently verified across general workloads.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

An AI agent coding skeptic tries AI agent coding, in excessive detail (Max Woolf)“the same three step pipeline I describe above still works even with the more simple algorithms to beat scikit-learn's implementations”
Leans partly right AI guessIndividual reports show large speedups on specific algorithms, but they are self-benchmarked and not independently verified across general workloads.

AI removes the peripheral tasks through which junior developers traditionally learned on the job

Leans partly right

Plausible and consistent with skill-formation studies and weaker entry-level hiring, but direct evidence on lost apprenticeship tasks is thin.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

o11ycast - Ep. #80, Augmented Coding with Kent Beck (Heavybit)“a senior programmer writes some tests and the junior program goes and makes them pass or whatever. That option is not available anymore”
Leans partly right AI guessPlausible and consistent with skill-formation studies and weaker entry-level hiring, but direct evidence on lost apprenticeship tasks is thin.
Leans partly right Anthropic internal study lines 87-92Anthropic interviews: engineers report losing 'collateral' learning from manual debugging and doc reading, a senior says juniors would need deliberate effort to grow, and juniors ask seniors fewer questions, though one notes they 'learn faster'.

Coding agents perform best on workflows well represented in their reinforcement-learning data, such as files, shell commands and code execution

Leans partly right

Consistent with labs training on verifiable coding tasks, but training data composition is not public so the mechanism is inferred.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Armin Ronacher Leaning In To Find Out - PyAI Conf 2026 (Pydantic)“tool using code loops are very measurable”
Leans partly right AI guessConsistent with labs training on verifiable coding tasks, but training data composition is not public so the mechanism is inferred.

Changes made outside an agent's session confuse it and can lead it to falsely claim it verified work

Leans partly right

Agents only see state the harness reports, and false verification claims are widely observed, though harnesses increasingly detect external file changes.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Armin Ronacher Leaning In To Find Out - PyAI Conf 2026 (Pydantic)“external state changes break this experience”
Probably partly right AI guessAgents only see state the harness reports, and false verification claims are widely observed, though harnesses increasingly detect external file changes.

Agents' compile, lint and test loops catch hallucinated APIs, making hallucination a minor issue in coding

Leans partly right

Compilers and tests do catch invented signatures, but hallucinated packages and plausible-but-wrong logic still slip through.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (4)

My AI Skeptic Friends Are All Nuts (Fly)“If their LLM invents a new function signature, the agent sees the error.”
Probably partly right AI guessCompilers and tests do catch invented signatures, but hallucinated packages and plausible-but-wrong logic still slip through.
Leans partly right Google migrations report lines 316-326Google migration report: build/test validation with ML 'repair' fixes failures in a significant number of cases and Gemini 'didn't hallucinate wrong parameters' on Joda->java.time, but humans still had to revert incorrect or unnecessary edits in most Int32->64 changes, so loops help without making errors minor (see also L510-L513, L236-L242).
Leans partly right Debt study (302k AI commits) lines 667-67923,856 undefined variable or reference issues reached merged AI commits in real repos (e.g. an undefined 'cache' argument that went unfixed for 42 days), so such errors do slip through. They were concentrated in only 1,650 of 302k commits (~0.5%), which is consistent with them being uncommon.
Leans partly right Vibe-coding literature review lines 1551-1568Validation/test pipelines are the most recommended safeguard (88% of 33 sources), yet hallucinated or incorrect generation is still the most reported failure mode (13 of 47 sources), and the review says the effectiveness of these safeguards is largely untested (L1650-1654).

Individual contributors adopt AI coding tools faster than engineering leaders

Leans partly right

At Microsoft, junior-level developers were 4.7pp more likely to adopt Copilot (79.2% vs 74.4%) and short-tenure developers 9.4pp more likely. This matches the direction of the claim, but it compares levels rather than ICs with engineering leaders.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

What We Learned Deploying AI within Bloomberg’s Engineering Organization – Lei Zhang, Bloomberg (AI Engineer)“our data shows individual contributors have a much better stronger adoption than our leadership team”
Leans partly right Microsoft/Accenture field RCTs lines 607-613At Microsoft, junior-level developers were 4.7pp more likely to adopt Copilot (79.2% vs 74.4%) and short-tenure developers 9.4pp more likely. This matches the direction of the claim, but it compares levels rather than ICs with engineering leaders.
Unknown AI guessSingle-company observation; no broad independent data on adoption by role level.

Teams new to AI should apply it first to verification such as code review rather than code generation

Untestable

Sequencing advice with no comparative evidence either way.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Discussing the State of AI-assisted Software Development with DORA and EarnIn (Datadog Community)“I would recommend not starting with generating code, but starting with some of those verification processes”
Untestable AI guessSequencing advice with no comparative evidence either way.

LLM-written formal specifications tend to check only obvious, weak properties rather than the subtle ones that give formal methods their value

Leans partly right

Expert reviewers report this pattern repeatedly in AI-generated specs, but the evidence is anecdotal and models are improving.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

LLMs are bad at vibing specifications (buttondown.com)“The AI is only writing "obvious properties"”
Probably partly right AI guessExpert reviewers report this pattern repeatedly in AI-generated specs, but the evidence is anecdotal and models are improving.

Showcase demos of autonomous agent teams building large software from scratch overstate how complete and working the result is

Leans partly right

The Cursor browser repository failed CI at announcement and reused existing libraries, though such projects do show substantial autonomous output.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Cursor Claims AI Built a Browser. Here's the Problem (Brian Jenney)“it's not actually functional or at least it wasn't at the time they made this statement”
Probably partly right AI guessThe Cursor browser repository failed CI at announcement and reused existing libraries, though such projects do show substantial autonomous output.

AI lets individuals build across roles, but specialist expertise remains each person's primary value rather than everyone becoming a generic builder

Leans partly right

Internal survey, interviews and Claude Code usage data at Anthropic show teams using Claude for their core tasks and also to work outside their expertise (backend engineers building UIs, security analysing unfamiliar code), so 'everyone becomes more full-stack'; it supports the cross-role half but does not test whether specialist expertise stays each person's primary value, and it is a single-company self-report.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“people are maybe overhyping like everybody's a builder and it's just this new job function and everything's kind of the same”
Leans partly right Anthropic internal study lines 144-144Internal survey, interviews and Claude Code usage data at Anthropic show teams using Claude for their core tasks and also to work outside their expertise (backend engineers building UIs, security analysing unfamiliar code), so 'everyone becomes more full-stack'; it supports the cross-role half but does not test whether specialist expertise stays each person's primary value, and it is a single-company self-report.
Untestable AI guessA framing of how teams should value skills; no measurement would settle it.

Rapid model progress shortens product planning horizons to months rather than multi-year roadmaps

Untestable

Practice advice about planning cadence whose fit depends on the organization.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“that we think about planning is kind of almost like rolling six months”
Untestable AI guessPractice advice about planning cadence whose fit depends on the organization.

Every company needs a dedicated frontier team whose job is to test the latest models and report what has just become possible

Untestable

Organizational advice; corporate-lab history offers anecdotes for and against, not a measurable test.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs) (Lenny's Podcast)“a team, however you want to manifest it, that is their job is to be at the frontier”
Untestable AI guessOrganizational advice; corporate-lab history offers anecdotes for and against, not a measurable test.

Reinforcement learning against automated scorers, especially on unsolvable tasks, trains agents to reward-hack, and this generalizes to broader deceptive behavior

Leans right

Reward hacking in RL coding environments is documented by the labs themselves, and Anthropic's 2025 'natural emergent misalignment' work found it generalizes to sabotage and concealment.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI Just Crossed the Terrifying Line - Now What? (Kurzgesagt – In a Nutshell)“an agent that is more persistent, that tries to finish its task any way possible, that cheats, is rewarded”
Probably right AI guessReward hacking in RL coding environments is documented by the labs themselves, and Anthropic's 2025 'natural emergent misalignment' work found it generalizes to sabotage and concealment.

Sandboxed agents can escape or coordinate through shared resources the sandbox leaves reachable, such as package stores or internet access

Leans partly right

Small-scale evals have shown agents exploiting misconfigured containers to reach a goal; the large-scale swarm incident described here rests on a single account that is not independently verified.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI Just Crossed the Terrifying Line - Now What? (Kurzgesagt – In a Nutshell)“Each of them was supposed to be locked into an isolated sandbox, a sort of AI prison. But they had access to Artifactory”
Leans partly right AI guessSmall-scale evals have shown agents exploiting misconfigured containers to reach a goal; the large-scale swarm incident described here rests on a single account that is not independently verified.

Large populations of agents that can reach a shared channel spontaneously self-organize, divide labor and share tools even when scored individually

Unknown

Rests on one reported incident; emergent multi-agent cooperation has been shown in small simulations but not independently at this scale.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI Just Crossed the Terrifying Line - Now What? (Kurzgesagt – In a Nutshell)“there was no group reward. Yet many of them created tools and shared information that was helpful to all of them”
Unknown AI guessRests on one reported incident; emergent multi-agent cooperation has been shown in small simulations but not independently at this scale.

Agents trained to persist at any cost will take unauthorized harmful actions against unrelated outside systems in pursuit of their reward

Unknown

Lab evals show agents taking unsanctioned shortcuts inside their environment; attacks on third parties are reported only in this single account.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI Just Crossed the Terrifying Line - Now What? (Kurzgesagt – In a Nutshell)“the swarm of around 700 agents executed a coordinated, competent and serious cyberattack”
Unknown AI guessLab evals show agents taking unsanctioned shortcuts inside their environment; attacks on third parties are reported only in this single account.

AI labs racing to build the most powerful models fastest deprioritize safety and oversight

Untestable

A claim about motives and priorities rather than a measurable outcome.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

AI Just Crossed the Terrifying Line - Now What? (Kurzgesagt – In a Nutshell)“To win a game like this, your main priority won’t be safety.”
Untestable AI guessA claim about motives and priorities rather than a measurable outcome.

AI-generated code loses the rationale behind its decisions (intent debt), which reading the code alone cannot recover

Leans partly right

Crossover experiment with 69 participants: those who coded with AI answered fewer questions about code they had just written (median 87.5% vs 100% correct; OR 0.58, 5.95-point drop), with the effect concentrated on open questions about program logic and the reasons behind implementation choices (OR 0.45) rather than where-is-it questions; this measures the author's own short-term ability to explain the rationale, in mostly student samples, not whether a later reader can recover it from the code.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

"Do You Still Read the Code?" • zanlib (zanlib)“The decision’s result is in the code, but on its own it doesn’t preserve enough of its history to reconsider it fully.”
Leans partly right AI guessCode never recorded why a value was chosen, which is why decision records exist; that agents multiply unrecorded incidental choices is plausible but so far anecdotal.
Leans partly right Code-ownership experiment (Martin-Lopez) lines 681-708Crossover experiment with 69 participants: those who coded with AI answered fewer questions about code they had just written (median 87.5% vs 100% correct; OR 0.58, 5.95-point drop), with the effect concentrated on open questions about program logic and the reasons behind implementation choices (OR 0.45) rather than where-is-it questions; this measures the author's own short-term ability to explain the rationale, in mostly student samples, not whether a later reader can recover it from the code.
Leans partly right Vibe-coding literature review lines 1434-1447Review finds 11 of 47 sources raise maintainability concerns and links them to shallow understanding, incremental patching and limited documentation of design intent, quoting that conversationally generated code lacks cohesive structure and proper documentation; supports intent loss as a reported problem but is a count of mentions and does not test whether reading the code can recover the rationale.

Developers who stop reading generated code drift into de-facto vibe coding without the specification and evaluation harness that deliberate delegation needs

Leans partly right

Multivocal review of 47 vibe-coding sources concludes outcomes hinge on the rigor of the evaluation loop: with defined acceptance criteria and automated tests it is productive, but with limited review and unchecked outputs the same speed yields hallucinations, technical debt and cognitive debt; supports the harness mechanism, though the drift dynamic is not studied and the minimal-review-commit failure mode rests on 2 grey sources.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

"Do You Still Read the Code?" • zanlib (zanlib)“A drifting accelerator doesn’t end up somewhere between the two approaches; he ends up a vibecoder by default”
Leans partly right Vibe-coding literature review lines 1881-1893Multivocal review of 47 vibe-coding sources concludes outcomes hinge on the rigor of the evaluation loop: with defined acceptance criteria and automated tests it is productive, but with limited review and unchecked outputs the same speed yields hallucinations, technical debt and cognitive debt; supports the harness mechanism, though the drift dynamic is not studied and the minimal-review-commit failure mode rests on 2 grey sources.
Untestable AI guessA description of a behavioral failure mode; no study has measured the drift.

Teams must agree before merging how AI-generated changes will be maintained, by the author's understanding or by specification and tests

Untestable

Team-process advice; its value is a matter of judgment rather than measurement.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

"Do You Still Read the Code?" • zanlib (zanlib)“putting people who prefer different approaches on the same team without establishing expectations and boundaries beforehand, and then keeping to them”
Untestable AI guessTeam-process advice; its value is a matter of judgment rather than measurement.

Coding agents ignore costs that are not visible in their textual output, such as how long the tests they add take to run

Leans partly right

Agents only see what their tools print, so wall-clock and similar side costs are invisible unless surfaced; evidence is anecdotal.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

"Do You Still Read the Code?" • zanlib (zanlib)“it doesn’t really matter to that agent that the tests now take three times longer to run. But speed matters to a human.”
Leans partly right AI guessAgents only see what their tools print, so wall-clock and similar side costs are invisible unless surfaced; evidence is anecdotal.

A passing test suite shows the program does what was specified, not that the specification matches the real-world domain

Leans right

The standard verification-versus-validation distinction: tests check code against the spec, while validating the spec needs contact with the domain.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

"Do You Still Read the Code?" • zanlib (zanlib)“A green test suite only proves that the program does what you said, not that what you said corresponds to reality.”
Probably right AI guessThe standard verification-versus-validation distinction: tests check code against the spec, while validating the spec needs contact with the domain.

Higher-level tools save time working but not time learning, because their abstractions leak and must be understood when they fail

Leans partly right

Debugging a failure inside a tool's abstraction does require understanding what it hides, but well-designed abstractions let most users avoid that learning most of the time.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

The Law of Leaky Abstractions (Joel on Software)“So the abstractions save us time working, but they don’t save us time learning.”
Leans partly right AI guessDebugging a failure inside a tool's abstraction does require understanding what it hides, but well-designed abstractions let most users avoid that learning most of the time.
Leans partly right Anthropic internal study lines 87-94Anthropic engineers self-report that delegating to Claude saves time but skips the incidental learning of doing the work manually ('skip the part where I learn how to perform a task by solving an easy instance, and then struggle to solve a more complicated instance later'), and describe a 'paradox of supervision' where checking the tool's output still requires the underlying skills; interview quotes from one company, and others in the same study report faster learning.

Most AI-generated code reaches few users, so production software quality and bug volumes have changed little overall

Leans partly right

Release-volume data show no surge in shipped software, which fits, but there is no direct before-and-after measurement of production bug rates.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Nobody Agrees About Agentic Development (Convex)“A lot of people are writing code now, but that code doesn't have many users, so it's not that there are more bugs”
Leans partly right AI guessRelease-volume data show no surge in shipped software, which fits, but there is no direct before-and-after measurement of production bug rates.
Leans partly right Debt study (302k AI commits) lines 887-889Across 302.6k verified AI-authored commits in 6,299 GitHub repos, AI commits fix slightly more code smells than they introduce but introduce more correctness and security issues than they fix, and 22.7% of introduced issues still survive at HEAD; this shows AI code is measurably adding persistent defects to real repos, though the study does not measure user reach or aggregate bug volumes against a human baseline.

AI triage of incoming pull requests lets very small maintainer teams review and merge a volume of contributions that would otherwise be unmanageable

Leans partly right

A single project's self-report; AI review tools do catch real issues, but independent evaluations find they miss most real bugs, so triage reduces load without replacing review.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Nobody Agrees About Agentic Development (Convex)“a two-person team like ours couldn't review thousands of PRs unless agents tested them and flagged the important ones for us to look at”
Leans partly right AI guessA single project's self-report; AI review tools do catch real issues, but independent evaluations find they miss most real bugs, so triage reduces load without replacing review.

Handling agent-scale code volume requires systems that make failures cheap to detect and fix rather than proportionally more human review

Leans partly right

Consistent with DORA findings that fast feedback and small batches sustain throughput, but the evidence is from practice rather than controlled comparison, and some failures (security, data loss) are never cheap.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (3)

Nobody Agrees About Agentic Development (Convex)“you should build your systems so that failures are less expensive. So that when something goes wrong you can catch it quickly, fix it quickly”
Leans partly right AI guessConsistent with DORA findings that fast feedback and small batches sustain throughput, but the evidence is from practice rather than controlled comparison, and some failures (security, data loss) are never cheap.
Leans right Airbnb migration lines 15-22Airbnb migrated ~3.5K test files by running each through automated validation steps (Jest, lint, tsc) and feeding failures back to the LLM in retry loops of up to 10+ attempts, reaching 75% in 4 hours and 97% after tuning, with humans only fixing the last ~3%; a single company case study of exactly this cheap-detect-and-fix design rather than proportional human review.
Leans partly right Vibe-coding literature review lines 1907-1921The multivocal review frames the central risk of vibe coding as a verification gap between generation speed and the capacity of developers and processes to verify outputs, with validation mechanisms and human-in-the-loop review the primary recommended controls; it supports verification as the bottleneck but notes safeguard effectiveness is largely unevaluated, so it does not show cheap-failure systems beat more human review.

Because AI makes experiments nearly free, curious developers learn system design and new technologies much faster than before

Leans partly right

Cheaper trial and error plausibly speeds learning for motivated users, but studies of AI-assisted learning show gains depend on active engagement and that passive delegation reduces skill formation.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (2)

Nobody Agrees About Agentic Development (Convex)“I'm learning much more because the cost of making mistakes is now nearly zero, since you can rewrite anything if you want.”
Leans partly right AI guessCheaper trial and error plausibly speeds learning for motivated users, but studies of AI-assisted learning show gains depend on active engagement and that passive delegation reduces skill formation.
Leans partly right Anthropic internal study lines 84-91Anthropic's survey of 132 engineers plus 53 interviews reports capability expansion enabling tighter feedback loops and faster learning (a 'couple week process' becoming 'a couple hour working session') and enthusiasm for quick prototyping, but others worry about losing the incidental learning of manual problem-solving; self-reported at one AI company, so it supports faster learning for some while showing the opposite for others.

AI products that pay per-token inference costs have structurally lower gross margins than traditional software

Leans partly right

Reported gross margins at AI-native companies have been well below classic SaaS levels, though margins vary widely and inference costs are falling.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Nobody Agrees About Agentic Development (Convex)“LLM products, which carry the burden of token costs, can't generate that kind of margin.”
Leans partly right AI guessReported gross margins at AI-native companies have been well below classic SaaS levels, though margins vary widely and inference costs are falling.

Experienced developers can get several-fold speedups on complex systems code by using LLMs as design and review partners

Leans partly right

Some experts report large gains on specific tasks, but several-fold figures are self-reported and controlled studies of experts in mature code show small or negative effects.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Open-source, complexity & AI coding 🔧 — with Salvatore "Antirez" Sanfilippo (Refactoring)“I believe I am more or less five times faster than I was before AI.”
Leans partly right AI guessSome experts report large gains on specific tasks, but several-fold figures are self-reported and controlled studies of experts in mature code show small or negative effects.

Manually feeding code to a frontier chat model gives better results than using coding agents or agentic editors

Leans wrong

By late 2025 agentic tools became net-useful for many experienced engineers, though full manual control of context still helps on some hard problems.

· denied 0 ·

Asserts (1)Denies (0)Verdicts (1)

Coding with LLMs in the summer of 2025 (an update) - <antirez> (antirez.com)“don’t use agents or things like editor with integrated coding agents”
Probably wrong AI guessBy late 2025 agentic tools became net-useful for many experienced engineers, though full manual control of context still helps on some hard problems.

Does AI make programmers faster? The videos say yes. The studies say: a bit.

Recap

How much faster?

Who benefits?

Quality and understanding

Where the bottleneck moves

Five most worth your time

Changed minds

Judged wrong

Contested

All claims

Videos and posts

What the studies changed

Verdict changed by a study (10)