In a 2025 METR study, experienced developers using AI tools were 19% slower than developers without them. The same developers believed they were 24% faster.
That’s a 43-percentage-point gap between perception and reality. I’ll be honest: when I first saw those numbers, my reaction wasn’t “See? AI is overhyped.” It was “That explains a lot about the conversations I’ve been having.”
The Productivity Paradox
If you stop at the headline, the conclusion writes itself: AI tools don’t work, full stop. But the study doesn’t say that. It says bolting AI onto existing workflows makes things worse. Those are very different claims.
The METR study (a proper randomised controlled trial with 16 experienced open-source developers, 246 real tasks, screen recordings, and $150/hour compensation) is one of the most methodologically rigorous studies of AI developer productivity to date. The developers used primarily Cursor Pro with Claude 3.5/3.7 Sonnet, frontier models at the time. Tasks averaged two hours each, on repositories the developers had contributed to for years, averaging over a million lines of code and 22,000+ GitHub stars.
This may be part of a J-curve productivity dip: a new tool can make experienced people slower before their workflow adapts. That interpretation is plausible, but the study does not prove it. What it does show is narrower: for these developers, on these repositories and tasks, adding AI did not produce an immediate speed gain.
What I find most revealing is that developers misjudged their own performance so badly. Screen recordings showed that AI-assisted coding had more idle time, including periods beyond simply waiting for the model. One plausible explanation is that each moment of active work feels easier, even while total wall-clock time grows. The study cannot establish that mechanism, so it is a hypothesis rather than a conclusion.
I recognise this in myself. On days when I lean heavily on AI tools, I feel like I’ve accomplished more. Sometimes I have. Sometimes I check the commit log and discover I’ve spent three hours on what should have been forty-five minutes, because I kept iterating on prompts instead of just writing the code. The perception-reality gap is something you have to actively monitor, and that’s an uncomfortable admission for someone who writes about these tools.
Domenic Denicola, the jsdom maintainer and one of the study participants, pushed back on the “learning curve” explanation. Agent mode, he said, “is just not that hard to learn.” The real factors were different: these were expert developers on codebases they’d worked on for five or more years with 1,500+ commits. They were already fast. The room for AI to help was narrow, and the overhead of integrating AI suggestions into deeply familiar code was real.
Google’s 2024 DORA report offers a related result at organisational scale: higher AI adoption was associated with lower delivery throughput and stability, even while respondents reported productivity benefits. It does not establish the same mechanism as the METR trial, but it reinforces the need to measure delivery rather than rely on sentiment.
Better prompts alone will not resolve that tension. The workflow around the tool matters.
The METR Follow-Up
METR attempted a follow-up study in late 2025 with a larger pool of developers and newer tools. Recruitment and task-selection effects made the resulting data difficult to interpret: many developers were reluctant to accept work that barred AI use.
One developer from the original study, when asked to return: “I’m torn. I’d like to help provide updated data on this question but also I really like using AI!”
Another from the new cohort: “My head’s going to explode if I try to do too much the old-fashioned way because it’s like trying to get across the city walking when all of a sudden I was more used to taking an Uber.”
A third admitted to biased task selection: “I avoid issues like AI can finish things in just 2 hours, but I have to spend 20 hours. I will feel so painful if the task is decided as AI-disallowed.”
METR concluded that the selection effects made the new data unreliable. The selection effect is still interesting: some developers valued access to the tools enough that they would not accept AI-disallowed tasks. That is evidence of preference, not proof of higher productivity.
I get it. The cognitive texture of coding with AI assistance is different in a way that’s hard to articulate to someone who hasn’t experienced it. It’s not that every task is faster. It’s that the frustrating parts are less frustrating, and that changes how the entire workday feels.
For the subset of original developers who did participate in the later study, METR estimated an 18% speedup. METR cautions against reading too much into that estimate because of selection bias. Newer tools or adapted workflows may explain the direction, but the follow-up cannot separate those possibilities.
The Multiplier Thesis
The previous chapter argued that code production has become cheap. Here the relevant consequence is that AI amplifies both the quality of the input and the user’s ability to judge what comes back.
Deep domain knowledge and clear constraints can produce useful output quickly. Vague instructions without a way to evaluate the result can produce confident-sounding garbage at the same speed.
That is why the human evaluator remains part of the system. A model can generate a plausible answer from the context it receives, but it cannot reliably supply missing organisational history or evaluate its own output with stakes-aware judgement.
I see this daily in my own work. The output I get from AI tools when I know a domain well is categorically different from the output I get when I’m exploring unfamiliar territory. In the first case, I can spot the subtle errors, redirect when the approach is wrong, and extract real value. In the second case, I’m essentially hoping the AI got it right, and hope is not a strategy.
Why “Exoskeleton” Is the Right Metaphor
I’ve used several metaphors for AI tools in the past. Nail guns. Monkey Paws. Multipliers. Each captures something real, but none of them quite captured the thing that makes developers unable to go back even when the data says they’re slower.
Ben Gregory at Kasava proposed a frame that I think gets closest: AI isn’t an autonomous coworker. It’s an exoskeleton. And he didn’t just assert this, he grounded it in actual exoskeleton deployment data, which turns a nice analogy into something with real teeth.
Physical exoskeletons support a person doing the work; they do not decide which work matters. The metaphor transfers imperfectly to software, but it captures the relationship I care about: AI can amplify production while the human remains responsible for direction and evaluation.
The follow-up hints at why some developers were reluctant to go back. They valued using the tools even though the study could not establish whether that preference reflected higher productivity, less frustration, or something else.
The Compounding Effect
Gregory’s most important insight, and the one I keep coming back to, is that the real gains from exoskeletons aren’t linear. They compound.
In software, reducing effort on boilerplate, commit messages, planning documents, and issue formatting can preserve attention for work that requires human judgement. That benefit is difficult to measure and does not appear on every task, but it helps explain why wall-clock time is not the only outcome developers care about.
This is the version of the argument I find most compelling. If judgement remains scarce, preserving attention for evaluation may matter even when a particular task is not faster.
Cognitive Debt
I wouldn’t be giving you the full picture if I presented the exoskeleton metaphor without the corresponding risk. Margaret-Anne Storey, a well-known software engineering researcher, published a concept in February 2026 that I think every developer using AI tools needs to understand: cognitive debt.
Technical debt lives in the code. Cognitive debt lives in the developers’ minds. Even if AI agents produce code that’s clean and well-structured, the humans involved may have lost their shared understanding of what the programme does, how their intentions were implemented, and how it can be changed over time.
Storey coached a student team that used AI to “prompt” features into existence for weeks. By weeks 7-8, they hit a wall. They could no longer make even simple changes without breaking something unexpected. The team initially blamed technical debt, but the real problem was deeper: no one could explain why certain design decisions had been made or how parts of the system worked together. Their shared theory of the code had fragmented.
Simon Willison reports the same phenomenon on his own projects: after prompting entire features without reviewing their implementations, he found himself getting lost, no longer having a firm mental model of what the code can do and how it works.
I’ve felt the early signs of this myself. There’s a specific moment I’ve learned to recognise: when I accept a large AI-generated change without fully understanding it because “the tests pass and I need to move on.” Each time that happens, my mental model of the codebase gets a little blurrier. Do it enough times and you’re navigating by GPS in a city you used to know by heart. You can still get where you’re going, but you’ve lost the ability to take shortcuts or spot when the directions are wrong.
Martin Fowler, commenting on Storey’s work, drew a useful parallel to his own long-standing distinction between “cruft” and “debt” in codebases. The cognitive equivalent of cruft is ignorance, of the code and of the domain. The debt metaphor still applies: either it costs more to add new capabilities (paying interest), or you make an explicit investment to gain knowledge (paying down principal). Which you do depends on the relative costs.
Fowler also noted something that connects cognitive debt to context engineering: “The Venn Diagram of Developer Experience and Agent Experience is a circle.” The practices that make codebases easier for humans to understand, like clear modularity, descriptive naming, good documentation, also make them easier for AI agents to work with. Fixing cognitive debt and improving agent performance are the same activity. Investing in code quality isn’t at odds with AI-assisted development; it’s a prerequisite for it.
This is one material risk of AI-assisted development. The code may be fine, but velocity without understanding is difficult to sustain. The exoskeleton makes you stronger; it does not choose the destination.
The Adoption Journey
Mitchell Hashimoto, creator of Terraform, Vagrant, and the Ghostty terminal, described his personal AI adoption journey in a way that resonated with my own experience. His framing: meaningful tool adoption necessarily goes through inefficiency, then adequacy, then workflow-altering discovery. You must force yourself through the first two phases. There are no shortcuts.
Hashimoto’s first move was to reproduce all his manual commits with agentic ones, literally doing the work twice. Excruciating, but he rediscovered from first principles what others were saying. Three things emerged:
- First, break sessions into separate clear, actionable tasks. Don’t try to do everything in one mega session.
- Second, for vague requests, split into planning versus execution sessions.
- Third, and this is the most important, if you give an agent a way to verify its work, it usually fixes its own mistakes and prevents regressions.
The Window
If you’ve read this far and haven’t started experimenting seriously with AI tools, here’s what I’d tell you.
Do not limit experiments to toy prompts. Try a bounded problem from real work, preferably one where you can state the constraints and verify the result. Harder problems force you to articulate what you are trying to accomplish, what you have tried, and what risks matter. Then compare the result with how you would have worked without the tool. The point is to build evidence about your own workflow, not borrow somebody else’s confidence.