Twelve Ways to Be Wrong About AI-Assisted Coding
The article discusses common misconceptions about measuring the effectiveness of AI-assisted coding tools. It highlights various flawed metrics that companies use to assess productivity, such as counting lines of code or relying on developer surveys. The author emphasizes the importance of proper research methods to accurately evaluate the impact of these tools.
- ▪Counting lines of code can misrepresent productivity as it does not account for code quality or maintenance burden.
- ▪Surveys indicating developers feel more productive with AI tools can be misleading due to biases like the Hawthorne effect and social desirability bias.
- ▪Using metrics like commits and pull requests can lead to distorted perceptions of productivity as developers may game the system to improve numbers.
Opening excerpt (first ~120 words) tap to expand
Twelve Ways to Be Wrong About AI-Assisted Coding ⇐ previous Posted 2026-05-20 next ⇒ Suppose your manager asks you next week to demonstrate that the AI coding tools your company signed up for are worth the subscription cost. Would you measure lines of code generated, or tickets closed? Or would you send out a survey asking whether developers feel more productive? Each of those approaches is flawed in a different way; the sections below explain why. Note: this post is about how people are assessing AI, not at LLM-assisted coding itself; with a little rewording, these criticisms could be applied to a lot of the claims that have been made about agile development, test-driven development, and other practices.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Third-bit.
Discussion
0 commentsMore from Third-bit
