Researchers at METR conducted a randomized controlled trial showing that experienced open-source developers using early-2025 AI tools experienced a 19% increase in task completion time, contrary to developer and expert predictions of a speedup. The study identifies factors like developer over-optimism, high human expertise on familiar codebases, and current AI limitations in reliability and contextual understanding as contributors to this unexpected slowdown.
Here is the link to the paper: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf