Model Evaluation and Threat Research is an AI research charity that looks into the threat of AI agents! That sounds a bit AI doomsday cult, and they take funding from the AI doomsday cult organisat…
One interesting feature in this paper is that the programmers who used LLMs thought they were faster, they estimated it was saving about 20% of the time it would have taken without LLMs. I think that’s a clear sign that you shouldn’t trust your gut about how much time LLMs save you, you should definitely try to measure it.
The study did find a correlation between prior experience and performance. One of the developers who showed a positive speedup with AI was the one with the most previous experience using Cursor (over 50 hours).
One interesting feature in this paper is that the programmers who used LLMs thought they were faster, they estimated it was saving about 20% of the time it would have taken without LLMs. I think that’s a clear sign that you shouldn’t trust your gut about how much time LLMs save you, you should definitely try to measure it.
The study did find a correlation between prior experience and performance. One of the developers who showed a positive speedup with AI was the one with the most previous experience using Cursor (over 50 hours).