It Still Can't Do My Job: Four Years of Moving Goalposts (2022, 2026)

ChatGPT launched in December 2022 and failed basic tasks like Snake before Stack Overflow banned its answers due to low accuracy. GPT-4 later solved the same game, while OpenAI claimed bar exam success that re-evaluation adjusted to a lower percentile. Devin generated errors during demos but did not complete assigned Upwork tasks from its showcase video. Google reported AI-generated code acceptance metrics without counting boilerplate or import statements in production workflows. METR research found developers were nineteen percent slower using tools on mature repositories despite believing they gained speed.