AISpotify Engineeringabout 2 hours ago
When Can LLMs Replace Humans in A/B Tests?
1 min read
LLM predictions can replace human outcomes in A/B tests only under strict assumptions, not by design. On the Upworthy dataset, raw gpt-4o-mini predictions recovered just 39% of the human treatment effect. Machine learning calibration recovered it, but assumptions remain unverifiable for new treatments.
Level
Hype check
Tap to vote and see what everyone thinks.
