AIAI Snake Oil2 months ago
Open-world evaluations for measuring frontier AI capabilities
43 min read
AI models saturate major benchmarks. Researchers introduce open-world evaluations to test real-world tasks. The CRUX collaboration of 17 researchers will regularly evaluate frontier AI. In a first experiment, an AI agent built and published an iOS app to the App Store, making two errors, one requiring manual intervention.
Level
Hype check
Tap to vote and see what everyone thinks.
