AIHuggingFaceabout 2 hours ago
BenchMIRT: What are LLM benchmarks actually measuring?
1 min read
BenchMIRT, a new method from Ai2, audits LLM benchmarks at the individual prompt level using multidimensional Item Response Theory. Trained on 100 LLMs across 16 benchmarks and 34K questions, it independently recovered safety and general reasoning dimensions, revealing that BBQ and WMDP align more with reasoning than safety.
Level
Hype check
Tap to vote and see what everyone thinks.
Summary by ByteBrief
