Building better AI benchmarks: How many raters are enough?

Google Research introduced an evaluation framework that optimizes the trade-off between the number of items and raters per item for building reproducible AI benchmarks. The framework addresses how human disagreement is often ignored in benchmarking, using "gold" ratings data to capture nuance while managing annotation costs.