Audit LLM Judges Before Trusting Them

An LLM judge used to gate a summarization feature rewarded longer summaries, not better ones. The model change boosted the judge score by six points, but real users saw worse summaries and filed complaints. The judge had a verbosity bias that was never audited. Calibrating against human labels, probing position bias, and testing verbosity bias are required before trusting any LLM judge.