How DoorDash Built a Testing System to Evaluate LLMs

DoorDash built a testing system to evaluate large language models. The system enables real-time monitoring of model performance and cost. It tracks AI spending by token, model, provider, and team. The approach allows teams to detect cost spikes immediately. DoorDash uses this to correlate spending with infrastructure changes. The system helps reduce waste and improve model efficiency in production.