KV Cache Tax Causes GPU Memory Bottlenecks

A vLLM analysis found naive KV cache management wastes 60 to 80 percent of reserved GPU memory, causing out-of-memory errors during traffic spikes even when compute is underutilized. The cache scales with concurrent requests, not just prompt length.