AIGoogle Cloud Blogabout 10 hours ago
Google Cloud KV Cache Offloading Cuts GPU Hours 60%
12 min read
Google Cloud Managed Lustre with llm-d offloading serves as a cluster-wide attention cache tier for multi-node LLM inference. On a six-node A3 Mega cluster running Llama-3.3-70B, the approach achieves over 50% TCO savings and nearly 60% fewer GPU-hours with a 95% cache hit rate.
Level
Hype check
Tap to vote and see what everyone thinks.
In this storyGoogle
