ChipsTowards Data Scienceabout 2 hours ago
Disaggregation Is a Thousand-GPU Problem
1 min read
Disaggregating prefill and decode onto separate GPU pools only pays off above roughly a thousand GPUs with fast interconnect. Chunked prefill on the same pool handles most scheduling interference without network KV transfer costs. DistServe measured 7.4x more requests within latency constraints at scale.
Level
Hype check
Tap to vote and see what everyone thinks.
Summary by ByteBrief
