GPU Wiring Matters in Distributed Training

Distributed training strategies like DDP and FSDP trade memory for communication speed. DDP keeps full model copies on each GPU, solving time but not memory. FSDP splits the model across GPUs, requiring constant data movement. Physical GPU wiring determines how fast that data moves, impacting performance regardless of strategy.