The paradox of LLM self-distillation: Faster reasoning, weaker generalization

Self-distillation in LLMs improves reasoning speed but can cause up to 40% accuracy drops on unseen tasks. Research from Microsoft Research, KAIST, and Seoul National University found the technique suppresses self-correction and exploration behaviors, destroying a model's ability to generalize to out-of-distribution problems.