Spectral Bias in Neural Networks Explained

Neural networks fit regression targets from low to high frequencies, a phenomenon called spectral bias. This means learning high-frequency functions requires many training epochs. The bias stems from activation functions like ReLU and tanh, whose Fourier spectra decay rapidly at high frequencies, inherently favoring low-frequency learning.