Dust Pretrains Transformers Without Backprop

Researchers present Dust, the first zeroth-order method competitive with backprop for pretraining transformer language models. Dust perturbs activations independently at every token, allowing one forward pass to evaluate a virtual population in parallel. From 1M tokens up, Dust is on the order of 10^3 to 10^4 times more efficient than a transformer implementation of EGGROLL, a state-of-the-art evolution strategy method.