Sequence Weighting Scaling Laws Show Non-Monotonic Trend

Researchers studied data weighting across in-house and open-weight language models, finding that the effective sequence weight exponent rises then falls with scale. Medium models learn patterns proportional to data weight, while small and large models learn independently of weight.