AITowards Data Scienceabout 2 hours ago
hRoPE Encoding Tracks Paragraph Structure in LLMs
10 min read
A researcher built hRoPE, a hierarchical rotary positional encoding giving paragraph, sentence and token indices separate channels, then trained matched controls. Across WikiText-2, OpenWebText and Python code, removing a real paragraph boundary lowered cross-paragraph attention while inserting a fake one raised it. Shuffling paragraph order moved the compression depth for the real model in every corpus, but never for the random-label control.
Level
Hype check
Tap to vote and see what everyone thinks.