AIHacker Newsabout 3 hours ago
Kimi Delta Attention Explained Simply
1 min read
Kimi Delta Attention (KDA) replaces Gated DeltaNet's scalar retention gate with a per-channel vector, enabling finer control over memory. The delta rule corrects the state along the current key direction, while the vector gate forgets selectively per key channel.
Level
Hype check
Tap to vote and see what everyone thinks.
Summary by ByteBrief
