Sliding-Window Attention with Attention Sinks Outperforms Linear Attention Post-Training

核心信息

A new paper demonstrates that replacing standard attention with a sliding-window attention mask combined with attention sinks outperforms linear attention during post-training, with no extra cost.

要点

• The approach is simple: switch to a sliding-window attention mask with attention sinks.
• It beats linear attention in post-training settings, despite being conceptually straightforward.
• The work is credited to collaborators and the full paper is available online.