Sliding Window Attention: A Simple Post-Training Switch That Beats Linear Attention

Core Information

Sliding Window Attention (SWA) restricts each token to attend to a fixed-size local neighborhood, cutting attention and KV-cache memory for long-context models. A new paper shows that switching to an SWA mask with attention sinks, at no extra cost, beats linear attention after post-training.

Key Points

• SWA reduces memory and computation by limiting attention to a local window, with periodic global-attention layers to preserve broader context.
• The paper's result: a simple sliding-window mask with attention sinks outperforms linear attention in post-training, without added cost.
• The method is available on Papers with Code, making it easy to explore and compare.