Machine Learning

SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
[Under review] This study introduces SeerAttention, an attention mechanism that learns block-level sparsity directly from LLMs, enhancing efficiency and scalability in long-context processing without relying on predefined patterns.
Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, Mao Yang
PDF Cite Code