一句话：提出 DynaX 算法-硬件协同设计框架，用动态 X:M 细粒度结构化剪枝加速 Transformer 注意力计算。
针对长序列 Transformer 注意力二次复杂度瓶颈及动态稀疏中模式不规则、预测开销大的问题，本文提出算法-硬件协同设计框架 DynaX。其核心是动态 X:M 细粒度结构化剪枝：通过两步剪枝在每组中动态选择可变 X 个重要分数，而非固定 N:M，从而在保持一定模式规则性的同时获得高稀疏度并降低预测内存开销。随后通过块调度将分数块重组为匹配处理单元阵列（PEA）的硬件块，提高利用率。实验表明，短/长序列任务平均稀疏度达 89.54%/91.77%，精度损失低于 1%；在 BERT-base 上相比 Sanger 和 SALO2 分别加速 1.99X/1.50X，能效提升 5.16X/4.20X。
Owning to the mechanism of self-attention, Transformers have exhibited incredible performance in a wide range of artificial intelligence tasks. With the growth of sequence length, attention computation with quadratic complexity becomes the bottleneck, and dynamic sparsity is an effective technique to alleviate this problem. However, dynamic attention sparsity for long-sequence tasks suffers from two challenges, i.e., irregular sparse patterns and heavy prediction overhead. To this end, this paper proposes DynaX, an algorithm-hardware co-design framework that accelerates attention computation via dynamic X:M fine-grained structured pruning. Different from traditional N:M pruning, DynaX dynamically selects variable X (rather than a fixed N) important scores from a group via a 2-step pruning method, which results in high sparsity and less prediction memory overhead while maintaining pattern regularity to a certain extent. After that, DynaX performs block scheduling to reorganize score blocks into hardware blocks that can perfectly match the size of the processing element array (PEA), resulting in a higher utilization rate. Experimental results show that DynaX can achieve average sparsity of 89.54% and 91.77% for short-sequence tasks and long-sequence tasks, respectively, with less than 1% accuracy loss. Compared to Sanger and SALO2, DynaX achieves a speedup of 1.99X and 1.50X on the BERT-base model, and an energy efficiency improvement of 5.16X and 4.20X, respectively.
@inproceedings{XiongCLTSZL25,
title = {DynaX: Sparse Attention Acceleration with Dynamic X:M Fine-Grained Structured Pruning},
author = {Xiao Xiong and Zhaorui Chen and Yue Liang and Minghao Tian and Jiaxing Shang and Jiang Zhong and Dajiang Liu},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3676641.3715991}
}