一句话：提出算法-架构协同设计的 DOTA，通过运行时检测并省略弱注意力连接加速长序列 Transformer 推理。
长序列 Transformer 推理中，自注意力具有二次计算复杂度和高内存开销，成为可扩展性瓶颈。DOTA 基于注意力图中并非所有连接同等重要的观察，提出算法-架构协同设计：联合优化一个轻量级 Detector 与 Transformer 模型，在运行时准确检测并省略弱注意力连接，同时设计专用系统架构实现端到端加速。在广泛基准上，DOTA 相比 GPU 取得 152.6 倍加速，相比定制硬件取得 4.5 倍加速，并带来数量级的能效提升。
Transformer Neural Networks have demonstrated leading performance in many applications spanning over language understanding, image processing, and generative modeling. Despite the impressive performance, long-sequence Transformer processing is expensive due to quadratic computation complexity and memory consumption of self-attention. In this paper, we present DOTA, an algorithm-architecture co-design that effectively addresses the challenges of scalable Transformer inference. Based on the insight that not all connections in an attention graph are equally important, we propose to jointly optimize a lightweight Detector with the Transformer model to accurately detect and omit weak connections during runtime. Furthermore, we design a specialized system architecture for end-to-end Transformer acceleration using the proposed attention detection mechanism. Experiments on a wide range of benchmarks demonstrate the superior performance of DOTA over other solutions. In summary, DOTA achieves 152.6x and 4.5x performance speedup and orders of magnitude energy-efficiency improvements over GPU and customized hardware, respectively.
@inproceedings{Qu0TCD022,
title = {DOTA: detect and omit weak attentions for scalable transformer acceleration},
author = {Zheng Qu and Liu Liu and Fengbin Tu and Zhaodong Chen and Yufei Ding and Yuan Xie},
booktitle = {ASPLOS},
year = {2022},
doi = {10.1145/3503222.3507738}
}