一句话：提出 PIM 架构 AttAcc 及 GPU-PIM 异构系统，加速批处理 Transformer 生成式模型推理的注意力层。
Transformer 生成式模型（TbGM）在总结与生成阶段中消耗大量计算与内存资源，其中生成阶段的注意力层和全连接层主导执行时间。论文指出，传统 GPU 系统即使在批处理下也难以高效执行注意力层。为解决该低效问题，作者提出处理近内存（PIM）架构 AttAcc，用于高效执行注意力层；并进一步提出异构系统架构与优化，策略性地将 xPU 与传统系统用于全连接层、将 AttAcc 用于注意力层，从而端到端加速 TbGM 推理。评估表明，在 175B TbGM 上，GPU-PIM 系统相比相同内存容量的传统系统，性能与能效分别最高提升 2.81 倍和 2.67 倍。
The Transformer-based generative model (TbGM), comprising summarization (Sum) and generation (Gen) stages, has demonstrated unprecedented generative performance across a wide range of applications. However, it also demands immense amounts of compute and memory resources. Especially, the Gen stages, consisting of the attention and fully-connected (FC) layers, dominate the overall execution time. Meanwhile, we reveal that the conventional system with GPUs used for TbGM inference cannot efficiently execute the attention layer, even with batching, due to various constraints. To address this inefficiency, we first propose AttAcc, a processing-in-memory (PIM) architecture for efficient execution of the attention layer. Subsequently, for the end-to-end acceleration of TbGM inference, we propose a novel heterogeneous system architecture and optimizations that strategically use xPU and PIM together. It leverages the high memory bandwidth of AttAcc for the attention layer and the powerful compute capability of the conventional system for the FC layer. Lastly, we demonstrate that our GPU-PIM system outperforms the conventional system with the same memory capacity, improving performance and energy efficiency of running a 175B TbGM by up to 2.81× and 2.67×, respectively.
@inproceedings{0006CKKKKA24,
title = {AttAcc! Unleashing the Power of PIM for Batched Transformer-based Generative Model Inference},
author = {Jaehyun Park and Jaewan Choi and Kwanhee Kyung and Michael Jaemin Kim and Yongsuk Kwon and Nam Sung Kim and Jung Ho Ahn},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3620665.3640422}
}