一句话：提出SpecPIM，通过架构与数据流协同设计空间探索，在PIM系统上加速LLM投机推理。
生成式大语言模型的自回归解码存在token依赖，导致推理效率低下。投机推理借助小模型生成草稿token、大模型验证来缓解该问题，但两类模型计算模式不同，带来可变资源需求，使现有系统难以充分释放其加速潜力。为此，作者提出SpecPIM，在启用PIM的系统上加速投机推理。该方法构建架构设计空间以满足不同模型的差异化资源需求，并设计数据流设计空间以充分利用硬件资源，进而提出设计空间探索框架，为不同目标场景给出最优设计。评估表明，与GPU上的投机推理及现有基于PIM的LLM加速器相比，SpecPIM取得1.52×/2.02×的几何平均加速和6.67×/2.68×的几何平均能效提升。
Generative large language models' (LLMs) inference suffers from inefficiency because of the token dependency brought by autoregressive decoding. Recently, speculative inference has been proposed to alleviate this problem, which introduces small language models to generate draft tokens and adopts the original large language model to conduct verification. Although speculative inference can enhance the efficiency of the decoding procedure, we find that it presents variable resource demands due to the distinct computation patterns of the models used in speculative inference. This variability impedes the full realization of speculative inference's acceleration potential in current systems. To tackle this problem, we propose SpecPIM to accelerate speculative inference on the PIM-enabled system. SpecPIM aims to boost the performance of speculative inference by extensively exploring the heterogeneity brought by both the algorithm and the architecture. To this end, we construct the architecture design space to satisfy each model's disparate resource demands and dedicate the dataflow design space to fully utilize the system's hardware resources. Based on the co-design space, we propose a design space exploration (DSE) framework to provide the optimal design under different target scenarios. Compared with speculative inference on GPUs and existing PIM-based LLM accelerators, SpecPIM achieves 1.52×/2.02× geomean speedup and 6.67×/2.68× geomean higher energy efficiency.
@inproceedings{LiZZZL024,
title = {SpecPIM: Accelerating Speculative Inference on PIM-Enabled System via Architecture-Dataflow Co-Exploration},
author = {Cong Li and Zhe Zhou and Size Zheng and Jiaxi Zhang and Yun Liang and Guangyu Sun},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3620666.3651352}
}