一句话：提出 NPU-PIM 异构加速系统 NeuPIMs，通过双行缓冲与运行时子批次交错实现 GEMM/GEMV 并发，提升批量 LLM 推理吞吐。
现代基于 Transformer 的大语言模型由多个解码器块组成，其中 QKV 生成和前馈网络以计算密集的 GEMM 为主，多头注意力则需要高带宽的 GEMV。NPU 等机器学习加速器擅长 GEMM 但对 GEMV 效率较低，而 PIM 适合高效 GEMV 却缺乏处理 GEMM 的算力。为此，作者提出 NeuPIMs，将面向 GEMM 的 NPU 与面向 GEMV 的 PIM 设备结合成异构加速系统。核心挑战在于让两类平台并发执行：现有 PIM 常处于阻塞模式，且 LLM 中 GEMM 与 GEMV 存在依赖。NeuPIMs 在每个 bank 中设置双行缓冲，以同时管理内存读写和 PIM 命令，并采用运行时子批次交错，将两个独立子批次流水化。评估显示，相比仅 GPU、仅 NPU 和朴素 NPU+PIM 方案，吞吐分别提升 3 倍、2.4 倍和 1.6 倍。
Modern transformer-based Large Language Models (LLMs) are constructed with a series of decoder blocks. Each block comprises three key components: (1) QKV generation, (2) multi-head attention, and (3) feed-forward networks. In batched processing, QKV generation and feed-forward networks involve compute-intensive matrix-matrix multiplications (GEMM), while multi-head attention requires bandwidth-heavy matrix-vector multiplications (GEMV). Machine learning accelerators like TPUs or NPUs are proficient in handling GEMM but are less efficient for GEMV computations. Conversely, Processing-in-Memory (PIM) technology is tailored for efficient GEMV computation, while it lacks the computational power to handle GEMM effectively. Inspired by this insight, we propose NeuPIMs, a heterogeneous acceleration system that jointly exploits a conventional GEMM-focused NPU and GEMV-optimized PIM devices. The main challenge in efficiently integrating NPU and PIM lies in enabling concurrent operations on both platforms, each addressing a specific kernel type. First, existing PIMs typically operate in a "blocked" mode, allowing only either NPU or PIM to be active at any given time. Second, the inherent dependencies between GEMM and GEMV in LLMs restrict their parallel processing. To tackle these challenges, NeuPIMs is equipped with dual row buffers in each bank, facilitating the simultaneous management of memory read/write operations and PIM commands. Further, NeuPIMs employs a runtime sub-batch interleaving technique to maximize concurrent execution, leveraging batch parallelism to allow two independent sub-batches to be pipelined within a single NeuPIMs device. Our evaluation demonstrates that compared to GPU-only, NPU-only, and a naïve NPU+PIM integrated acceleration approaches, NeuPIMs achieves 3×, 2.4× and 1.6× throughput improvement, respectively.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{HeoLCCLHK0P24,
title = {NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing},
author = {Guseul Heo and Sangyeop Lee and Jaehong Cho and Hyunmin Choi and Sanghyeon Lee and Hyungkyu Ham and Gwangsun Kim and Divya Mahajan and Jongse Park},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3620666.3651380}
}