一句话：系统刻画云端 LLM 训练/推理功耗特征，并提出可超分功耗的 POLCA 框架，在推理集群多部署 30% 服务器。
LLM 规模持续增长，使数据中心 GPU 的算力需求与功耗迅速攀升，功耗已成为关键瓶颈。本文广泛刻画多种 LLM 及其配置的功耗模式，重点区分训练与推理的功耗差异，并结合生产集群数据指出：LLM 推理集群的平均与峰值功耗利用率不应很高，因而存在可观的功耗超分空间。然而，虚拟化环境中 GPU 的遥测与控制能力有限，构建可靠稳健的功耗管理框架并不容易。基于这些观察，作者提出 POLCA 框架，用于在 LLM 推理云中实现功耗超分。该框架强调鲁棒、可靠且易于部署。评估上，作者用开源模型复现生产环境观察到的功耗模式，并在模拟中展示 POLCA 可使现有集群多部署 30% 的服务器，同时性能损失很小。
Recent innovation in large language models (LLMs), and their myriad use cases have rapidly driven up the compute demand for datacenter GPUs. Several cloud providers and other enterprises plan to substantially grow their datacenter capacity to support these new workloads. A key bottleneck resource in datacenters is power, which LLMs are quickly saturating due to their rapidly increasing model sizes. We extensively characterize the power consumption patterns of a variety of LLMs and their configurations. We identify the differences between the training and inference power consumption patterns. Based on our analysis, we claim that the average and peak power utilization in LLM inference clusters should not be very high. Our deductions align with data from production LLM clusters, revealing that inference workloads offer substantial headroom for power oversubscription. However, the stringent set of telemetry and controls that GPUs offer in a virtualized environment make it challenging to build a reliable and robust power management framework. We leverage the insights from our characterization to identify opportunities for better power management. As a detailed use case, we propose a new framework called POLCA, which enables power oversubscription in LLM inference clouds. POLCA is robust, reliable, and readily deployable. Using open-source models to replicate the power patterns observed in production, we simulate POLCA and demonstrate that we can deploy 30% more servers in existing clusters with minimal performance loss.
@inproceedings{PatelCZGWMB24,
title = {Characterizing Power Management Opportunities for LLMs in the Cloud},
author = {Pratyush Patel and Esha Choukse and Chaojie Zhang and Íñigo Goiri and Brijesh Warrier and Nithish Mahalingam and Ricardo Bianchini},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3620666.3651329}
}