一句话：提出 ExeGPT 分布式系统，在延迟约束下搜索最优执行调度，以最大化 LLM 推理吞吐。
本文提出 ExeGPT，一个面向 LLM 推理的约束感知分布式系统，旨在给定延迟约束下寻找并执行最优调度以最大化推理吞吐。它利用输入/输出序列分布分配资源，并确定批大小和部分张量并行等执行配置；同时提出基于轮询分配和工作负载感知分配的两种调度策略，以适应不同 NLP 负载。作者在 T5、OPT 和 GPT-3 的六个 LLM 实例及五个 NLP 任务上评估，每个任务设四种延迟约束。相比 FasterTransformer，ExeGPT 吞吐最高提升 15.2 倍、延迟最高改善 6 倍，二十个场景平均吞吐增益 2.9 倍；适应序列分布变化时调整调度开销较合理。
This paper presents ExeGPT, a distributed system designed for constraint-aware LLM inference. ExeGPT finds and runs with an optimal execution schedule to maximize inference throughput while satisfying a given latency constraint. By leveraging the distribution of input and output sequences, it effectively allocates resources and determines optimal execution configurations, including batch sizes and partial tensor parallelism. We also introduce two scheduling strategies based on Round-Robin Allocation and Workload-Aware Allocation policies, suitable for different NLP workloads. We evaluate ExeGPT on six LLM instances of T5, OPT, and GPT-3 and five NLP tasks, each with four distinct latency constraints. Compared to FasterTransformer, ExeGPT achieves up to 15.2× improvements in throughput and 6× improvements in latency. Overall, ExeGPT achieves an average throughput gain of 2.9× across twenty evaluation scenarios. Moreover, when adapting to changing sequence distributions, the cost of adjusting the schedule in ExeGPT is reasonably modest. ExeGPT proves to be an effective solution for optimizing and executing LLM inference for diverse NLP workload and serving conditions.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2404-07947,
title = {ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference},
author = {Hyungjun Oh and Kihong Kim and Jaemin Kim and Sungkyun Kim and Junyeol Lee and Du-seong Chang and Jiwon Seo},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3620665.3640383}
}