一句话：首个面向可抢占 GPU 实例的分布式 LLM 服务系统，通过动态并行、迁移优化与有状态恢复降低推理延迟与成本。
生成式大语言模型服务计算与显存开销高，利用云上廉价但随时可能被抢占的 spot GPU 实例可显著降低成本，但频繁抢占与实例迁移带来严峻挑战。本文提出 SpotServe，首个在可抢占实例上服务 LLM 的分布式系统。它动态调整 LLM 并行化配置，在吞吐、延迟与费用间权衡；将抢占迁移建模为二分图匹配并用 KM 算法求最小通信成本迁移方案；还提出有状态推理恢复，以更细粒度提交推理进度，利用云平台宽限期在抢占后低成本恢复。基于真实 spot 抢占轨迹和多种流行 LLM 的评估显示，SpotServe 相比最佳现有系统将 P99 尾延迟降低 2.4–9.1 倍，相比仅用按需实例节省 54% 费用。
The high computational and memory requirements of generative large language models (LLMs) make it challenging to serve them cheaply. This paper aims to reduce the monetary cost for serving LLMs by leveraging preemptible GPU instances on modern clouds, which offer accesses to spare GPU resources at a much cheaper price than regular instances but may be preempted by the cloud provider at any time. Serving LLMs on preemptible instances requires addressing challenges induced by frequent instance preemptions and the necessity of migrating instances to handle the preemptions. This paper presents SpotServe, the first distributed LLM serving system on preemptible instances. Several key techniques of SpotServe realize fast and reliable serving of generative LLMs on cheap preemptible instances. First, SpotServe dynamically adapts the LLM parallelization configuration for dynamic instance availability and fluctuating workload, while balancing the trade-off among the overall throughput, inference latency and monetary costs. Second, to minimize the cost of migrating instances for dynamic reparallelization, the task of migrating instances is formulated as a bipartite graph matching problem in SpotServe, which uses the Kuhn-Munkres algorithm to identify an optimal migration plan that minimizes communication cost. Finally, to take advantage of the grace period offered by modern cloud platforms, we introduce stateful inference recovery, a new inference mechanism that commits inference progress at a much finer granularity and allows SpotServe to cheaply resume inference upon preemption. We evaluate SpotServe on real spot instance preemption traces and various popular LLMs and show that SpotServe can reduce the P99 tail latency by 2.4 - 9.1× compared with the best existing LLM serving systems. We also show that SpotServe can leverage the price advantage of preemptive instances, saving 54% monetary cost compared with only using on-demand instances. The code is publicly available at: https://github.com/Hsword/SpotServe.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2311-15566,
title = {SpotServe: Serving Generative Large Language Models on Preemptible Instances},
author = {Xupeng Miao and Chunan Shi and Jiangfei Duan and Xiaoli Xi and Dahua Lin and Bin Cui and Zhihao Jia},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3620665.3640411}
}