一句话：提出 Past-Future 调度器，通过历史与未来内存占用估计提升 LLM 服务在 SLA 下的 goodput。
大语言模型服务中，continuous batching 的效果依赖对请求内存需求的准确估计。然而请求输出长度差异很大，现有框架常采用激进或保守调度器，导致内存消耗被显著高估或低估，进而引发有害的请求驱逐或过长排队，难以在严格 SLA 保证下获得满意的吞吐（goodput）。本文提出 Past-Future 调度器，利用请求输出长度的历史分布，并计算未来各时间点的内存占用，从而精确估计运行批所需的峰值内存。该方法能适应不同输入输出长度分布的应用，平衡排队与驱逐之间的权衡，持续获得更好的 goodput。作者还实现了高性能 LLM 服务框架 LightLLM 来验证该调度器。实验表明，在重负载下 LightLLM 的 goodput 相比现有激进或保守调度器最高提升 2-3 倍，且框架已开源。
The exploration and application of Large Language Models (LLMs) is thriving. To reduce deployment costs, continuous batching has become an essential feature in current service frameworks. The effectiveness of continuous batching relies on an accurate estimate of the memory requirements of requests. However, due to the diversity in request output lengths, existing frameworks tend to adopt aggressive or conservative schedulers, which often result in significant overestimation or underestimation of memory consumption. Consequently, they suffer from harmful request evictions or prolonged queuing times, failing to achieve satisfactory throughput under strict Service Level Agreement (SLA) guarantees (a.k.a. goodput), across various LLM application scenarios with differing input-output length distributions. To address this issue, we propose a novel Past-Future scheduler that precisely estimates the peak memory resources required by the running batch via considering the historical distribution of request output lengths and calculating memory occupancy at each future time point. It adapts to applications with all types of input-output length distributions, balancing the trade-off between request queuing and harmful evictions, thereby consistently achieving better goodput. Furthermore, to validate the effectiveness of the proposed scheduler, we developed a high-performance LLM serving framework, LightLLM, that implements the Past-Future scheduler. Compared to existing aggressive or conservative schedulers, LightLLM demonstrates superior goodput, achieving up to 2-3× higher goodput than other schedulers under heavy loads. LightLLM is open source to boost the research in such direction (https://github.com/ModelTC/lightllm).
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{GongBWFWLY025,
title = {Past-Future Scheduler for LLM Serving under SLA Guarantees},
author = {Ruihao Gong and Shihao Bai and Siyu Wu and Yunqian Fan and Zaijun Wang and Xiuhong Li and Hailong Yang and Xianglong Liu},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3676641.3716011}
}