一句话：提出分层预测驱动的 LMaaS 管理框架，提前调整资源分配与请求路由，显著降低尾延迟与资源开销。
面向每天处理数百万查询的语言模型即服务（LMaaS）平台，论文指出传统面向传统工作负载的云服务管理技术在动态服务负载与波动请求负载下难以同时满足延迟、SLO 与资源利用目标。为此作者提出 PreServe，一个以分层预测为核心的 LMaaS 管理框架：其粗粒度服务负载预测器估计周期性的 token 密度，细粒度请求负载预测器评估单个 LLM 请求的资源需求，二者共同为每个 LLM 实例构建负载预判。框架结合长期与短期预测，提前调整资源分配，从而缓解实例资源不足或过度供给的风险；同时在请求路由时同时考虑当前与预期未来实例负载，实现更均衡的负载分布。基于真实生产数据集的评估表明，PreServe 优于现有最先进方法：尾延迟降低 41.3%，资源消耗减少 49.38%，而额外开销仅为 0.23%。
Large Language Models (LLMs) have revolutionized numerous domains, driving the rise of Language-Model-as-a-Service (LMaaS) platforms that process millions of queries daily. These platforms must minimize latency and meet Service Level Objectives (SLOs) while optimizing resource usage. However, conventional cloud service management techniques, designed for traditional workloads, are suboptimal for LMaaS due to its dynamic service workloads and variable request loads. To address this, we propose nm, a tailored LMaaS management framework centered on hierarchical prediction. nmincorporates a service workload predictor to estimate periodic token density at a coarse granularity and a novel request load predictor to assess the resource demand of individual LLM requests, enabling the construction of a load anticipator for each LLM instance. By integrating both long-term and short-term predictions, nmadjusts resource allocation in advance, mitigating the risks of instance under- or over-provisioning. Besides, nmoptimizes request routing by considering both current and anticipated future instance loads, ensuring balanced load distribution across instances. Evaluations on real-world production datasets show that nmoutperforms state-of-the-art methods, reducing tail latency by 41.3%, cutting resource consumption by 49.38%, while incurring only 0.23% additional overhead.
@inproceedings{abs-2504-03702,
title = {PreServe: Intelligent Management for LMaaS Systems via Hierarchical Prediction},
author = {Zhihan Jiang and Yujie Huang and Guangba Yu and Junjie Huang and Jiazhen Gu and Michael R. Lyu},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773173}
}