一句话：提出 QoS 驱动的 LLM 推理服务系统，通过细粒度 QoS 与动态调度实现交互式与批处理负载的共享混部。
现有 LLM 服务框架依赖孤岛式基础设施，粗粒度隔离交互式与批处理负载，资源利用率低且难以细粒度区分 QoS。QOSERVE 是 QoS 驱动的推理服务系统，支持共享基础设施上协同调度多样化负载：引入细粒度 QoS 分类，让应用指定精确延迟要求，并依据实时系统状态动态调度；利用 LLM 推理可预测性实现动态分块，在保证 QoS 的同时提升吞吐；混合优先级平衡公平与效率，过载时选择性降级请求。在 A100 集群上，服务容量提升 23% 且维持 QoS；共享集群中每副本有效吞吐最高为 Sarathi 的 2.4 倍；极端负载下 SLO 违约减少一个数量级。
The widespread adoption of Large Language Models (LLMs) has enabled diverse applications with very different latency requirements. Existing LLM serving frameworks rely on siloed infrastructure with coarse-grained workload segregation --- interactive and batch --- leading to inefficient resource utilization and limited support for fine-grained Quality-of-Service (QoS) differentiation. We present QOSERVE, a novel QoS-driven inference serving system that enables efficient co-scheduling of diverse workloads on shared infrastructure. QOSERVE introduces fine-grained QoS classification allowing applications to specify precise latency requirements, and dynamically adapts scheduling decisions based on real-time system state. Leveraging the predictable execution characteristics of LLM inference, QOSERVE implements dynamic chunking to improve overall throughput while maintaining strict QoS guarantees. Additionally, QOSERVE introduces hybrid prioritization to balance fairness and efficiency, and employs selective request relegation for graceful service degradation during overloads. Our evaluation demonstrates that QOSERVE increases serving capacity by 23% compared to current siloed deployments, while maintaining QoS guarantees on an A100 cluster, and improves per-replica goodput by up to 2.4x compared to Sarathi on a shared cluster. Notably, under extreme load, our system reduces SLO violations by an order of magnitude compared to current strategies.
@inproceedings{GoelMKAR26,
title = {QoServe: Breaking the Silos of LLM Inference Serving},
author = {Kanishk Goel and Jayashree Mohan and Nipun Kwatra and Ravi Shreyas Anupindi and Ramachandran Ramjee},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790206}
}