一句话：提出 BlendServe，通过请求重排混合计算与访存密集型请求并协调前缀共享，提升离线推理吞吐。
离线批量推理因延迟不敏感而用于模型评估、数据整理等任务，主要目标是最大化吞吐。已有工作只优化单个批次内部吞吐，但应用间资源需求差异大（计算密集型与访存密集型），批间资源不均衡限制了优化空间。本文的洞见是通过请求重排将两类请求混批，以最大化资源重叠；然而该调度会与最大化前缀共享的调度冲突。为此，作者先建立性能模型分析请求资源需求，再设计 BlendServe：利用资源感知前缀树组织请求，并提出双扫描算法获得调度，从而协调资源重叠与前缀共享。在多种模型和工作负载上，BlendServe 可达最优吞吐的 90%。
Offline batch inference is gaining popularity as a cost-effective solution for latency-insensitive tasks, such as model evaluation and data curation. As the latency objective is highly relaxed, maximizing throughput is the primary goal in offline inference. Previous studies focused solely on throughput optimization within a batch. However, the diverse resource demands (compute-intensive vs. memory-intensive) across a wide range of applications make these approaches less effective, as imbalanced resource demands between batches restrict optimization opportunities. Our insight is to create batches with mixed compute- and memory-intensive requests through request reordering to maximize resource overlapping. However, such a request schedule can conflict with the schedule that maximizes prefix sharing, a widely-used performance optimization, causing suboptimal inference throughput. In this paper, we first build a performance model to analyze request resource demands. Based on it, we design BlendServe, which harmonizes both resource overlapping and prefix sharing to maximize throughput. BlendServe organizes all requests using a resource-aware prefix tree and proposes a dual scanning algorithm to obtain the request schedule. Our evaluation on various models and workloads shows that BlendServe can achieve up to 90% of the optimal throughput.
@inproceedings{Zhao0ZZK00XS26,
title = {BlendServe: Optimizing Offline Inference with Resource-Aware Batching},
author = {Yilong Zhao and Shuo Yang and Kan Zhu and Lianmin Zheng and Baris Kasikci and Yifan Qiao and Yang Zhou and Jiarong Xing and Ion Stoica},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790133}
}