一句话：提出精度缩放的高吞吐推理服务系统Proteus，联合优化模型变体选择、设备放置与负载分配。
现有机器学习推理服务系统主要依赖硬件扩容来应对查询增长，但固定规模的边缘集群或私有云硬件资源有限，难以持续扩容。本文研究精度缩放（accuracy scaling），即通过调整ML模型精度而非硬件资源来满足吞吐需求并最大化精度。为此，作者提出联合优化三个子问题：模型变体选择、异构设备放置以及查询工作负载分配，并设计自适应批处理算法以应对查询到达时间波动、减少SLO违反。基于这些技术实现了推理服务系统Proteus。在真实和合成轨迹上的实验表明，与基线方案相比，Proteus在满足吞吐要求的同时，将精度下降最多降低3倍，并将延迟超时减少2至10倍。
Existing machine learning inference-serving systems largely rely on hardware scaling by adding more devices or using more powerful accelerators to handle increasing query demands. However, hardware scaling might not be feasible for fixed-size edge clusters or private clouds due to their limited hardware resources. A viable alternate solution is accuracy scaling, which adapts the accuracy of ML models instead of hardware resources to handle varying query demands. This work studies the design of a high-throughput inference-serving system with accuracy scaling that can meet throughput requirements while maximizing accuracy. To achieve the goal, this work proposes to identify the right amount of accuracy scaling by jointly optimizing three sub-problems: how to select model variants, how to place them on heterogeneous devices, and how to assign query workloads to each device. It also proposes a new adaptive batching algorithm to handle variations in query arrival times and minimize SLO violations. Based on the proposed techniques, we build an inference-serving system called Proteus and empirically evaluate it on real-world and synthetic traces. We show that Proteus reduces accuracy drop by up to 3× and latency timeouts by 2--10× with respect to baseline schemes, while meeting throughput requirements.
@inproceedings{AhmadGFWSW24,
title = {Proteus: A High-Throughput Inference-Serving System with Accuracy Scaling},
author = {Sohaib Ahmad and Hui Guan and Brian D. Friedman and Thomas Williams and Ramesh K. Sitaraman and Thomas Woo},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3617232.3624849}
}