一句话：提出自适应粒度调度与自适应编译策略，降低通用 CPU 上多租户深度学习服务的干扰与调度冲突，提升吞吐并降低延迟。
多租户深度学习服务在工业界日益重要，但已有研究多集中于传统负载，对通用 CPU 上的深度学习服务缺乏系统探索。本文从调度粒度与代码生成两个角度系统分析其机会与挑战，提出自适应粒度调度方案，在保证资源利用效率的同时降低调度冲突率；并提出自适应编译策略，动态且智能地挑选在独占与共享资源使用上合适的程序版本，以减少整体由干扰导致的性能损失。评估在轻、中、重三类负载下进行，结果显示在相同 QoS 目标下可服务更多请求（分别提升 71%、62%、45%），并将平均查询延迟降低 50%。
Deep learning (DL) models have achieved great success in many application domains. As such, many industrial companies such as Google and Facebook have acknowledged the importance of multi-tenant DL services. Although the multi-tenant service has been studied in conventional workloads, it is not been deeply studied on deep learning service, especially on general-purpose hardware. In this work, we systematically analyze the opportunities and challenges of providing multi-tenant deep learning services on the general-purpose CPU architecture from the aspects of scheduling granularity and code generation. We propose an adaptive granularity scheduling scheme to both guarantee resource usage efficiency and reduce the scheduling conflict rate. We also propose an adaptive compilation strategy, by which we can dynamically and intelligently pick a program with proper exclusive and shared resource usage to reduce overall interference-induced performance loss. Compared to the existing works, our design can serve more requests under the same QoS target in various scenarios (e.g., +71%, +62%, +45% for light, medium, and heavy workloads, respectively), and reduce the averaged query latency by 50%.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{LiuLZCLG22,
title = {VELTAIR: towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling},
author = {Zihan Liu and Jingwen Leng and Zhihui Zhang and Quan Chen and Chao Li and Minyi Guo},
booktitle = {ASPLOS},
year = {2022},
doi = {10.1145/3503222.3507752}
}