一句话：提出 DREAM 动态调度器，在多加速器系统上高效应对实时多模型 ML 负载的多粒度动态性，显著降低能耗-延迟积。
实时多模型 ML（RTMM）负载（如 AR/VR、无人机控制）在任务、模型及模型内层等多个粒度上呈现动态行为，导致系统总负载无法完全预测，这给 ML 系统软件带来新挑战。同时 RTMM 要求实时处理、模型高度异构且面向资源受限设备，因此调度器对硬件利用至关重要。本文提出 DREAM，面向多加速器系统有效处理 RTMM 负载中的多种动态性。DREAM 量化 RTMM 负载的独特需求，并利用量化分数结合当前系统负载与不同模型、输入帧上的其他推理任务来驱动调度决策；其可调参数能快速有效地适应负载动态变化。在五个 RTMM 负载场景中，相比现有最优基线，DREAM 将论文定义的 RTMM 能耗-延迟积等价指标 UXCosT 在几何平均上降低 32.2% 和 50.0%，最高降低 80.8% 和 97.6%，验证了该调度方法的有效性。
Emerging real-time multi-model ML (RTMM) workloads such as AR/VR and drone control involve dynamic behaviors in various granularity; task, model, and layers within a model. Such dynamic behaviors introduce new challenges to the system software in an ML system since the overall system load is not completely predictable, unlike traditional ML workloads. In addition, RTMM workloads require real-time processing, involve highly heterogeneous models, and target resource-constrained devices. Under such circumstances, developing an effective scheduler gains more importance to better utilize underlying hardware considering the unique characteristics of RTMM workloads. Therefore, we propose a new scheduler, DREAM, which effectively handles various dynamicity in RTMM workloads targeting multi-accelerator systems. DREAM quantifies the unique requirements for RTMM workloads and utilizes the quantified scores to drive scheduling decisions, considering the current system load and other inference jobs on different models and input frames. DREAM utilizes tunable parameters that provide fast and effective adaptivity to dynamic workload changes. In our evaluation of five scenarios of RTMM workload, DREAM reduces the overall UXCosT, which is an equivalent metric of the energy-delay product (EDP) for RTMM defined in the paper, by 32.2% and 50.0% in the geometric mean (up to 80.8% and 97.6%) compared to state-of-the-art baselines, which shows the efficacy of our scheduling methodology.
@inproceedings{paperbot10115,
title = {DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloads},
author = {Seah Kim and Hyoukjun Kwon and Jinook Song and Jihyuck Jo and Yu-Hsin Chen and Liangzhen Lai and Vikas Chandra},
booktitle = {ASPLOS},
year = {2023},
doi = {10.1145/3623278.3624753}
}