一句话：提出 Adopter，用 DSL 表达 DL 模型与转换规则，自动将 DL kernel 集成到模型中，提升精度召回与训练效率。
随着深度学习模型日益增大和复杂，提升训练与推理效率至关重要。虽然已有大量高度优化的 DL kernel，但人工决定用哪个 kernel、在哪里用以及如何正确使用仍十分繁琐耗时。本文提出自动化深度学习优化方法 Adopter：设计 DSL 表示 DL 模型架构，并用该 DSL 指定将 DL kernel 集成到模型所需的模型转换规则。给定模型源代码和一组 kernel 的转换规则，Adopter 先进行过程间分析，将模型架构识别并表达为 DSL；再通过作用域分析和子序列匹配定位可应用转换规则的位置；最后提出基于合成的代码转换方法应用规则。作者从 Hugging Face 收集 199 个模型和多种 DL kernel 构建基准。相比最先进的自动化代码转换技术，Adopter 的精度和召回率分别提升 3% 和 56%；对 9 个模型的深入分析表明，训练速度平均提升 22.7%，GPU 显存使用降低 10.5%。
As deep learning models become increasingly bigger and more complex, it is critical to improve model training and inference efficiency. Though a variety of highly optimized libraries and packages (known as DL kernels) have been developed, it is tedious and time-consuming to figure out which kernel to use, where to use, and how to use them correctly. To address this challenge, we propose an Automated Deep learning OPTimization approach called Adopter. We design a Domain-Specific Language (DSL) to represent DL model architectures and leverage this DSL to specify model transformation rules required to integrate a DL kernel into a model. Given the source code of a DL model and the transformation rules for a set of kernels, Adopter first performs inter-procedural analysis to identify and express the model architecture in our DSL. Then, Adopter performs scope analysis and sub-sequence matching to identify locations in the model architecture where the transformation rules can be applied. Finally, Adopter proposes a synthesis-based code transformation method to apply the transformation rule. We curated a benchmark with 199 models from Hugging Face and a diverse set of DL kernels. We found that, compared to a state-of-the-art automated code transformation technique, Adopter helps improve the precision and recall by 3% and 56%, respectively. An in-depth analysis of 9 models revealed that on average, Adopter improved the training speed by 22.7% while decreasing the GPU memory usage by 10.5%.
@inproceedings{paperbot8594,
title = {Automated Deep Learning Optimization via DSL-Based Source Code Transformation},
author = {Ruixin Wang and Minghai Lu and Cody Hao Yu and Yi-Hsiang Lai and Tianyi Zhang},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3652143}
}