一句话：提出并行语义分析方法，自动选择混合并行策略，相比数据并行取得 260% 加速。
深度神经网络规模与数据量不断增长，高效并行训练成为关键需求。数据并行虽最常用，但在大参数模型上效率不足；混合并行对不同部分采用不同策略，效率更高却依赖专家级配置。本文提出一种并行语义分析方法，用于分析数据并行、混合并行等不同并行方式之间的权衡，并系统性地选择具有良好训练时间性能的混合并行策略。该方法旨在让 AI 研究者无需精通并行计算即可自动获得较优配置。实验中，与常规数据并行方法相比，该方案实现了 260% 的加速，说明自动化混合并行配置能显著提升 DNN 训练效率，并减轻研究者的工程负担。
Deep neural network (DNN) has become the leading technology to realize Artificial Intelligence (AI). As DNN models become larger and more complex, so do datasets. Being able to efficiently train DNNs in parallel has become a crucial need. Data Parallelism (DP) is the widest-used solution today to accelerate DNN training but could be inefficient when processing DNNs with large-size parameters. Hybrid Parallelism (HP), which applies different parallel strategies on different parts of DNNs, is more efficient but requires advanced configurations. Not all AI researchers are experts in parallel computing, thus automating the configuration of HP strategies is very desirable for all AI frameworks. We propose a parallel semantics analysis method, which can analyze the trade-offs among different kinds of parallelisms and systematically choose the HP strategies with good training time performance. We demonstrated experimentally 260% speedup when applying our method compared to using a conventional DP approach. With our proposal, AI researchers would be able to focus more on AI algorithm research without being disturbed by parallel analysis and engineering concerns.
DOI 原文 ·
@inproceedings{paperbot6287,
title = {Freeing hybrid distributed AI training configuration},
author = {Haoran Wang},
booktitle = {FSE},
year = {2021},
doi = {10.1145/3468264.3473104}
}