一句话：揭示深度学习训练中算法与库实现级非确定性导致多次运行产生不同模型，并讨论问题与机会。
论文关注深度学习训练软件系统的可靠性问题。由于训练算法本身利用非确定性来提升模型精度与训练效率，即使训练数据、算法和网络完全相同，多次训练也会得到精度和训练时间不同的模型。除这些算法因素外，TensorFlow、cuDNN 等深度学习库还会因并行、优化和浮点计算引入额外的实现级方差。论文将这些差异归纳为训练深度学习软件系统面临的问题，并探讨相应的研究机会。摘要未给出具体实验方法或量化结果。
Deep learning (DL) training algorithms utilize nondeterminism to improve models' accuracy and training efficiency. Hence, multiple identical training runs (e.g., identical training data, algorithm, and network) produce different models with different accuracies and training times. In addition to these algorithmic factors, DL libraries (e.g., TensorFlow and cuDNN) introduce additional variance (referred to as implementation-level variance) due to parallelism, optimization, and floating-point computation.
DOI 原文 ·
@inproceedings{paperbot6013,
title = {Problems and opportunities in training deep learning software systems},
author = {Hung Viet Pham and Shangshu Qian and Jiannan Wang and Thibaud Lutellier and Jonathan Rosenthal and Lin Tan and Yaoliang Yu and Nachiappan Nagappan},
booktitle = {ASE},
year = {2020},
doi = {10.1145/3324884.3416545}
}