一句话：提出一套系统化方法（评判准则、统一框架与指南），通过记录回放与剖析修补技术实现深度学习模型的可复现训练。
论文关注深度学习模型的可复现性这一日益突出的问题：可复现性直接关系到训练、测试、调试与审计等任务，但深度学习软件中的随机性（如算法实现）与硬件层面的非确定性（如 GPU）使模型难以被复现，已有实践要么侵入性过强，要么只适用于特定场景。作者提出一套系统化方法，包含三个部分：(1) 面向两类不同领域的通用评判准则，用以全面评估深度学习模型的可复现性；(2) 统一框架，利用 record-and-replay 技术缓解软件相关随机性，并用 profile-and-patch 技术控制硬件相关非确定性；(3) 可复现性指南，解释背后的原理与缓解策略。案例研究表明，该方法成功复现了六个开源模型和一个商业深度学习模型。
Reproducibility is an increasing concern in Artificial Intelligence (AI), particularly in the area of Deep Learning (DL). Being able to reproduce DL models is crucial for AI-based systems, as it is closely tied to various tasks like training, testing, debugging, and auditing. However, DL models are challenging to be reproduced due to issues like randomness in the software (e.g., DL algorithms) and non-determinism in the hardware (e.g., GPU). There are various practices to mitigate some of the aforementioned issues. However, many of them are either too intrusive or can only work for a specific usage context. In this paper, we propose a systematic approach to training reproducible DL models. Our approach includes three main parts: (1) a set of general criteria to thoroughly evaluate the reproducibility of DL models for two different domains, (2) a unified framework which leverages a record-and-replay technique to mitigate software-related randomness and a profile-and-patch technique to control hardware-related non-determinism, and (3) a reproducibility guideline which explains the rationales and the mitigation strategies on conducting a reproducible training process for DL models. Case study results show our approach can successfully reproduce six open source and one commercial DL models.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{ChenWSLRJ22,
title = {Towards training reproducible deep learning models},
author = {Boyuan Chen and Mingzhi Wen and Yong Shi and Dayi Lin and Gopi Krishnan Rajbahadur and Zhen Ming (Jack) Jiang},
booktitle = {ICSE},
year = {2022},
doi = {10.1145/3510003.3510163}
}