一句话：DEVIATE：自动化测量深度学习训练方差并对准确率和方差进行统计检验的框架。
深度学习训练具有非确定性，会带来显著的模型准确率方差（最高可达10.8%），从而影响新方法与基线比较的有效性。为可靠且高效地复现并量化这种方差，作者提出可直接部署的框架DEVIATE：它自动分析深度学习训练代码，抽取准确率、损失等被监控的重要指标，并以最小人工代价测量训练方差；同时提供准确率与方差的统计检验，输出p值、效应量及不同置信水平下的报告。评估通过对抗训练案例进行：在使用FGSM生成对抗样本作为训练数据的过程中，DEVIATE测得固定随机种子的8次相同训练之间准确率最大差异可达5.1%。
Deep learning (DL) training is nondeterministic and such nondeterminism was shown to cause significant variance of model accuracy (up to 10.8%). Such variance may affect the validity of the comparison of newly proposed DL techniques with baselines. To ensure such validity, DL researchers and practitioners must replicate their experiments multiple times with identical settings to quantify the variance of the proposed approaches and baselines. Replicating and measuring DL variances reliably and efficiently is challenging and understudied.We propose a ready-to-deploy framework DEVIATE that (1) measures DL training variance of a DL model with minimal manual efforts, and (2) provides statistical tests of both accuracy and variance. Specifically, DEVIATE automatically analyzes the DL training code and extracts monitored important metrics (such as accuracy and loss). In addition, DEVIATE performs popular statistical tests and provides users with a report of statistical p-values and effect sizes along with various confidence levels when comparing to selected baselines.We demonstrate the effectiveness of DEVIATE by performing case studies with adversarial training. Specifically, for an adversarial training process that uses the Fast Gradient Signed Method to generate adversarial examples as the training data, DEVIATE measures a max difference of accuracy among 8 identical training runs with fixed random seeds to be up to 5.1%.Tool and demo links: https://github.com/lin-tan/DEVIATE
DOI 原文 ·
@inproceedings{paperbot6482,
title = {DEVIATE: A Deep Learning Variance Testing Framework},
author = {Hung Viet Pham and Mijung Kim and Lin Tan and Yaoliang Yu and Nachiappan Nagappan},
booktitle = {ASE},
year = {2021},
doi = {10.1109/ase51524.2021.9678540}
}