一句话：系统实证研究 DNN 模型覆盖准则与模型质量的相关性，并评估覆盖引导的对抗样本生成与重训练效果。
受代码覆盖率指导软件测试成功的启发，研究者提出了多种神经网络覆盖准则以指导模型测试（如对抗攻击下的准确率）。然而，神经网络模型覆盖与模型质量之间是否像代码覆盖与软件质量那样存在单调关系，仍不清楚。本文对此展开实证研究，考察 DNN 模型质量与覆盖准则之间的相关性、覆盖引导的对抗样本生成相较于梯度下降方法的效果、覆盖引导重训练相较于现有对抗训练的有效性，以及各覆盖准则之间的内部关系。摘要未报告具体结论，但为理解覆盖准则能否作为 DNN 质量与测试充分性的可靠代理提供了研究框架。
Inspired by the great success of using code coverage as guidance in software testing, a lot of neural network coverage criteria have been proposed to guide testing of neural network models (e.g., model accuracy under adversarial attacks). However, while the monotonic relation between code coverage and software quality has been supported by many seminal studies in software engineering, it remains largely unclear whether similar monotonicity exists between neural network model coverage and model quality. This paper sets out to answer this question. Specifically, this paper studies the correlation between DNN model quality and coverage criteria, effects of coverage guided adversarial example generation compared with gradient decent based methods, effectiveness of coverage based retraining compared with existing adversarial training, and the internal relationships among coverage criteria.
DOI 原文 ·
@inproceedings{paperbot5855,
title = {Correlations between deep neural network model coverage criteria and model quality},
author = {Shenao Yan and Guanhong Tao and Xuwei Liu and Juan Zhai and Shiqing Ma and Lei Xu and Xiangyu Zhang},
booktitle = {FSE},
year = {2020},
doi = {10.1145/3368089.3409671}
}