一句话：提出 RobOT 测试框架，用量化指标指导生成能提升模型鲁棒性的测试用例并判断鲁棒性改进是否收敛。
现有深度学习测试常通过 fuzzing 或基于神经元覆盖率等指标的引导搜索寻找对抗样本，但已有研究表明这些覆盖率指标与模型鲁棒性并不相关，也无法作为测试后鲁棒性提升的置信度度量。本文提出面向鲁棒性的测试框架 RobOT，其核心是一个量化指标，用于衡量每个测试用例在通过重训练提升模型鲁棒性方面的价值，以及鲁棒性改进的收敛质量。RobOT 利用该指标自动生成对提升鲁棒性有价值的测试用例，并可指示鲁棒性改进通过测试是否已充分收敛。在多个基准数据集上的实验证实了 RobOT 在提升深度学习模型鲁棒性方面的有效性与效率，对抗鲁棒性提升 67.02%，比当前最优工作 DeepGini 高出 50.65%。
Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems. One popular direction is deep learning testing, where adversarial examples (a.k.a.~bugs) of DL systems are found either by fuzzing or guided search with the help of certain testing metrics. However, recent studies have revealed that the commonly used neuron coverage metrics by existing DL testing approaches are not correlated to model robustness. It is also not an effective measurement on the confidence of the model robustness after testing. In this work, we address this gap by proposing a novel testing framework called Robustness-Oriented Testing (RobOT). A key part of RobOT is a quantitative measurement on 1) the value of each test case in improving model robustness (often via retraining), and 2) the convergence quality of the model robustness improvement. RobOT utilizes the proposed metric to automatically generate test cases valuable for improving model robustness. The proposed metric is also a strong indicator on how well robustness improvement has converged through testing. Experiments on multiple benchmark datasets confirm the effectiveness and efficiency of RobOT in improving DL model robustness, with 67.02% increase on the adversarial robustness that is 50.65% higher than the state-of-the-art work DeepGini.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2102-05913,
title = {RobOT: Robustness-Oriented Testing for Deep Learning Systems},
author = {Jingyi Wang and Jialuo Chen and Youcheng Sun and Xingjun Ma and Dongxia Wang and Jun Sun and Peng Cheng},
booktitle = {ICSE},
year = {2021},
doi = {10.1109/icse43902.2021.00038}
}