一句话：提出 DeepGauge 多粒度测试准则，从多个粒度刻画深度学习系统的测试集，弥补仅靠准确率衡量测试充分性的不足。
深度学习以数据驱动的编程范式，通过训练数据构建神经元网络的内部逻辑，已广泛用于安全攸关场景，但已有研究显示当前 DL 系统存在多种脆弱性。目前 DL 系统的测试充分性通常仅由测试数据的准确率衡量；由于高质量测试数据难以获取，良好的准确率难以保证测试充分性与泛化能力。与逻辑和功能清晰可控的传统软件不同，DL 系统缺乏可解释性，使系统分析与缺陷检测变得困难。为此，作者提出 DeepGauge——一组多粒度测试准则，旨在从多个粒度对测试集给出多侧面的刻画。评估在两个知名数据集、五个 DL 系统上进行，并结合四种先进的对抗攻击技术；结果表明该准则具有潜在实用性，可为构建更通用、更鲁棒的 DL 系统提供启示。
Deep learning (DL) defines a new data-driven programming paradigm that constructs the internal system logic of a crafted neuron network through a set of training data. We have seen wide adoption of DL in many safety-critical scenarios. However, a plethora of studies have shown that the state-of-the-art DL systems suffer from various vulnerabilities which can lead to severe consequences when applied to real-world applications. Currently, the testing adequacy of a DL system is usually measured by the accuracy of test data. Considering the limitation of accessible high quality test data, good accuracy performance on test data can hardly provide confidence to the testing adequacy and generality of DL systems. Unlike traditional software systems that have clear and controllable logic and functionality, the lack of interpretability in a DL system makes system analysis and defect detection difficult, which could potentially hinder its real-world deployment. In this paper, we propose DeepGauge, a set of multi-granularity testing criteria for DL systems, which aims at rendering a multi-faceted portrayal of the testbed. The in-depth evaluation of our proposed testing criteria is demonstrated on two well-known datasets, five DL systems, and with four state-of-the-art adversarial attack techniques against DL. The potential usefulness of DeepGauge sheds light on the construction of more generic and robust DL systems.
@inproceedings{paperbot4746,
title = {DeepGauge: multi-granularity testing criteria for deep learning systems},
author = {Lei Ma and Felix Juefei-Xu and Fuyuan Zhang and Jiyuan Sun and Minhui Xue and Bo Li and Chunyang Chen and Ting Su and Li Li and Yang Liu and Jianjun Zhao and Yadong Wang},
booktitle = {ASE},
year = {2018},
doi = {10.1145/3238147.3238202}
}