一句话：提出 DeepImportance 方法及重要性驱动准则 IDC，用组件重要性刻画测试集的语义多样性。
深度学习系统在安全与安保攸关场景中的使用需要提供测试证据，但现有工作多沿用传统软件工程中的测试准则，难以刻画这类系统固有的特性。本文提出 DeepImportance——一套系统化的深度学习系统测试方法，以及配套的重要性驱动测试充分性准则 IDC。IDC 通过逐层评估深度学习系统各组件的重要性，形成对网络功能的分层理解，并以此度量测试集的语义多样性，从而指导测试的评估与生成。作者在多个深度学习系统、多个数据集上，结合当前最先进的对抗样本生成技术进行了实证评估，结果表明该方法有效且实用，能够支持构建更鲁棒的深度学习系统。
Deep Learning (DL) systems are key enablers for engineering intelligent applications due to their ability to solve complex tasks such as image recognition and machine translation. Nevertheless, using DL systems in safety- and security-critical applications requires to provide testing evidence for their dependable operation. Recent research in this direction focuses on adapting testing criteria from traditional software engineering as a means of increasing confidence for their correct behaviour. However, they are inadequate in capturing the intrinsic properties exhibited by these systems. We bridge this gap by introducing DeepImportance, a systematic testing methodology accompanied by an Importance-Driven (IDC) test adequacy criterion for DL systems. Applying IDC enables to establish a layer-wise functional understanding of the importance of DL system components and use this information to assess the semantic diversity of a test set. Our empirical evaluation on several DL systems, across multiple DL datasets and with state-of-the-art adversarial generation techniques demonstrates the usefulness and effectiveness of DeepImportance and its ability to support the engineering of more robust DL systems.
DOI 原文 ·
@inproceedings{paperbot5628,
title = {Importance-driven deep learning system testing},
author = {Simos Gerasimou and Hasan Ferit Eniser and Alper Sen and Alper Cakan},
booktitle = {ICSE},
year = {2020},
doi = {10.1145/3377811.3380391}
}