一句话：提出基于搜索的测试输入生成工具 DeepMetis，自动扩充 DL 测试集以提升变异得分。
深度学习组件被广泛集成到复杂软件系统中，其测试数据充分性可通过发现人工注入变异（模拟真实 DL 故障）的能力来评估。本文提出自动生成新测试输入的方法 DeepMetis，用于扩充已有测试集以提高其检测 DL 变异的能力。该工具采用基于搜索的输入生成策略，并针对训练与变异过程的非确定性，在适应度函数中同时考虑被测 DL 模型的多个实例。实验表明，DeepMetis 能有效扩充给定测试集，平均将变异体检测能力提升 63%；留一实验进一步显示，扩充后的测试集能够暴露未见变异体，从而发现此前未检测到的故障。
Deep Learning (DL) components are routinely integrated into software systems that need to perform complex tasks such as image or natural language processing. The adequacy of the test data used to test such systems can be assessed by their ability to expose artificially injected faults (mutations) that simulate real DL faults.In this paper, we describe an approach to automatically generate new test inputs that can be used to augment the existing test set so that its capability to detect DL mutations increases. Our tool DeepMetis implements a search based input generation strategy. To account for the non-determinism of the training and the mutation processes, our fitness function involves multiple instances of the DL model under test. Experimental results show that DeepMetis is effective at augmenting the given test set, increasing its capability to detect mutants by 63% on average. A leave-one-out experiment shows that the augmented test set is capable of exposing unseen mutants, which simulate the occurrence of yet undetected faults.
DOI 原文 ·
@inproceedings{paperbot6601,
title = {DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score},
author = {Vincenzo Riccio and Nargiz Humbatova and Gunel Jahangirova and Paolo Tonella},
booktitle = {ASE},
year = {2021},
doi = {10.1109/ase51524.2021.9678764}
}