一句话：提出首个基于真实 DL 故障的源代码级预训练变异测试工具 DeepCrime，并定义 35 个变异算子。
深度学习系统日益普及，但其测试仍是开放问题。现有变异测试虽被用于 DL 系统，但尚无工作研究如何通过变异算子模拟真实 DL 故障的影响。本文基于 3 项关于 DL 系统真实故障的实证研究，定义了 35 个 DL 变异算子；这些算子从已有故障分类中经系统化流程提取，并在出现分歧时采用形式化冲突消解阶段。作者将其中 24 个算子实现为 DeepCrime——首个基于真实 DL 故障的源代码级预训练变异工具。评估关注这些算子是否产生“有趣”的变异，即可被杀死但非平凡的变异；随后将工具对测试数据质量变化的敏感性与已有后训练 DL 变异工具 DeepMutation++ 进行比较。
Deep Learning (DL) solutions are increasingly adopted, but how to test them remains a major open research problem. Existing and new testing techniques have been proposed for and adapted to DL systems, including mutation testing. However, no approach has investigated the possibility to simulate the effects of real DL faults by means of mutation operators. We have defined 35 DL mutation operators relying on 3 empirical studies about real faults in DL systems. We followed a systematic process to extract the mutation operators from the existing fault taxonomies, with a formal phase of conflict resolution in case of disagreement. We have implemented 24 of these DL mutation operators into DeepCrime, the first source-level pre-training mutation tool based on real DL faults. We have assessed our mutation operators to understand their characteristics: whether they produce interesting, i.e., killable but not trivial, mutations. Then, we have compared the sensitivity of our tool to the changes in the quality of test data with that of DeepMutation++, an existing post-training DL mutation tool.
DOI 原文 ·
@inproceedings{paperbot6661,
title = {DeepCrime: mutation testing of deep learning systems based on real faults},
author = {Nargiz Humbatova and Gunel Jahangirova and Paolo Tonella},
booktitle = {ISSTA},
year = {2021},
doi = {10.1145/3460319.3464825}
}