一句话：提出PRIMA，利用变异分析与学习排序优先标注更可能暴露DNN缺陷的测试输入，降低标注成本。
深度神经网络（DNN）测试中，标注测试输入以验证预测正确性成本高昂，严重影响测试与开发效率。本文提出测试输入优先级排序方法 PRIMA，通过智能变异分析，在有限标注预算内优先选择更可能揭示缺陷的输入。其核心洞察是：能杀死更多变异模型、且与更多变异输入产生不同预测的测试输入更可能暴露 DNN 缺陷。PRIMA 针对每个输入执行设计的模型与输入变异规则，并利用 learning-to-rank 融合多种变异结果进行排序。基于 36 个流行基准、覆盖五类差异的广泛实验表明，PRIMA 显著优于最先进方法，排序有效性平均提升 8.50%~131.01%；并在大型车企的自动驾驶测试中，于 4 个真实场景识别模型上验证了实用性。
Deep Neural Network (DNN) testing is one of the most widely-used ways to guarantee the quality of DNNs. However, labeling test inputs to check the correctness of DNN prediction is very costly, which could largely affect the efficiency of DNN testing, even the whole process of DNN development. To relieve the labeling-cost problem, we propose a novel test input prioritization approach (called PRIMA) for DNNs via intelligent mutation analysis in order to label more bug-revealing test inputs earlier for a limited time, which facilitates to improve the efficiency of DNN testing. PRIMA is based on the key insight: a test input that is able to kill many mutated models and produce different prediction results with many mutated inputs, is more likely to reveal DNN bugs, and thus it should be prioritized higher. After obtaining a number of mutation results from a series of our designed model and input mutation rules for each test input, PRIMA further incorporates learning-to-rank (a kind of supervised machine learning to solve ranking problems) to intelligently combine these mutation results for effective test input prioritization. We conducted an extensive study based on 36 popular subjects by carefully considering their diversity from five dimensions (i.e., different domains of test inputs, different DNN tasks, different network structures, different types of test inputs, and different training scenarios). Our experimental results demonstrate the effectiveness of PRIMA, significantly outperforming the state-of-the-art approaches (with the average improvement of 8.50%~131.01% in terms of prioritization effectiveness). In particular, we have applied PRIMA to the practical autonomous-vehicle testing in a large motor company, and the results on 4 real-world scene-recognition models in autonomous vehicles further confirm the practicability of PRIMA.
@inproceedings{WangY0ZDZ21,
title = {Prioritizing Test Inputs for Deep Neural Networks via Mutation Analysis},
author = {Zan Wang and Hanmo You and Junjie Chen and Yingyi Zhang and Xuyuan Dong and Wenbin Zhang},
booktitle = {ICSE},
year = {2021},
doi = {10.1109/icse43902.2021.00046}
}