一句话：提出用子专业化模型作为差分测试参照来选择 DNN 测试输入，提升错误检测率。
深度学习模型测试缺乏自动化测试预言，且人工标注成本高昂。差分测试可作为替代预言，但此前没有关于如何选择参照模型的系统指南。本文提出一种基于子专业化模型（subspecialized models）的新型差分测试方法：这些模型仅在训练数据的特定切片上训练，因而专精于该切片，可作为差分测试的参照来筛选测试输入。初步评估使用基于 CNN 的 EMNIST 图像分类器，结果表明相比以更先进的 ResNet 和 LeNet 作为参照模型，该方法在所选输入上取得更高的错误检测率，同时也优于 N 版本测试（使用相同架构、相同数据分别训练）。
Testing of Deep Learning (DL) models is difficult due to the lack of automated test oracle and the high cost of human labelling. Differential testing has been used as a surrogate oracle, but there is no systematic guide on how to choose the reference model to use for differential testing. We propose a novel differential testing approach based on subspecialized models, i.e., models that are trained on sliced training data only (hence specialized for the slice). A preliminary evaluation of our approach with an CNN-based EMNIST image classifier shows that it can achieve higher error detection rate with selected inputs compared to using more advanced ResNet and LeNet as the reference model for differential testing. Our approach also outperforms N-version testing, i.e., the use of the same DL model architecture trained separately but using the same data.
DOI 原文 ·
@inproceedings{paperbot6330,
title = {Selecting test inputs for DNNs using differential testing with subspecialized model instances},
author = {Yu-Seung Ma and Shin Yoo and Taeho Kim},
booktitle = {FSE},
year = {2021},
doi = {10.1145/3468264.3473131}
}