一句话：提出首个动态数据故障定位技术 DFauLo，基于 DNN 变异体差异定位数据集中的误标与噪声数据。
深度学习数据集中的误标和噪声等数据故障会导致模型行为不可预测，甚至造成严重损失。本文提出动态数据故障定位方法 DFauLo，其思想受基于变异的代码故障定位启发，利用 DNN 变异体之间的差异来放大并识别潜在数据故障。DFauLo 首先生成原始已训练模型的多个 DNN 变异体，然后从这些变异体中提取特征并映射为可疑度分数，以表示给定数据是数据故障的概率；同时，它根据用户反馈对待疑数据进行优先级排序，并能泛化到训练中未见的数据故障。作者在 26 个涵盖不同故障类型、数据类型和模型结构的案例以及三个常用基准数据集上进行评估，结果表明 DFauLo 在几乎所有案例中优于现有最先进技术，并在基准数据集中定位出数百个不同类型的真实数据故障。
Rich datasets have empowered various deep learning (DL) applications, leading to remarkable success in many fields. However, data faults hidden in the datasets could result in DL applications behaving unpredictably and even cause massive monetary and life losses. To alleviate this problem, in this paper, we propose a dynamic data fault localization approach, namely DFauLo, to locate the mislabeled and noisy data in the deep learning datasets. DFauLo is inspired by the conventional mutation-based code fault localization, but utilizes the differences between DNN mutants to amplify and identify the potential data faults. Specifically, it first generates multiple DNN model mutants of the original trained model. Then it extracts features from these mutants and maps them into a suspiciousness score indicating the probability of the given data being a data fault. Moreover, DFauLo is the first dynamic data fault localization technique, prioritizing the suspected data based on user feedback, and providing the generalizability to unseen data faults during training. To validate DFauLo, we extensively evaluate it on 26 cases with various fault types, data types, and model structures. We also evaluate DFauLo on three widely-used benchmark datasets. The results show that DFauLo outperforms the state-of-the-art techniques in almost all cases and locates hundreds of different types of real data faults in benchmark datasets.
@inproceedings{Yin0WLYZZ023,
title = {Dynamic Data Fault Localization for Deep Neural Networks},
author = {Yining Yin and Yang Feng and Shihao Weng and Zixi Liu and Yuan Yao and Yichi Zhang and Zhihong Zhao and Zhenyu Chen},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616345}
}