一句话：提出 Aries，利用决策边界距离估计 DNN 在无标签数据上的准确率，实现高效测试。
深度神经网络在工业应用中广泛使用，但其质量评估通常依赖带标签测试数据，而数据标注成本高昂。现有测试选择方法仍需人工标注，难以自动化。本文提出 Aries，一种仅利用原始测试数据信息来估计 DNN 在新无标签数据上性能的技术。其核心洞察是：与决策边界距离相似的数据，模型应具有相似的预测准确率。作者在 CIFAR-10 和 Tiny-ImageNet 两个数据集、四种常用 DNN 模型以及 13 种数据变换方法上进行了大规模评估。结果表明，Aries 估计准确率与真实准确率仅相差 0.03% 至 2.60%；在 52 个案例中的 50 个优于当前最优无标签方法，在 128 个案例中的 96 个优于基于选择标注的方法。
Deep learning (DL) plays a more and more important role in our daily life due to its competitive performance in industrial application domains. As the core of DL-enabled systems, deep neural networks (DNNs) need to be carefully evaluated to ensure the produced models match the expected requirements. In practice, the de facto standard to assess the quality of DNNs in the industry is to check their performance (accuracy) on a collected set of labeled test data. However, preparing such labeled data is often not easy partly because of the huge labeling effort, i.e., data labeling is labor-intensive, especially with the massive new incoming unlabeled data every day. Recent studies show that test selection for DNN is a promising direction that tackles this issue by selecting minimal representative data to label and using these data to assess the model. However, it still requires human effort and cannot be automatic. In this paper, we propose a novel technique, named Aries, that can estimate the performance of DNNs on new unlabeled data using only the information obtained from the original test data. The key insight behind our technique is that the model should have similar prediction accuracy on the data which have similar distances to the decision boundary. We performed a large-scale evaluation of our technique on two famous datasets, CIFAR-10 and Tiny-ImageNet, four widely studied DNN models including ResNetl0l and DenseNetl21, and 13 types of data transformation methods. Results show that the estimated accuracy by Aries is only 0.03% - 2.60% off the true accuracy. Besides, Aries also outperforms the state-of-the-art labeling-free methods in 50 out of 52 cases and selection-labeling-based methods in 96 out of 128 cases.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{HuGXCPMT23,
title = {Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimation},
author = {Qiang Hu and Yuejun Guo and Xiaofei Xie and Maxime Cordy and Mike Papadakis and Lei Ma and Yves Le Traon},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00152}
}