一句话：提出分布引导的测试套件优化方法，生成分布多样且有效的测试用例，以评估并增强深度学习模型鲁棒性。
深度学习模型在采样数据上训练，训练数据与真实数据存在分布偏移，会降低模型鲁棒性。现有测试方法中，分布无关测试未显式考虑测试用例分布，容易产生同一分布内的冗余错误；分布感知测试主要关注符合训练分布的用例，忽略可能有效且应被测试的分布外数据。本文提出 DistXplore，一种分布引导的测试方法。不同于逐例优化，DistXplore 以代表特定分布的测试套件为优化对象。为评估和增强鲁棒性，作者设计两个度量：分布差异最大化不同类别数据间的分布相似性，以生成难以检测的错误；分布多样性提高生成用例的分布多样性，以增强模型鲁棒性。在4个数据集、10个模型上与14个先进基线比较，DistXplore 发现错误数平均达2倍以上，经验鲁棒性提升平均比基线高5.2%准确率。
Deep learning (DL) models are trained on sampled data, where the distribution of training data differs from that of real-world data (i.e., the distribution shift), which reduces the model's robustness. Various testing techniques have been proposed, including distribution-unaware and distribution-aware methods. However, distribution-unaware testing lacks effectiveness by not explicitly considering the distribution of test cases and may generate redundant errors (within same distribution). Distribution-aware testing techniques primarily focus on generating test cases that follow the training distribution, missing out-of-distribution data that may also be valid and should be considered in the testing process. In this paper, we propose a novel distribution-guided approach for generating valid test cases with diverse distributions, which can better evaluate the model's robustness (i.e., generating hard-to-detect errors) and enhance the model's robustness (i.e., enriching training data). Unlike existing testing techniques that optimize individual test cases, DistXplore optimizes test suites that represent specific distributions. To evaluate and enhance the model's robustness, we design two metrics: distribution difference, which maximizes the similarity in distribution between two different classes of data to generate hard-to-detect errors, and distribution diversity, which increase the distribution diversity of generated test cases for enhancing the model's robustness. To evaluate the effectiveness of DistXplore in model evaluation and enhancement, we compare DistXplore with 14 state-of-the-art baselines on 10 models across 4 datasets. The evaluation results show that DisXplore not only detects a larger number of errors (e.g., 2×+ on average). Furthermore, DistXplore achieves a higher improvement in empirical robustness (e.g., 5.2% more accuracy improvement than the baselines on average).
@inproceedings{WangX0T0Y023,
title = {DistXplore: Distribution-Guided Testing for Evaluating and Enhancing Deep Learning Systems},
author = {Longtian Wang and Xiaofei Xie and Xiaoning Du and Meng Tian and Qing Guo and Zheng Yang and Chao Shen},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616266}
}