一句话：提出基于样本区分度的选择方法 SDS，用少量样本高效且准确地排序多个深度学习模型。
深度学习模型大量构建与共享，使同一任务常存在多个功能相同的候选模型，测试者需要在标注成本受限下比较并排序这些模型。本文提出 Sample Discrimination based Selection（SDS）方法，通过度量样本对多个模型预测行为（正确/错误）的区分能力，选择能揭示模型性能趋势的高效样本子集，从而提升多个深度学习模型比较测试中的排序精度。作者在三个常用图像数据集和80个真实世界深度学习模型上开展广泛实证研究，并与当前最先进的基线方法比较。结果表明，SDS 是一种有效且高效的样本选择方法，能够更好地用于多个深度学习模型的排序。
The boom of DL technology leads to massive DL models built and shared, which facilitates the acquisition and reuse of DL models. For a given task, we encounter multiple DL models available with the same functionality, which are considered as candidates to achieve this task. Testers are expected to compare multiple DL models and select the more suitable ones w.r.t. the whole testing context. Due to the limitation of labeling effort, testers aim to select an efficient subset of samples to make an as precise rank estimation as possible for these models. To tackle this problem, we propose Sample Discrimination based Selection (SDS) to select efficient samples that could discriminate multiple models, i.e., the prediction behaviors (right/wrong) of these samples would be helpful to indicate the trend of model performance. To evaluate SDS, we conduct an extensive empirical study with three widely-used image datasets and 80 real world DL models. The experiment results show that, compared with state-of-the-art baseline methods, SDS is an effective and efficient sample selection method to rank multiple DL models.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2103-04333,
title = {Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning Models},
author = {Linghan Meng and Yanhui Li and Lin Chen and Zhi Wang and Di Wu and Yuming Zhou and Baowen Xu},
booktitle = {ICSE},
year = {2021},
doi = {10.1109/icse43902.2021.00045}
}