一句话：大规模复现表明，简单的不确定性量化基线在 DNN 测试优先级排序与主动学习中与 DeepGini 表现相当。
深度神经网络测试集通常规模巨大，测试输入优先级排序（TIP）可减少计算与标注成本。该复现研究针对 DeepGini 这一快速简单的 TIP 方法，在 4 个案例研究、8 个测试数据集和 32200 个训练模型上进行大规模实验，验证其相对 neuron coverage 与 surprise coverage 的优势，并考察主动学习场景。结果确认 DeepGini 的有效性，但同时发现来自不确定性量化的更简单或相近基线，如预测 softmax 似然或其熵，与 DeepGini 表现相当。这表明简单技术已足够强，后续工作需谨慎选择更复杂方法。
Test Input Prioritizers (TIP) for Deep Neural Networks (DNN) are an important technique to handle the typically very large test datasets efficiently, saving computation and labelling costs. This is particularly true for large scale, deployed systems, where inputs observed in production are recorded to serve as potential test or training data for next versions of the system. Feng et. al. propose DeepGini, a very fast and simple TIP and show that it outperforms more elaborate techniques such as neuron- and surprise coverage. In a large-scale study (4 case studies, 8 test datasets, 32’200 trained models) we verify their findings. However, we also find that other comparable or even simpler baselines from the field of uncertainty quantification, such as the predicted softmax likelihood or the entropy of the predicted softmax likelihoods perform equally well as DeepGini
@inproceedings{paperbot7403,
title = {Simple techniques work surprisingly well for neural network test prioritization and active learning (replicability study)},
author = {Michael Weiss and Paolo Tonella},
booktitle = {ISSTA},
year = {2022},
doi = {10.1145/3533767.3534375}
}