一句话：实验表明提升神经元覆盖率反而降低缺陷检测、输入自然性与预测中立性，质疑其作为测试目标的合理性。
受传统代码覆盖率启发，神经元覆盖率（NC）被广泛用作深度学习系统的测试充分性准则，并隐含假定提高 NC 能改善测试套件质量。为检验这一假设，作者设计了一种可插入现有对抗攻击算法的多样性促进正则化器，用以自动生成提高 NC 的测试套件。随后从三个维度评估这些测试套件：(1) 是否能成功检测对抗攻击，(2) 是否产生自然输入，(3) 是否对特定类别预测存在偏好。大量实验得出的结论与预期相悖：提高 NC 反而使测试套件更难有效，表现为检测到的缺陷更少、输入更不自然、预测偏好更偏斜。该结果对将 NC 作为深度神经网络测试生成目标提出质疑，并呼吁同时考虑缺陷检测、输入自然性与输出中立性的新测试生成技术。
Recent effort to test deep learning systems has produced an intuitive and compelling test criterion called neuron coverage (NC), which resembles the notion of traditional code coverage. NC measures the proportion of neurons activated in a neural network and it is implicitly assumed that increasing NC improves the quality of a test suite. In an attempt to automatically generate a test suite that increases NC, we design a novel diversity promoting regularizer that can be plugged into existing adversarial attack algorithms. We then assess whether such attempts to increase NC could generate a test suite that (1) detects adversarial attacks successfully, (2) produces natural inputs, and (3) is unbiased to particular class predictions. Contrary to expectation, our extensive evaluation finds that increasing NC actually makes it harder to generate an effective test suite: higher neuron coverage leads to fewer defects detected, less natural inputs, and more biased prediction preferences. Our results invoke skepticism that increasing neuron coverage may not be a meaningful objective for generating tests for deep neural networks and call for a new test generation technique that considers defect detection, naturalness, and output impartiality in tandem.
@inproceedings{paperbot5849,
title = {Is neuron coverage a meaningful measure for testing deep neural networks?},
author = {Fabrice Harel-Canada and Lingxiao Wang and Muhammad Ali Gulzar and Quanquan Gu and Miryung Kim},
booktitle = {FSE},
year = {2020},
doi = {10.1145/3368089.3409754}
}