一句话:提出首个深度学习蜕变测试用例对优先级排序方法,用神经元输出分布差异度量多样性,加速违规暴露。
由于能够绕过 oracle 问题,蜕变测试(MT)已成为深度学习(DL)软件测试的常用技术。然而,现有工作尚未关注蜕变测试用例对(MP)的优先级排序,而这对于提升 DL 测试中 MT 的有效性十分关键。若故障敏感的 MP 未能优先执行,已检测违规的揭示可能被大幅延迟甚至遗漏,从而掩盖关键缺陷。本文提出首个面向 DL 软件的 MP 优先级排序方法,以促进 DL 测试中已检测违规的揭示。具体地,作者基于神经元输出的分布差异设计了一种新的度量,用于衡量 DL 软件在 MP 上的执行多样性,并据此优先排序故障敏感 MP。综合评估结果表明,所提出的优先级排序方法和多样性度量能有效优先故障敏感 MP、提升已检测违规的揭示,甚至有助于为图像分类 DL 软件选择和设计有效的蜕变关系。
Due to the ability to bypass the oracle problem, Metamorphic Testing (MT) has been a popular technique to test deep learning (DL) software. However, no work has taken notice of the prioritization for Metamorphic test case Pairs (MPs), which is quite essential and beneficial to the effectiveness of MT in DL testing. When the fault-sensitive MPs apt to trigger violations and expose defects are not prioritized, the revealing of some detected violations can be greatly delayed or even missed to conceal critical defects. In this paper, we propose the first method to prioritize the MPs for DL software, so as to boost the revealing of detected violations in DL testing. Specifically, we devise a new type of metric to measure the execution diversity of DL software on MPs based on the distribution discrepancy of the neuron outputs. The fault-sensitive MPs are next prioritized based on the devised diversity metric. Comprehensive evaluation results show that the proposed prioritization method and diversity metric can effectively prioritize the fault-sensitive MPs, boost the revealing of detected violations, and even facilitate the selection and design of the effective Metamorphic Relations for the image classification DL software.