一句话：揭示 DNN 测试中错误触发输入（ETI）的陷阱，提出以错误的输入变换属性（EP）为导向的测试与修复新范式。
该文揭示神经网络（DNN）测试中错误触发输入（ETI）的两大陷阱：一味追求更多 ETI 会使测试陷入局部平台，只在少数固定输入变换下反复生成相似样本；用 ETI 微调虽能修复已暴露的误判，却削弱 DNN 对特定输入变换的鲁棒性。由于以往研究使用的输入变换通常不足 10 种，这些问题长期被忽视。作者将测试流水线扩充为大规模输入变换集合，并以人类研究验证其正确性；进而提出以“错误的输入变换属性”（EP）为中心的视角：记录哪种输入变换能生成 ETI，而非统计单个触发输入。实验表明，EP 导向的测试显著扩展了探索到的错误空间，用 EP 微调也能有效提升 DNN 对多种输入变换的鲁棒性。
Recent efforts in deep neural network (DNN) testing commonly use error-triggering inputs (ETIs) to quantify DNN errors and to fine-tune the tested DNN for repairing. This study reveals the pitfalls of ETIs in DNN testing. Specifically, merely seeking for more ETIs “traps” the testing campaign into local plateaus, where similar ETIs are continuously generated using a few fixed input transformations. Similarly, fine-tuning the DNN with ETIs, while capable of fixing the exposed DNN mis-predictions, undermines the DNN’s resilience towards certain input transformations. However, these ETI-induced pitfalls have been overlooked in previous research, due to the insufficient input transformations (usually < 10), and we show that the severity of such deceptive phenomena is enlarged when testing DNNs with more and diverse real-life input transformations. This paper presents a comprehensive study on the pitfalls of ETIs in DNN testing. We first augment conventional DNN testing pipelines with a large set of input transformations; the correctness and validity of these new transformations are verified with large-scale human studies. Based on this, we show that launching an endless pursuit for ETIs cannot alleviate the “trapped testing” issue, and the undermined resilience pervasively occurs in many input transformations. Accordingly, we propose a novel and holistic viewpoint over DNN errors: instead of counting which input triggers a DNN mis-prediction, we record which input transformation can generate ETIs. The targeted input property of this transformation, termed erroneous property (EP), counts one DNN error and guides DNN testing (i.e., our new paradigm aims to find more EPs rather than ETIs). Evaluation shows that this EP-oriented testing paradigm significantly expands the explored DNN error space. Moreover, fine-tuning DNNs with EPs effectively improves their resilience towards different input transformations.
DOI 原文 ·
@inproceedings{paperbot8585,
title = {See the Forest, not Trees: Unveiling and Escaping the Pitfalls of Error-Triggering Inputs in Neural Network Testing},
author = {Yuanyuan Yuan and Shuai Wang and Zhendong Su},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680385}
}