一句话：提出黑盒高效测试方法 BET，通过连续扰动与可调目标在查询预算内发现 CNN 的致错输入。
在安全敏感场景部署卷积神经网络（CNN）前，发现致错输入等缺陷至关重要。现有白盒测试虽能实现高神经元覆盖，但依赖模型内部信息，不适用于隐私敏感的黑盒场景。本文提出黑盒高效测试方法 BET。其核心洞察是 CNN 通常易受连续扰动影响；BET 在黑盒下生成连续扰动，并用可调目标函数引导测试，系统探索目标 CNN 不同决策边界附近的缺陷。为在固定查询预算下找到更多致错输入，作者提出以效率为中心的策略。基于三个常用数据集和五种流行 CNN 结构的实验表明，在固定查询/推理预算下，BET 找到的有效致错输入显著多于现有白盒和黑盒方法；这些输入还可微调目标模型，使准确率最多提升 3%。
It is important to test convolutional neural networks (CNNs) to identify defects (e.g. error-inducing inputs) before deploying them in security-sensitive scenarios. Although existing white-box testing methods can effectively test CNN models with high neuron coverage, they are not applicable to privacy-sensitive scenarios where full knowledge of target CNN models is lacking. In this work, we propose a novel Black-box Efficient Testing (BET) method for CNN models. The core insight of BET is that CNNs are generally prone to be affected by continuous perturbations. Thus, by generating such continuous perturbations in a black-box manner, we design a tunable objective function to guide our testing process for thoroughly exploring defects in different decision boundaries of the target CNN models. We further design an efficiency-centric policy to find more error-inducing inputs within a fixed query budget. We conduct extensive evaluations with three well-known datasets and five popular CNN structures. The results show that BET significantly outperforms existing white-box and black-box testing methods considering the effective error-inducing inputs found in a fixed query/inference budget. We further show that the error-inducing inputs found by BET can be used to fine-tune the target model, improving its accuracy by up to 3%.
@inproceedings{paperbot7429,
title = {BET: black-box efficient testing for convolutional neural networks},
author = {Jialai Wang and Han Qiu and Yi Rong and Hengkai Ye and Qi Li and Zongpeng Li and Chao Zhang},
booktitle = {ISSTA},
year = {2022},
doi = {10.1145/3533767.3534386}
}