一句话：提出黑盒测试框架 ATOM，以标签组合为准则并借助图像搜索、NLP 和蜕变测试检测多标签图像分类错误。
多标签图像分类系统（MICS）广泛用于日常场景，但现有 DNN 测试多依赖模型内部信息，未针对其“识别多个对象”的核心规约。本文提出 ATOM：以标签组合为测试充分性准则，系统考察标签间相关性对分类能力的影响；利用图像搜索引擎与 NLP 寻找真实且关联目标标签组合的图像；结合蜕变测试与标签信息识别测试预言。VOC 和 COCO（各五个 DNN 模型）及工业照片标注应用的实验表明，即使仅两标签相关性，现有 MICS 仍不理想，共触发 6,049 个相关错误；工业系统上 587 张错误暴露图像中 92% 获开发者确认。
Multi-label Image Classification Systems (MICSs) developed based on Deep Neural Networks (DNNs) are extensively used in people's daily life. Currently, although there are a variety of approaches to test DNN-based systems, they typically rely on the internals of DNNs to design test cases, and do not take the core specification of MICS (i.e., correctly recognizing multiple objects in a given image) into account. In this paper, we propose ATOM, an automated and systematic black-box testing framework for testing MICS. Specifically, ATOM exploits the label combination as the testing adequacy criteria, hoping to systematically examine the impact of correlations between a fixed number of labels on the classification ability of MICS. Then, ATOM leverages image search engine and natural language processing to find test images that are not only common to the real-world, but also relevant to target label combinations. Finally, ATOM combines metamorphic testing and label information to realize test oracle identification, based on which the ability of MICS in classifying different label combinations is evaluated. To evaluate the effectiveness of ATOM, we have performed experiments on two popular datasets of MICS, VOC and COCO (each with five state-of-the-art DNN models), and one real-world photo tagging application from our industrial partner. The experimental results reveal that the performance of current DNN-based MICSs remains less satisfactory even in recognizing correlations between only two labels, as ATOM triggers a total number of 6,049 such label combination related errors for all MICSs studied. In particular, ATOM reports 587 error-revealing images for the industrial MICS, in which 92% of them are confirmed by the developers.
@inproceedings{HuWWCTJNN23,
title = {ATOM: Automated Black-Box Testing of Multi-Label Image Classification Systems},
author = {Shengyou Hu and Huayao Wu and Peng Wang and Jing Chang and Yongjun Tu and Xiu Jiang and Xintao Niu and Changhai Nie},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00156}
}