一句话：提出递归对象熔化（ROME）蜕变测试方法，通过移除并修复图像对象生成自然测试对，有效检测图像描述系统错误。
图像描述（IC）系统会为图像显著对象生成文本描述，但先进系统也可能产生错误描述；现有测试方法要么难以处理自然语言输出，要么生成不自然图像。本文提出递归对象熔化（ROME）蜕变测试方法：通过递归移除并修复（inpaint）图像对象生成多对自然图像，并假设熔化后生成图像描述的对象集合应包含于原图描述的对象集合。作者测试一个广泛使用的 IC API 和四个 SOTA 算法，在 226 张种子图像上报告 9,121 个错误，精度 86.47%–92.17%，用例自然度接近原图；用其重训练 Oscar 还提升了多个指标。
Image captioning (IC) systems aim to generate a text description of the salient objects in an image. In recent years, IC systems have been increasingly integrated into our daily lives, such as assistance for visually-impaired people and description generation in Microsoft Powerpoint. However, even the cutting-edge IC systems (e.g., Microsoft Azure Cognitive Services) and algorithms (e.g., OFA) could produce erroneous captions, leading to incorrect captioning of important objects, misunderstanding, and threats to personal safety. The existing testing approaches either fail to handle the complex form of IC system output (i.e., sentences in natural language) or generate unnatural images as test cases. To address these problems, we introduce Recursive Object MElting (ROME), a novel metamorphic testing approach for validating IC systems. Different from existing approaches that generate test cases by inserting objects, which easily make the generated images unnatural, ROME melts (i.e., remove and inpaint) objects. ROME assumes that the object set in the caption of an image includes the object set in the caption of a generated image after object melting. Given an image, ROME can recursively remove its objects to generate different pairs of images. We use ROME to test one widely-adopted image captioning API and four state-of-the-art (SOTA) algorithms. The results show that the test cases generated by ROME look much more natural than the SOTA IC testing approach and they achieve comparable naturalness to the original images. Meanwhile, by generating test pairs using 226 seed images, ROME reports a total of 9,121 erroneous issues with high precision (86.47%-92.17%). In addition, we further utilize the test cases generated by ROME to retrain the Oscar, which improves its performance across multiple evaluation metrics.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2306-02228,
title = {ROME: Testing Image Captioning Systems via Recursive Object Melting},
author = {Boxi Yu and Zhiqing Zhong and Jiaqi Li and Yixing Yang and Shilin He and Pinjia He},
booktitle = {ISSTA},
year = {2023},
doi = {10.1145/3597926.3598094}
}