一句话：提出 OASIS 蜕变测试框架，将有毒文本转为图像以规避内容审核，并评估与提升审核模型鲁棒性。
社交媒体中有毒内容泛滥，恶意用户将文本嵌入图像并加干扰以规避审核，但内容审核软件对此类输入的鲁棒性尚未被充分研究。本文提出 OASIS，一个面向内容审核软件的蜕变测试框架。作者从 Twitter、Instagram、新浪微博和百度贴吧的 5000 条真实有毒内容中总结出 21 条变换规则，能将有毒文本转换为保持毒性但可能绕过审核的图像测试用例。评估中，OASIS 测试了五家商业文本审核服务和一 SOTA 审核模型，错误发现率最高达 100%；用其生成用例重训练还能提升模型鲁棒性且不降低原有性能。
The exponential growth of social media platforms has brought about a revolution in communication and content dissemination in human society. Nevertheless, these platforms are being increasingly misused to spread toxic content, including hate speech, malicious advertising, and pornography, leading to severe negative consequences such as harm to teenagers' mental health. Despite tremendous efforts in developing and deploying textual and image content moderation methods, malicious users can evade moderation by embedding texts into images, such as screenshots of the text, usually with some interference. We find that modern content moderation software's performance against such malicious inputs remains underexplored. In this work, we propose OASIS, a metamorphic testing framework for content moderation software. OASIS employs 21 transform rules summarized from our pilot study on 5,000 real-world toxic contents collected from 4 popular social media applications, including Twitter, Instagram, Sina Weibo, and Baidu Tieba. Given toxic textual contents, OASIS can generate image test cases, which preserve the toxicity yet are likely to bypass moderation. In the evaluation, we employ OASIS to test five commercial textual content moderation software from famous companies (i.e., Google Cloud, Microsoft Azure, Baidu Cloud, Alibaba Cloud and Tencent Cloud), as well as a state-of-the-art moderation research model. The results show that OASIS achieves up to 100% error finding rates. Moreover, through retraining the models with the test cases generated by OASIS, the robustness of the moderation model can be improved without performance degradation.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2308-09810,
title = {An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software},
author = {Wenxuan Wang and Jingyuan Huang and Jen-tse Huang and Chang Chen and Jiazhen Gu and Pinjia He and Michael R. Lyu},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00189}
}