一句话：提出蜕变测试框架 MTTM，通过字符、词、句级扰动生成对抗性有毒文本，发现并提升文本审核软件的鲁棒性。
社交媒体有害内容治理依赖文本内容审核软件，但恶意用户仅改动少量词即可规避审核，现代审核软件的鲁棒性仍缺乏研究。本文提出蜕变测试框架 MTTM：先基于 2000 条真实文本总结字符、词语和句子三个层次的 11 条蜕变关系，再用其生成仍有害但可能规避审核的测试用例。评估测试了 Google、百度、华为三款商业软件和两种前沿算法，覆盖三类有害内容；商业软件的发现错误率最高分别达 83.9%、51%、82.5%，前沿算法最高达 91.2%。用生成用例重训练模型还可提升鲁棒性 0%～5.9% EFR，并保持原始测试集准确率。
The exponential growth of social media platforms such as Twitter and Facebook has revolutionized textual communication and textual content publication in human society. However, they have been increasingly exploited to propagate toxic content, such as hate speech, malicious advertisement, and pornography, which can lead to highly negative impacts (e.g., harmful effects on teen mental health). Researchers and practitioners have been enthusiastically developing and extensively deploying textual content moderation software to address this problem. However, we find that malicious users can evade moderation by changing only a few words in the toxic content. Moreover, modern content moderation software's performance against malicious inputs remains underexplored. To this end, we propose MTTM, a Metamorphic Testing framework for Textual content Moderation software. Specifically, we conduct a pilot study on 2, 000 text messages collected from real users and summarize eleven metamorphic relations across three perturbation levels: character, word, and sentence. MTTM employs these metamorphic relations on toxic textual contents to generate test cases, which are still toxic yet likely to evade moderation. In our evaluation, we employ MTTM to test three commercial textual content moderation software and two state-of-the-art moderation algorithms against three kinds of toxic content. The results show that MTTM achieves up to 83.9%, 51%, and 82.5% error finding rates (EFR) when testing commercial moderation software provided by Google, Baidu, and Huawei, respectively, and it obtains up to 91.2% EFR when testing the state-of-the-art algorithms from the academy. In addition, we leverage the test cases generated by MTTM to retrain the model we explored, which largely improves model robustness 0% ~ 5.9% EFR) while maintaining the accuracy on the original test set. A demo can be found in this link1.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2302-05706,
title = {MTTM: Metamorphic Testing for Textual Content Moderation Software},
author = {Wenxuan Wang and Jen-tse Huang and Weibin Wu and Jianping Zhang and Yizhan Huang and Shuqing Li and Pinjia He and Michael R. Lyu},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00200}
}