一句话：通过 3000 万条被移除评论的大规模纵向研究，归纳 iOS App Store 违规评论成因，并实现 F1=97% 的自动识别方法。
应用商店评论本应帮助开发者和用户决策，但虚假、低质和违规评论泛滥，市场维护者只能定期被动删除，学术界却缺乏对已移除评论的整体理解。本文与工业伙伴合作，收集 2020 全年 33,665 个热门 iOS 应用的 3000 多万条被移除评论，开展大规模纵向研究。作者刻画其整体特征，归纳出虚假评论、冒犯性评论等删除原因，并发现多数不当行为可从发布者、文本内容和发布时间等基本信息中反映。基于此，他们设计自动标记违反政策评论的方法。在标注基准上，该方法达到 F1=97%，并在大规模工业数据上验证了应用前景，有望作为事前把关者改进评论维护流程。
The app markets enable users to submit feedback for downloaded apps in the form of star ratings and text reviews, which are meant to be helpful and trustworthy for decision making to both developers and other users. App markets have released strict guidelines/policies for user review submissions. However, there has been growing evidence showing the untrustworthy and poor-quality of app reviews, making the app store review environment a shambles. Therefore, review removal is a common practice, and market maintainers have to remove undesired reviews from the market periodically in a reactive manner. Although some reports and news outlets have mentioned removed reviews, our research community still lacks the comprehensive understanding of the landscape of this kind of reviews. To fill the void, in this paper, we present a large-scale and longitudinal study of removed reviews in iOS App Store. We first collaborate with our industry partner to collect over 30 million removed reviews for 33,665 popular apps over the course of a full year in 2020. This comprehensive dataset enables us to characterize the overall landscape of removed reviews. We next investigate the practical reasons leading to the removal of policy-violating reviews, and summarize several interesting reasons, including fake reviews, offensive reviews, etc. More importantly, most of these mis-behaviors can be reflected on reviews’ basic information including the posters, narrative content, and posting time. It motivates us to design an automated approach to flag the policy-violation reviews, and our experiment result on the labelled benchmark can achieve a good performance (F1=97%). We further make an attempt to apply our approach to the large-scale industry setting, and the result suggests the promising industry usage scenario of our approach. Our approach can act as a gatekeeper to pinpoint policy-violation reviews beforehand, which will be quite effective in improving the maintenance process of app reviews in the industrial setting.
DOI 原文 ·
@inproceedings{paperbot7055,
title = {Demystifying “removed reviews” in iOS app store},
author = {Liu Wang and Haoyu Wang and Xiapu Luo and Tao Zhang and Shangguang Wang and Xuanzhe Liu},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3558966}
}