一句话：提出 AidUI，利用计算机视觉与 NLP 从截图自动检测、分类并定位十类 UI 暗黑模式。
UI 暗黑模式（Dark Patterns）通过欺骗性界面设计诱导用户做出非本意操作，可能造成隐私过度分享或经济损失；但由于同一模式实例形式多样，自动识别十分困难，开发者和用户也缺乏识别与规避指导。本文提出 AidUI，一种结合计算机视觉与自然语言处理的方法，从应用截图中识别视觉与文本线索，以检测、分类并定位十类 UI 暗黑模式。作者构建了 ContextDP 数据集，包含 175 张移动端和 83 张 Web 端 UI 截图、301 个暗黑模式实例，并带有完整定位标注。评估显示 AidUI 检测的总体 precision 0.66、recall 0.67、F1 0.65，误报较少，定位 IoU 0.84；部分模式可可靠检测（F1 超过 0.82）。
Past studies have illustrated the prevalence of UI dark patterns, or user interfaces that can lead end-users toward (unknowingly) taking actions that they may not have intended. Such deceptive UI designs can be either intentional (to benefit an online service) or unintentional (through complicit design practices) and can result in adverse effects on end users, such as oversharing personal information or financial loss. While significant research progress has been made toward the development of dark pattern taxonomies across different software domains, developers and users currently lack guidance to help recognize, avoid, and navigate these often subtle design motifs. However, automated recognition of dark patterns is a challenging task, as the instantiation of a single type of pattern can take many forms, leading to significant variability. In this paper, we take the first step toward understanding the extent to which common UI dark patterns can be automatically recognized in modern software applications. To do this, we introduce AidUI, a novel automated approach that uses computer vision and natural language processing techniques to recognize a set of visual and textual cues in application screenshots that signify the presence of ten unique UI dark patterns, allowing for their detection, classification, and localization. To evaluate our approach, we have constructed ContextDP, the current largest dataset of fully-localized UI dark patterns that spans 175 mobile and 83 web UI screenshots containing 301 dark pattern instances. The results of our evaluation illustrate that AidUI achieves an overall precision of 0.66, recall of 0.67, F1-score of 0.65 in detecting dark pattern instances, reports few false positives, and is able to localize detected patterns with an IoU score of 0.84. Furthermore, a significant subset of our studied dark patterns can be detected quite reliably (F1 score of over 0.82), and future research directions may allow for improved detection of additional patterns. This work demonstrates the plausibility of developing tools to aid developers in recognizing and appropriately rectifying deceptive UI patterns.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2303-06782,
title = {AidUI: Toward Automated Recognition of Dark Patterns in User Interfaces},
author = {S M Hasan Mansur and Sabiha Salma and Damilola Awofisayo and Kevin Moran},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00166}
}