一句话：提出基于深度学习的计算机视觉模型，从屏幕录像中识别用户动作的命令、控件和位置，以支持 UI 自动化脚本生成。
UI 自动化可用于 UI 测试、缺陷复现和 RPA，但现有录制用户动作的技术往往具有侵入性，依赖操作系统/GUI 框架辅助功能，或假设特定应用实现。从屏幕录像逆向工程用户动作虽非侵入式，但缺少关键一步：从动作录像中识别人类可理解的结构化用户动作（命令、控件、位置）。为此，本文提出 SeeAction：一个基于深度学习的计算机视觉模型，通过联合学习和多任务学习识别 11 种命令和 11 种控件，并生成位置短语。作者标注了包含 7260 个视频-动作对的数据集，覆盖 Word、Zoom、Firefox、Photoshop 和 Windows 10 设置中的交互。大量实验验证了模型的有效性和通用性，并展示了基于该模型的屏幕录像到动作脚本工具在缺陷复现中的实用性。
UI automation is an useful technique for UI testing, bug reproduction and robotic process automation. Recording the user actions with an application assists rapid development of UI automation scripts, but existing recording techniques are intrusive, rely on OS or GUI framework accessibility support or assume specific app implementations. Reverse engineering user actions from screencasts is non-intrusive, but a key reverse-engineering step is currently missing - recognize human-understandable structured user actions ([command] [widget] [location]) from action screencasts. To fill the gap, we propose a deep learning based computer vision model which can recognize 11 commands and 11 widgets, and generate location phrases from action screencasts, through joint learning and multi-task learning. We label a large dataset with 7260 video-action pairs, which record the user interactions with Word, Zoom, Firefox, Photoshop and Windows 10 Settings. Through extensive experiments, we confirm the effectiveness and generality of our model, and demonstrate the usefulness of a screencast-to-action-script tool built upon our model for bug reproduction.
DOI 原文 ·
@inproceedings{paperbot8835,
title = {SeeAction: Towards Reverse Engineering How-What-Where of HCI Actions from Screencasts for UI Automation},
author = {Dehai Zhao and Zhenchang Xing and Qinghua Lu and Xiwei Xu and Liming Zhu},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00144}
}