一句话：提出SeeHow工具，利用计算机视觉从编程录屏中自动提取代码行编辑步骤与代码片段，生成可理解的工作流。
编程录屏（如 YouTube 教程、Twitch 直播）是开发者学习编程知识与任务工作流的重要来源，但其图像本质限制了内容与工作流的可访问性。现有非侵入式方法只能提取原始人机交互动作或粗粒度视频片段。本文提出 SeeHow，利用计算机视觉分析编程录屏，自动抽取代码行编辑步骤（输入、删除、编辑、选择文本），并输出编码步骤序列及每步代码片段，即编程工作流。在 41 小时教程视频与直播录屏上评估，覆盖多种编程环境；结果表明工具能准确抽取代码行编辑步骤，且工作流步骤可被开发者直观理解。
Programming screencasts (e.g., video tutorials on Youtube or live coding stream on Twitch) are important knowledge source for developers to learn programming knowledge, especially the workflow of completing a programming task. Nonetheless, the image nature of programming screencasts limits the accessibility of screencast content and the workflow embedded in it, resulting in a gap to access and interact with the content and workflow in programming screencasts. Existing non-intrusive methods are limited to extract either primitive human-computer interaction (HCI) actions or coarse-grained video fragments. In this work, we leverage Computer Vision (CV) techniques to build a programming screencast analysis tool which can automatically extract code-line editing steps (enter text, delete text, edit text and select text) from screencasts. Given a programming screencast, our approach outputs a sequence of coding steps and code snippets involved in each step, which we refer to as programming workflow. The proposed method is evaluated on 41 hours of tutorial videos and live coding screencasts with diverse programming environments. The results demonstrate our tool can extract code-line editing steps accurately and the extracted workflow steps can be intuitively understood by developers.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{ZhaoXXYXZ23,
title = {SeeHow: Workflow Extraction from Programming Screencasts through Action-Aware Video Analytics},
author = {Dehai Zhao and Zhenchang Xing and Xin Xia and Deheng Ye and Xiwei Xu and Liming Zhu},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00165}
}