一句话：提出视觉语义学习VSL，从软件视觉输出中借助图像/视频理解与自然语言生成自动推导高层功能描述。
论文设想一种新的软件理解方法——视觉语义学习（VSL）。其问题在于：自动理解软件行为是实现全自动软件工程的前提，但现有方法多依赖代码或运行日志，而非界面等视觉输出。VSL 从软件产生的图像/视频输出中提取视觉元素标签，再通过图像/视频理解与自然语言生成组合成关于软件行为的高层语义描述。作者给出了 VSL 的设计框架，并报告初步结果以展示其可行性；同时指出该方法仅适用于会产生有意义视觉输出的软件，但可作为迈向自动化软件理解的第一步。
We envision visual semantics learning (VSL), a novel methodology that derives high-level functional description of given software from its visual (graphical) outputs. By visual semantics, we mean the semantic description about the software’s behaviors that are exhibited in its visual outputs. VSL works by composing this description based on visual element labels extracted from these outputs through image/video understanding and natural language generation. The result of VSL can then support tasks that may benefit from the high-level functional description. Just like a developer relies on program understanding to conduct many of such tasks, automatically understanding software (i.e., by machine rather than by human developers) is necessary to eventually enable fully automated software engineering. Apparently, VSL only works with software that does produce visual outputs that meaningfully demonstrate the software’s behaviors. Nevertheless, learning visual semantics would be a useful first step towards automated software understanding. We outline the design of our approach to VSL and present early results demonstrating its merits.