一句话：通过19名数据科学家的实验，探索 IDE 中基于手绘草图与 LLM 生成代码的视觉代码助手，并评估其可用性与生成质量。
论文关注 IDE 中视觉代码助手如何弥合白板草图与代码之间的鸿沟。作者实现了一个原型工具，将开发者在设计 ML 工作流时绘制的草图发送给具备视觉能力的大语言模型，生成 Python notebook。通过对 19 名数据科学家的实验，分析其草图模式与心智模型，发现图示最常见（52.6%），常配合列表和编号点。用 LLM-as-judge 评估生成代码质量，表明简短草图也能产生有用的代码大纲，且草图时间与生成质量正相关。访谈显示该助手在教育、原型和协作场景有前景。
This paper explores the integration of Visual Code Assistants in Integrated Development Environments (IDEs). In Software Engineering, whiteboard sketching is often the initial step before coding, serving as a crucial collaboration tool for developers. Previous studies have investigated patterns in SE sketches and how they are used in practice, yet methods for directly using these sketches for code generation remain limited. The emergence of visually-equipped large language models presents an opportunity to bridge this gap, which is the focus of our research. In this paper, we built a first prototype of a Visual Code Assistant to get user feedback regarding in-IDE sketch-to-code tools. We conduct an experiment with 19 data scientists, most of whom regularly sketch as part of their job. We investigate developers' mental models by analyzing patterns commonly observed in their sketches when developing an ML workflow. Analysis indicates that diagrams were the preferred organizational component (52.6%), often accompanied by lists (42.1%) and numbered points (36.8%). Our tool converts their sketches into a Python notebook by querying an LLM. We use an LLM-as-judge setup to score the quality of the generated code, finding that even brief sketching can effectively generate useful code outlines. We also find a positive correlation between sketch time and the quality of the generated code. We conclude the study by conducting extensive interviews to assess the tool's usefulness, explore potential use cases, and understand developers' needs. As noted by participants, promising applications for these assistants include education, prototyping, and collaborative settings. Our findings signal promise for the next generation of Code Assistants to integrate visual information, both to improve code generation and to better leverage developers' existing sketching practices.
DOI 原文 ·
@inproceedings{paperbot8745,
title = {An Exploratory Study of ML Sketches and Visual Code Assistants},
author = {Luís F. Gomes and Vincent J. Hellendoorn and Jonathan Aldrich and Rui Abreu},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00124}
}