一句话：大规模众包实验表明，程序表示（Python vs 自然文本）和提示类型对用户找 bug、修 bug 的准确率有不同影响。
随着 AI 编程助手普及，终端用户也可用自然语言文本表达和求解编程任务，但算法理解与调试仍然困难。为探究提示如何帮助用户发现并修复 bug，作者开展了一项涉及 753 名参与者的大规模众包研究，交叉考察三种提示类型（测试用例、概念性、详细提示）、两种程序表示（Python 源代码与自然文本表示）以及两类用户（对算法任务理解清晰或存在困惑）。结果表明：程序表示显著影响找 bug 与修 bug 的准确率，且出乎意料地，自然文本表示下用户准确率更高；提示总体上能提升准确率，但不同提示的效果取决于表示形式与用户对任务的理解程度。这些发现为设计能根据用户技能和理解水平自适应选择编程模态与提示的下一代个性化编程工具提供了启示。
With the recent advances in AI programming assistants such as GitHub Copilot, programming is not limited to classical programming languages anymore-programming tasks can also be expressed and solved by end-users in natural text. Despite the availability of this new programming modality, users still face difficulties with algorithmic understanding and program debugging. One promising approach to support end-users is to provide hints to help them find and fix bugs while forming and improving their programming capabilities. While it is plausible that hints can help, it is unclear which type of hint is helpful and how this depends on program representations (classic source code or a textual representation) and the user's capability of understanding the algorithmic task. To understand the role of hints in this space, we conduct a large-scale crowd-sourced study involving 753 participants investigating the effect of three types of hints (test cases, conceptual, and detailed), across two program representations (Python and text-based), and two groups of users (with clear understanding or confusion about the algorithmic task). We find that the program representation (Python vs. text) has a significant influence on the users' accuracy at finding and fixing bugs. Surprisingly, users are more accurate at finding and fixing bugs when they see the program in natural text. Hints are generally helpful in improving accuracy, but different hints help differently depending on the program representation and the user's understanding of the algorithmic task. These findings have implications for designing next-generation programming tools that provide personalized support to users, for example, by adapting the programming modality and providing hints with respect to the user's skill level and understanding.
DOI 原文 ·
@inproceedings{paperbot8859,
title = {Hints Help Finding and Fixing Bugs Differently in Python and Text-Based Program Representations},
author = {Ruchit Rawal and Victor-Alexandru Pădurean and Sven Apel and Adish Singla and Mariya Toneva},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00192}
}