一句话:在IDE中构建无需显式提示的LLM对话界面,用户研究证明其辅助代码理解优于网络搜索。
理解代码在新颖复杂的开发环境中尤为困难,而代码注释和文档往往稀缺且难以检索。本文首次研究了直接内置于IDE、面向代码理解的大语言模型对话式界面。该插件在用户无需编写显式提示的情况下,向OpenAI GPT-3.5-turbo模型发起四类高层请求:解释高亮代码片段、提供代码中API调用的细节、解释关键领域术语,以及为API提供使用示例;同时支持开放式提示,并自动将正在编辑的程序作为上下文。作者通过32名参与者的用户研究进行评估,结果证实该插件在辅助任务完成方面优于网络搜索。文章还深入分析了开发者使用该系统的方式及其感知有用性,发现学生与专业开发者的使用方式和收益存在差异。结论是,IDE内无提示的LLM交互是工具构建者一个很有前景的未来方向。
Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.