一句话：评估 GPT-3.5/GPT-4 回答由其自身生成代码所衍生的程序理解问题，发现其会犯与新手类似的错误。
近期研究能从学生代码中通过程序分析提取执行路径，并据此生成关于学习者代码的问题（QLC），以支持程序理解学习。与此同时，大型语言模型在入门编程中展现出生成程序与解答编程题的潜力。本文探究当前先进 LLM（GPT-3.5 与 GPT-4）回答 QLC 的能力，这些 QLC 来自模型自己生成的代码。方法上，先让 LLM 生成程序，再基于程序分析生成代码理解问题，随后提示模型作答并分析其表现。结果显示，尽管模型能生成程序并在提示下跟踪执行，但在回答程序理解问题时仍会犯新手程序员常犯的错误。这表明这些模型并不可靠，也提示可利用其类似学生的行为来模拟学生，服务于编程教育研究。
Recent research has explored the creation of questions from code submitted by students. These Questions about Learners' Code (QLCs) are created through program analysis, exploring execution paths, and then creating code comprehension questions from these paths and the broader code structure. Responding to the questions requires reading and tracing the code, which is known to support students' learning. At the same time, computing education researchers have witnessed the emergence of Large Language Models (LLMs) that have taken the community by storm. Researchers have demonstrated the applicability of these models especially in the introductory programming context, outlining their performance in solving introductory programming problems and their utility in creating new learning resources. In this work, we explore the capability of the state-of-the-art LLMs (GPT-3.5 and GPT-4) in answering QLCs that are generated from code that the LLMs have created. Our results show that although the state-of-the-art LLMs can create programs and trace program execution when prompted, they easily succumb to similar errors that have previously been recorded for novice programmers. These results demonstrate the fallibility of these models and perhaps dampen the expectations fueled by the recent LLM hype. At the same time, we also highlight future research possibilities such as using LLMs to mimic students as their behavior can indeed be similar for some specific tasks.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{LehtinenKH24,
title = {Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions},
author = {Teemu Lehtinen and Charles Koutcheme and Arto Hellas},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3639474.3640058}
}