一句话：通过对照实验评估 ChatGPT 辅助学生完成软件工程任务，发现生产力与自我效能无显著提升、挫败感增加，并归纳 5 类故障与 7 类负面后果。
本文关注对话式生成式 AI 在软件工程（SE）教育中的潜力与陷阱：尽管工程师和学术界已在工作中采用该技术，但其支持学生完成 SE 任务的效果尚不明确。作者开展了一项被试间研究（N=22），比较学生使用 ChatGPT 与传统资源完成 SE 任务时的表现。评估指标包括生产力、自我效能与挫败感，并分析人-AI 交互中的故障与后果。结果显示，两组在生产力或自我效能上没有统计显著差异，但使用 ChatGPT 的参与者挫败感显著更高。研究还发现 5 类因违反人-AI 交互准则而产生的故障，并导致 7 种负面后果，提示生成式 AI 在 SE 学习中既有帮助也有局限。
Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.
@inproceedings{paperbot11398,
title = {How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering},
author = {Rudrajit Choudhuri and Dylan Liu and Igor Steinmacher and Marco Gerosa and Anita Sarma},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639201}
}