一句话：对照研究发现，研究生用 LLM 解决库升级任务时往往未迁移规范的软件工程实践，而是直接把构建错误喂给模型。
生成式 AI 与大语言模型（LLM）正迅速进入软件工程师的工具箱，但早期职业开发者如何在实际工作中使用这些工具、能否把课堂训练迁移到 LLM 辅助的实践，仍缺乏实证了解。本文在卡内基梅隆大学电气与计算机工程系软件工程硕士的软件重构课程中开展了一项对照研究，让研究生分别以非 LLM 的临时（ad hoc）方法与 LLM 支持的自动化方法解决同一个库升级问题。结果显示，学生在依赖 LLM 时常常没有应用规范的软件工程实践：他们只是把构建产生的错误文本直接喂给 LLM，而不是借助 LLM 理解问题、定位根因并探索解决方案。作者据此指出教育者面临的双重挑战：一方面要强化基础软件工程技能及其向 GenAI 使用的迁移，另一方面要引入有效提示与驾驭 LLM 驱动工作流等新能力。
Generative AI (GenAI), in particular large language models (LLMs), have rapidly become part of software engineers’ toolboxes. While the software engineering community has been actively debating efficacy and extent of use of GenAI tools, little is known about how early-career developers apply these tools and transfer their classroom training into practice. To address this gap, we conducted a controlled study with graduate students enrolled in a software refactoring course in the Master of Software Engineering program at Carnegie Mellon University’s Department of Electrical and Computer Engineering. The study compared how students solved a library upgrade problem using an ad hoc non-LLM approach versus an automated approach supported by LLMs. Our findings show that students often failed to apply disciplined software engineering practices when relying on LLMs, simply feeding the LLMs error text from builds rather than using them to understand issues and explore solutions. Our results highlight a dual challenge for educators: reinforcing foundational software engineering skills and their transfer to GenAI use, while also introducing new competencies such as effective prompting and navigating LLM-driven workflows.
@inproceedings{paperbot11969,
title = {Prompting Without Principles: Are Students Transferring Software Engineering Knowledge to LLM Use?},
author = {Leonardo Sousa and Ipek Ozkaya and James Ivers and Celina Cywinska and Bingyu Xie and Mena Kostial and Tapajit Dey and Robert Edman},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3786580.3786990}
}