一句话：通过自动评估工具测试 GPT-3.5、GPT-4 和 Bard 解决 OOP 作业的能力，发现可生成基本可用代码但常违背 OOP 最佳实践。
大语言模型虽被视为编程学习助手，但其在具有实体、关系与职责识别复杂性的面向对象编程（OOP）任务中的表现尚缺研究。本文选取教育场景中的真实 OOP 练习，让 GPT-3.5、GPT-4 和 Bard 求解，并用自动评估工具（AAT）验证解答。结果显示，模型常能给出基本可运行的方案，却频繁忽视 OOP 最佳实践；GPT-4 表现最佳，GPT-3.5 次之，Bard 最差。作者主张在使用这些模型时重新强调代码质量，并探索将 LLM 与 AAT 结合用于教学；结论认为，即便 GPT-4 展现潜力，部署到 OOP 教育中仍需人工监督。
Large Language Models (LLMs) have emerged as promising tools to assist students while solving programming assignments. However, object-oriented programming (OOP), with its inherent complexity involving the identification of entities, relationships, and responsibilities, is not yet mastered by these tools. Contrary to introductory programming exercises, there exists a research gap with regard to the behavior of LLMs in OOP contexts. In this study, we experimented with three prominent LLMs - GPT-3.5, GPT-4, and Bard - to solve real-world OOP exercises used in educational settings, subsequently validating their solutions using an Automatic Assessment Tool (AAT). The findings revealed that while the models frequently achieved mostly working solutions to the exercises, they often overlooked the best practices of OOP. GPT-4 stood out as the most proficient, followed by GPT-3.5, with Bard trailing last. We advocate for a renewed emphasis on code quality when employing these models and explore the potential of pairing LLMs with AATs in pedagogical settings. In conclusion, while GPT-4 show-cases promise, the deployment of these models in OOP education still mandates supervision.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{CiprianoA24,
title = {LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignments},
author = {Bruno Pereira Cipriano and Pedro Alves},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3639474.3640052}
}