一句话：提出非爬取基准TutorCode与利用导师指导等增强信息的会话式修复框架CREF，显著提升LLM修复效果。
本文关注LLM在程序修复中的真实能力评估与辅助编程导师的应用。针对现有修复基准可能已被LLM训练数据污染的问题，作者构建了包含1,239个C++缺陷代码及相关信息（导师指导、解答描述、失败测试用例、修正代码）的非爬取基准TutorCode，并用修复正确性（TOP-5、AVG-5）和补丁精度（RPSR）评估LLM。随后系统探究何种额外信息能提升修复性能，发现导师指导最有效。基于此，作者提出会话式半自动修复框架CREF，充分利用LLM对话能力与增强信息辅助人类导师。在TutorCode上，CREF相比基线将AVG-5提升17.2%–24.6%，使用GPT-4时AVG-5达76.6%。真实教育场景应用表明其能减少导师工作量并改善学生学习体验，也展现出在代码审查等软件工程任务中的潜力。
With the proven effectiveness of Large Language Models (LLMs) in code-related tasks, researchers have explored their potential for program repair. However, existing repair benchmarks might have influenced LLM training data, potentially causing data leakage. To evaluate LLMs’ realistic repair capabilities, (i) we introduce an extensive, non-crawled benchmark TutorCode, comprising 1,239 C++ defect codes and associated information such as tutor guidance, solution description, failing test cases, and the corrected code. Our work assesses LLM’s repair performance on TutorCode, measuring repair correctness (TOP-5 and AVG-5) and patch precision (RPSR). (ii) We then provide a comprehensive investigation into which types of extra information can help LLMs improve their repair performance. Among these types, tutor guidance was the most effective information. To fully harness LLMs’ conversational capabilities and the benefits of augmented information, (iii) we introduce a novel conversational semi-automatic repair framework CREF assisting human programming tutors. It demonstrates a remarkable AVG-5 improvement of 17.2%-24.6% compared to the baseline, achieving an impressive AVG-5 of 76.6% when utilizing GPT-4. These results highlight the potential for enhancing LLMs’ repair capabilities through tutor interactions and historical conversations. The successful application of CREF in a real-world educational setting demonstrates its effectiveness in reducing tutors’ workload and improving students’ learning experience, showing promise for code review and other software engineering tasks.
@inproceedings{paperbot8544,
title = {CREF: An LLM-Based Conversational Software Repair Framework for Programming Tutors},
author = {Boyang Yang and Haoye Tian and Weiguo Pian and Haoran Yu and Haitao Wang and Jacques Klein and Tegawendé F. Bissyandé and Shunfu Jin},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680328}
}