一句话：首次实证评估ChatGPT在基于代码评审的自动代码精化中的表现，并分析失败原因与改进策略。
代码评审是保障软件质量和可维护性的关键活动，但耗时且易错。本文首次开展实证研究，考察ChatGPT在基于给定代码评审意见的自动代码精化任务中的能力。作者选取现有CodeReview基准并构建高质量代码评审数据集，以当前最优工具CodeReviewer作为基线进行比较。结果表明，ChatGPT在代码精化任务上优于CodeReviewer，EM和BLEU分别达到22.78和76.44，而基线仅为15.50和62.88。作者进一步分析ChatGPT表现不佳的根本原因，并提出若干缓解策略，为利用大语言模型自动化代码评审流程提供了实证洞见和潜在研究方向。
Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks, suggesting its potential to automate code review processes. However, it is still unclear how well ChatGPT performs in code review tasks. To fill this gap, in this paper, we conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks, specifically focusing on automated code refinement based on given code reviews. To conduct the study, we select the existing benchmark CodeReview and construct a new code review dataset with high quality. We use CodeReviewer, a state-of-the-art code review tool, as a baseline for comparison with ChatGPT. Our results show that ChatGPT outperforms CodeReviewer in code refinement tasks. Specifically, our results show that ChatGPT achieves higher EM and BLEU scores of 22.78 and 76.44 respectively, while the state-of-the-art method achieves only 15.50 and 62.88 on a high-quality code review dataset. We further identify the root causes for ChatGPT's underperformance and propose several strategies to mitigate these challenges. Our study provides insights into the potential of ChatGPT in automating the code review process, and highlights the potential research directions.
@inproceedings{paperbot11308,
title = {Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study},
author = {Qi Guo and Junming Cao and Xiaofei Xie and Shangqing Liu and Xiaohong Li and Bihuan Chen and Xin Peng},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3623306}
}