一句话：通过29名开发者的对照实验，揭示LLM自动代码审查对审查行为、质量、耗时与信心的真实影响。
本文研究将自动生成代码审查（尤其LLM反馈）引入人工代码审查流程的实际影响。尽管LLM共同审查者日益流行，但缺乏对其影响审查质量、成本和信心的实证。作者开展受控实验：29名职业开发者分别在有/无自动代码审查支持下审查程序，并监控超过50小时审查活动。结果显示，审查者认为LLM识别的大多数问题有效；自动审查强烈影响其行为，使其聚焦LLM指出的位置，而非搜索其他问题。自动审查起点帮助发现更多低严重度问题，但未增加高严重度问题；且未节省时间，也未提高信心。
Several techniques have been proposed to (partially) automate code review. Early support consisted in recommending the most suited reviewer for a given change or in prioritizing the review tasks. With the advent of deep learning in software engineering, the level of automation has been pushed to new heights, with approaches able to provide feedback on source code in natural language as a human reviewer would do. Also, recent work documented open source projects adopting Large Language Models (LLMs) as co-reviewers. Although the research in this field is very active, little is known about the actual impact of including automatically generated code reviews in the code review process. While there are many aspects worth investigating (e.g., is knowledge transfer between developers affected?), in this work we focus on three of them: (i) review quality, i.e., the reviewer's ability to identify issues in the code; (ii) review cost, i.e., the time spent reviewing the code; and (iii) reviewer's confidence, i.e., how confident is the reviewer about the provided feedback. We run a controlled experiment with 29 professional developers who reviewed different programs with/without the support of an automatically generated code review. During the experiment we monitored the reviewers' activities, for over 50 hours of recorded code reviews. We show that reviewers consider valid most of the issues automatically identified by the LLM and that the availability of an automated review as a starting point strongly influences their behavior: Reviewers tend to focus on the code locations indicated by the LLM rather than searching for additional issues in other parts of the code. The reviewers who started from an automated review identified a higher number of low-severity issues while, however, not identifying more high-severity issues as compared to a completely manual process. Finally, the automated support did not result in saved time and did not increase the reviewers' confidence.
DOI 原文 ·
@inproceedings{paperbot8852,
title = {Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?},
author = {Rosalia Tufano and Alberto Martin-Lopez and Ahmad Tayeb and Ozren Dabić and Sonia Haiduc and Gabriele Bavota},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00060}
}