一句话：通过 LLM-as-a-Judge 对五类评审评论分类，发现可读性、bug 与可维护性评论的解决率高于设计类评论。
LLM 驱动的代码评审自动化工具已能生成评审评论，但并非所有评论都会促成代码修改，因此识别哪些类型的生成评论是“可操作”的至关重要。本文研究两个问题：（1）人类与 LLM 撰写的评审评论各属哪些类型；（2）哪些类型的生成评论最常被开发者解决。作者构建了一个 LLM-as-a-Judge，基于自建的五类分类法自动对评审评论进行分类，并开展实证研究。结果显示：LLM 评审者与人类评审者因项目背景不同而各有强弱；可读性、bug 和可维护性相关评论的解决率高于聚焦代码设计的评论。这表明相当比例的 LLM 生成评论是可操作的、能被开发者解决，也凸显了 LLM 与人类评审者之间的互补性。
Large language model (LLM)-powered code review automation tools have been introduced to generate code review comments. However, not all generated comments will drive code changes. Understanding what types of generated review comments are likely to trigger code changes is crucial for identifying those that are actionable. In this paper, we set out to investigate (1) the types of review comments written by humans and LLMs, and (2) the types of generated comments that are most frequently resolved by developers. To do so, we developed an LLM-as-a-Judge to automatically classify review comments based on our own taxonomy of five categories. Our empirical study confirms that (1) the LLM reviewer and human reviewers exhibit distinct strengths and weaknesses depending on the project context, and (2) readability, bugs, and maintainability-related comments had higher resolution rates than those focused on code design. These results suggest that a substantial proportion of LLM-generated comments are actionable and can be resolved by developers. Our work highlights the complementarity between LLM and human reviewers and offers suggestions to improve the practical effectiveness of LLM-powered code review tools.
DOI 原文 ·
@inproceedings{paperbot9441,
title = {What Types of Code Review Comments Do Developers Most Frequently Resolve?},
author = {Saul Goldman and Hong Yi Lin and Jirat Pasuksmit and Patanamon Thongtanunam and Kla Tantithamthavorn and Zhe Wang and Ray Zhang and Ali Behnaz and Fan Jiang and Michael Siers and Ryan Jiang and Mike Buller and Minwoo Jeong and Ming Wu},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00312}
}