一句话:本文研究大规模代码评审自动化的愿景与挑战,通过收集数十万个changeset并预测评论位置,揭示该任务的困难与随机性。
神经方法日益用于支持与自动化软件开发任务,代码评审是自然的下一目标。但基于历史评审训练模型模仿开发者并不简单:开源评审在质量、措辞与深度上因评审者而异,且 changeset 常达数千 token,超出当前神经模型容量;已有工作通过聚焦小输入且已知评论位置来回避问题。本文考察真实规模下自动化代码评审的愿景与挑战:从数百个常规进行评审的项目收集数十万个 changeset,聚焦于预测评论位置这一稀疏任务。通过分析模型性能与数据集统计,作者表明即使该任务也已颇具挑战,很大程度上源于评审中巨大的变化与(表面上的)随机性,并由此引出研究议程。
As neural methods are increasingly used to support and automate software development tasks, code review is a natural next target. Yet, training models to imitate developers based on past code reviews is far from straightforward: reviews found in open-source projects vary greatly in quality, phrasing, and depth depending on the reviewer. In addition, changesets are often large, stretching the capacity of current neural models. Recent work reported modest success at predicting review resolutions, but largely side-stepped the above issues by focusing on small inputs where comments were already known to occur. This work examines the vision and challenges of automating code review at realistic scale. We collect hundreds of thousands of changesets across hundreds of projects that routinely conduct code review, many of which change thousands of tokens. We focus on predicting just the locations of comments, which are quite rare. By analyzing model performance and dataset statistics, we show that even this task is already challenging, in no small part because of tremendous variation and (apparent) randomness in code reviews. Our findings give rise to a research agenda for realistically and impactfully automating code review.