一句话：通过观察与对照实验证明代码审查中文件展示位置会系统性影响评论分布与缺陷发现率。
论文研究代码审查工具（如 Gerrit、GitHub）按字母序展示文件是否会因相对位置偏见审查结果。作者采用两步三角验证：先分析 GitHub 上 138 个 Java 项目、219,476 个 PR 的审查评论，在控制讨论串和新增行数等混杂因素后发现，较早展示的文件获得更多评论；再招募 106 名参与者进行在线对照实验，在两个文件中植入两个不相关缺陷并交换其位置。结果显示，一类缺陷的发现不受文件位置影响，另一类缺陷在文件位于最后时被识别几率比位于第一时低 64%。研究表明文件位置会影响代码审查结果，并对工具设计与实践有启示。
The most popular code review tools (e.g., Gerrit and GitHub) present the files to review sorted in alphabetical order. Could this choice or, more generally, the relative position in which a file is presented bias the outcome of code reviews? We investigate this hypothesis by triangulating complementary evidence in a two-step study. First, we observe developers’ code review activity. We analyze the review comments pertaining to 219,476 Pull Requests (PRs) from 138 popular Java projects on GitHub. We found files shown earlier in a PR to receive more comments than files shown later, also when controlling for possible confounding factors: e.g., the presence of discussion threads or the lines added in a file. Second, we measure the impact of file position on defect finding in code review. Recruit- ing 106 participants, we conduct an online controlled experiment in which we measure participants’ performance in detecting two unrelated defects seeded into two different files. Participants are assigned to one of two treatments in which the position of the defective files is switched. For one type of defect, participants are not affected by its file’s position; for the other, they have 64% lower odds to identify it when its file is last as opposed to first. Overall, our findings provide evidence that the relative position in which files are presented has an impact on code reviews’ outcome; we discuss these results and implications for tool design and code review.
@inproceedings{paperbot7030,
title = {First come first served: the impact of file position on code review},
author = {Enrico Fregnan and Larissa Braz and Marco D'Ambros and Gül Çalıklı and Alberto Bacchelli},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3549177}
}