一句话：通过复现 CHREV 与 WLRREC 并结合问卷与访谈，发现基于历史的评审者推荐评估在 Gerrit 中系统性偏悲观。
评审者推荐系统通常用留出的历史数据评估推荐质量，但历史记录可能过于乐观（过去分配者未必最优）或过于悲观（未匹配历史分配者的推荐也可能合适）。本文复现 CHREV 与 WLRREC，在 Gerrit 开源社区的 9,679 条评审上生成推荐，并与 Gerrit 评审社区成员开展定量问卷（个性化任务舒适度）和定性半结构化访谈。结果表明，历史评估在 Gerrit 情境下远偏悲观：86% 曾担任评审者的人认为处理该评审舒适，而 74% 被标为“错误推荐”的人也表示会舒适地评审。这说明匹配历史分配者的推荐应视为正确，但未匹配者未必错误；未来评估需超越仓库数据以理解推荐的实际价值。
Reviewer recommendation systems are used to suggest community members to review change requests. Like several other recommendation systems, it is customary to evaluate recommendations using held out historical data. While history-based evaluation makes pragmatic use of available data, historical records may be: (1) overly optimistic, since past assignees may have been suboptimal choices for the task at hand; or (2) overly pessimistic, since "incorrect" recommendations may have been equal (or even better) choices.In this paper, we empirically evaluate the extent to which historical data is an appropriate benchmark for reviewer recommendation systems. We replicate the CHREV and WLRREC approaches and apply them to 9,679 reviews from the GERRIT open source community. We then assess the recommendations with members of the GERRIT reviewing community using quantitative methods (personalized questionnaires about their comfort level with tasks) and qualitative methods (semi-structured interviews).We find that history-based evaluation is far more pessimistic than optimistic in the context of GERRIT review recommendations. Indeed, while 86% of those who had been assigned to a review in the past felt comfortable handling the review, 74% of those labelled as incorrect recommendations also felt that they would have been comfortable reviewing the changes. This indicates that, on the one hand, when reviewer recommendation systems recommend the past assignee, they should indeed be considered correct. Yet, on the other hand, recommendations labelled as incorrect because they do not match the past assignee may have been correct as well.Our results suggest that current reviewer recommendation evaluations do not always model the reality of software development. Future studies may benefit from looking beyond repository data to gain a clearer understanding of the practical value of proposed recommendations.
DOI 原文 ·
@inproceedings{paperbot6515,
title = {Is Historical Data an Appropriate Benchmark for Reviewer Recommendation Systems? : A Case Study of the Gerrit Community},
author = {Ian X. Gauthier and Maxime Lamothe and Gunter Mussbacher and Shane McIntosh},
booktitle = {ASE},
year = {2021},
doi = {10.1109/ase51524.2021.9678640}
}