一句话：通过开发者意见与 transformer 模型度量 Meta 测试计划质量，并关联评审参与度和回退，以改进代码审查工具。
现代代码审查中，测试计划等工件以自由文本形式承担知识共享与沟通作用，但其质量难以度量。本文以 Meta 的测试计划为对象，收集 500 多名开发者对 650 多份测试计划的意见，构建基于 transformer 的模型来自动评估测试计划质量。结果显示，该模型能捕捉开发者情绪，并反映测试计划质量与评审参与度、代码回退之间的相关性；相比决策树模型，F1 分数提高 7%。最后，作者通过案例研究说明该度量如何用于实验，以指导开发者工具与体验改进。
In modern code reviews, many artifacts play roles in knowledge- sharing and documentation: summaries, test plans, and comments, etc. Improving developer tools and facilitating better code reviews require an understanding of the quality of pull requests and their artifacts. This is difficult to measure, however, because they are often free-form natural language and unstructured text data. In this paper, we focus on measuring the quality of test plans at Meta. Test plans are used as a communication mechanism between the author of a pull request and its reviewers, serving as walkthroughs to help confirm that the changed code is behaving as expected. We collected developer opinions on over 650 test plans from more than 500 Meta developers, then introduced a transformer-based model to leverage the success of natural language processing (NLP) tech- niques in the code review domain. In our study, we show that the learned model is able to capture the sentiment of developers and reflect a correlation of test plan quality with review engagement and reversions: compared to a decision tree model, our proposed transformer-based model achieves a 7% higher F1-score. Finally, we present a case study of how such a metric may be useful in experiments to inform improvements in developer tools and experiences.
DOI 原文 ·
@inproceedings{paperbot7045,
title = {Leveraging test plan quality to improve code review efficacy},
author = {Lawrence Chen and Rui Abreu and Tobi Akomolede and Peter C. Rigby and Satish Chandra and Nachiappan Nagappan},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3558952}
}