一句话：提出基于开发者显式引用与人工过滤构建 oracle，并评估多种 SZZ 变体的准确率。
SZZ 算法被广泛用于识别缺陷引入变更，并支撑缺陷预测与缺陷引入的实证研究。然而，现有 SZZ 变体缺乏可靠评测：常用做法是研究者手工分类输出或使用研究者构建的黄金集，但研究者对目标系统了解有限。本文提出构建“开发者知情”oracle 的方法：先用自然语言处理识别那些显式引用引入缺陷提交的 bug 修复提交，再通过人工过滤保证质量与准确性。基于该 oracle，作者评估了多种 SZZ 变体的准确率，并从中提炼出改进 SZZ 的若干经验教训。
The SZZ algorithm for identifying bug-inducing changes has been widely used to evaluate defect prediction techniques and to empirically investigate when, how, and by whom bugs are introduced. Over the years, researchers have proposed several heuristics to improve the SZZ accuracy, providing various implementations of SZZ. However, fairly evaluating those implementations on a reliable oracle is an open problem: SZZ evaluations usually rely on (i) the manual analysis of the SZZ output to classify the identified bug-inducing commits as true or false positives; or (ii) a golden set linking bug-fixing and bug-inducing commits. In both cases, these manual evaluations are performed by researchers with limited knowledge of the studied subject systems. Ideally, there should be a golden set created by the original developers of the studied systems. We propose a methodology to build a "developer-informed" oracle for the evaluation of SZZ variants. We use Natural Language Processing (NLP) to identify bug-fixing commits in which developers explicitly reference the commit(s) that introduced a fixed bug. This was followed by a manual filtering step aimed at ensuring the quality and accuracy of the oracle. Once built, we used the oracle to evaluate several variants of the SZZ algorithm in terms of their accuracy. Our evaluation helped us to distill a set of lessons learned to further improve the SZZ algorithm.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2102-03300,
title = {Evaluating SZZ Implementations Through a Developer-Informed Oracle},
author = {Giovanni Rosa and Luca Pascarella and Simone Scalabrino and Rosalia Tufano and Gabriele Bavota and Michele Lanza and Rocco Oliveto},
booktitle = {ICSE},
year = {2021},
doi = {10.1109/icse43902.2021.00049}
}