一句话：大规模实证发现，简单逻辑回归加新增行数特征可超越 CC2Vec 与 DeepJIT，且快数万倍。
即时（JIT）缺陷预测用于在持续测试中判断提交是否有缺陷。现有最优 CC2Vec 以层次注意力网络学习代码增删表示，但评估数据有限。本文在超过 310,370 个变更的大规模数据集上系统比较 CC2Vec、DeepJIT 与传统 JIT 方法，并分析传统特征影响。实验发现，CC2Vec 不能稳定优于 DeepJIT，二者也不能稳定优于传统方法；新增行数特征最佳。受此启发，作者仅用新增行数特征加逻辑回归，效果即可超过 CC2Vec 与 DeepJIT，训练/测试速度分别快约 8.1 万/12 万倍，并给出实践指南。
Defect prediction aims to automatically identify potential defective code with minimal human intervention and has been widely studied in the literature. Just-in-Time (JIT) defect prediction focuses on program changes rather than whole programs, and has been widely adopted in continuous testing. CC2Vec, state-of-the-art JIT defect prediction tool, first constructs a hierarchical attention network (HAN) to learn distributed vector representations of both code additions and deletions, and then concatenates them with two other embedding vectors representing commit messages and overall code changes extracted by the existing DeepJIT approach to train a model for predicting whether a given commit is defective. Although CC2Vec has been shown to be the state of the art for JIT defect prediction, it was only evaluated on a limited dataset and not compared with all representative baselines. Therefore, to further investigate the efficacy and limitations of CC2Vec, this paper performs an extensive study of CC2Vec on a large-scale dataset with over 310,370 changes (8.3 X larger than the original CC2Vec dataset). More specifically, we also empirically compare CC2Vec against DeepJIT and representative traditional JIT defect prediction techniques. The experimental results show that CC2Vec cannot consistently outperform DeepJIT, and neither of them can consistently outperform traditional JIT defect prediction. We also investigate the impact of individual traditional defect prediction features and find that the added-line-number feature outperforms other traditional features. Inspired by this finding, we construct a simplistic JIT defect prediction approach which simply adopts the added-line-number feature with the logistic regression classifier. Surprisingly, such a simplistic approach can outperform CC2Vec and DeepJIT in defect prediction, and can be 81k X/120k X faster in training/testing. Furthermore, the paper also provides various practical guidelines for advancing JIT defect prediction in the near future.
DOI 原文 ·
@inproceedings{paperbot6668,
title = {Deep just-in-time defect prediction: how far are we?},
author = {Zhengran Zeng and Yuqun Zhang and Haotian Zhang and Lingming Zhang},
booktitle = {ISSTA},
year = {2021},
doi = {10.1145/3460319.3464819}
}