一句话：本文指出现有软件分析分类器排序存在系统性偏差，并证明用 SMOTUNED 改善训练数据比仅优化分类器更能提升缺陷预测。
本文关注软件分析中分类器排序研究的一个系统性错误：既有工作往往只在单一指标上评估分类器，且未考察数据变化对结果的影响。为此，作者在 3681 个 Java 类（超过一百万行开源代码）上采用 5×5 交叉验证，并结合多性能指标进行评估；同时使用 SMOTUNED（SMOTE 的自动调参版本）修复训练数据中较弱的区域，以改善类别不平衡。实验表明，SMOTUNED 使 AUC 和召回率分别提升约 60% 与 20%，且该提升不依赖于所用分类器。将 SMOTE 与 SMOTUNED 同最新类不平衡技术比较时也观察到类似改进。论文因此主张：在缺陷预测中，改善数据可能比仅追求更好的数据挖掘器更有效。
We report and fix an important systematic error in prior studies that ranked classifiers for software analytics. Those studies did not (a) assess classifiers on multiple criteria and they did not (b) study how variations in the data affect the results. Hence, this paper applies (a) multi-performance criteria while (b) fixing the weaker regions of the training data (using SMOTUNED, which is an auto-tuning version of SMOTE). This approach leads to dramatically large increases in software defect predictions when applied in a 5*5 cross-validation study for 3,681 JAVA classes (containing over a million lines of code) from open source systems, SMOTUNED increased AUC and recall by 60% and 20% respectively. These improvements are independent of the classifier used to predict for defects. Same kind of pattern (improvement) was observed when a comparative analysis of SMOTE and SMOTUNED was done against the most recent class imbalance technique.
DOI 原文 ·
@inproceedings{paperbot4430,
title = {Is "better data" better than "better data miners"?},
author = {Amritanshu Agrawal and Tim Menzies},
booktitle = {ICSE},
year = {2018},
doi = {10.1145/3180155.3180197}
}