一句话：实证表明，漏洞预测若假设完美标注会严重高估效果，必须按真实可用标注评估。
本文研究软件漏洞预测中普遍存在的完美标注假设。作者对 Linux Kernel、OpenSSL 和 Wireshark 三个安全关键开源系统的 74 个版本中 1,898 个真实漏洞进行实证研究，比较三种已有漏洞预测方法在两种设置下的效果：采用不现实的完美标注，以及仅使用现实中可获得的标注。结果显示，完美标注假设会严重误导科学结论；一旦按真实标注评估，预测效果大幅下降，例如三个系统的 MCC 均值分别从 0.77、0.65、0.43 降至 0.08、0.22、0.10，precision、recall 等指标也类似。作者因此呼吁社区升级漏洞预测评估与开发的实验方法，以保证结果稳健且可操作。
Previous work on vulnerability prediction assume that predictive models are trained with respect to perfect labelling information (includes labels from future, as yet undiscovered vulnerabilities). In this paper we present results from a comprehensive empirical study of 1,898 real-world vulnerabilities reported in 74 releases of three security-critical open source systems (Linux Kernel, OpenSSL and Wiresark). Our study investigates the effectiveness of three previously proposed vulnerability prediction approaches, in two settings: with and without the unrealistic labelling assumption. The results reveal that the unrealistic labelling assumption can profoundly mis- lead the scientific conclusions drawn; suggesting highly effective and deployable prediction results vanish when we fully account for realistically available labelling in the experimental methodology. More precisely, MCC mean values of predictive effectiveness drop from 0.77, 0.65 and 0.43 to 0.08, 0.22, 0.10 for Linux Kernel, OpenSSL and Wiresark, respectively. Similar results are also obtained for precision, recall and other assessments of predictive efficacy. The community therefore needs to upgrade experimental and empirical methodology for vulnerability prediction evaluation and development to ensure robust and actionable scientific findings.
@inproceedings{paperbot5164,
title = {The importance of accounting for real-world labelling when predicting software vulnerabilities},
author = {Matthieu Jimenez and Renaud Rwemalika and Mike Papadakis and Federica Sarro and Yves Le Traon and Mark Harman},
booktitle = {FSE},
year = {2019},
doi = {10.1145/3338906.3338941}
}