一句话：复现9个SOTA深度学习漏洞检测模型，揭示运行间波动、模型间低一致性及训练数据对性能的影响。
深度学习代码模型在漏洞检测任务上进展显著，部分模型甚至超过静态分析工具，但学界对这些模型的行为缺乏系统理解，制约了其鲁棒性、调试与部署。本文调研并复现了9个最先进的深度学习漏洞检测模型，在Devign和MSR两个广泛使用的数据集上，围绕模型能力、训练数据与模型解释三个方面提出并回答了6个研究问题。实验表明：同一模型不同训练运行之间结果存在波动，不同模型的输出一致性很低；针对特定漏洞类型训练的模型与一次性在所有漏洞上训练的模型表现不同；论文还探究了哪些程序被深度学习模型视为“难”样本，分析了训练数据规模与组成同模型性能的关系，并通过模型解释方法分析了预测所依赖的重要特征。作者认为这些发现有助于理解模型结果、指导训练数据准备并提升模型鲁棒性，全部数据集、代码与结果已公开。
Deep learning (DL) models of code have recently reported great progress for vulnerability detection. In some cases, DL-based models have outperformed static analysis tools. Although many great models have been proposed, we do not yet have a good understanding of these models. This limits the further advancement of model robustness, debugging, and deployment for the vulnerability detection. In this paper, we surveyed and reproduced 9 state-of-the-art (SOTA) deep learning models on 2 widely used vulnerability detection datasets: Devign and MSR. We investigated 6 research questions in three areas, namely model capabilities, training data, and model interpretation. We experimentally demonstrated the variability between different runs of a model and the low agreement among different models' outputs. We investigated models trained for specific types of vulnerabilities compared to a model that is trained on all the vulnerabilities at once. We explored the types of programs DL may consider “hard” to handle. We investigated the relations of training data sizes and training data composition with model performance. Finally, we studied model interpretations and analyzed important features that the models used to make predictions. We believe that our findings can help better understand model results, provide guidance on preparing training data, and improve the robustness of the models. All of our datasets, code, and results are available at https://doi.org/10.6084/m9.figshare.20791240.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2212-08109,
title = {An Empirical Study of Deep Learning Models for Vulnerability Detection},
author = {Benjamin Steenhoek and Md Mahbubur Rahman and Richard Jiles and Wei Le},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00188}
}