一句话：系统剖析深度学习漏洞检测数据集的标签错误成因，量化其影响并提出两种去噪方法。
软件系统复杂性与安全漏洞的多样性使漏洞研究长期面临挑战，深度学习为自动漏洞检测提供了有效补充，但缺乏高质量基准数据集严重制约了检测效果——现有漏洞数据集中长期存在的错误标签会导致结果不准确、有偏甚至错误。本文系统分析了当前最先进的基于学习的漏洞检测方法所使用的多样化数据集，并考察其收集漏洞源码数据集的技术，从而深入理解并解释标签错误的成因。研究发现，标签错误严重影响主流漏洞检测模型，最坏情况下平均 F1 下降 20.7%。作为缓解，作者提出两种数据集去噪方法，平均可将模型性能提升 10.4%，并据此给出了获取高质量标注数据集的可行方案。
Software system complexity and security vulnerability diversity are plausible sources of the persistent challenges in software vulnerability research. Applying deep learning methods for automatic vulnerability detection has been proven an effective means to complement traditional detection approaches. Unfortunately, lacking well-qualified benchmark datasets could critically restrict the effectiveness of deep learning-based vulnerability detection techniques. Specifically, the long-term existence of erroneous labels in the existing vulnerability datasets may lead to inaccurate, biased, and even flawed results. In this paper, we aim to obtain an in-depth understanding and explanation of the label error causes. To this end, we systematically analyze the diversified datasets used by state-of-the-art learning-based vulnerability detection approaches, and examine their techniques for collecting vulnerable source code datasets. We find that label errors heavily impact the mainstream vulnerability detection models, with a worst-case average F1 drop of 20.7%. As mitigation, we introduce two approaches to dataset denoising, which will enhance the model performance by an average of 10.4%. Leveraging dataset denoising methods, we provide a feasible solution to obtain high-quality labeled datasets.
@inproceedings{NieLWWLW23,
title = {Understanding and Tackling Label Errors in Deep Learning-Based Vulnerability Detection (Experience Paper)},
author = {Xu Nie and Ningke Li and Kailong Wang and Shangguang Wang and Xiapu Luo and Haoyu Wang},
booktitle = {ISSTA},
year = {2023},
doi = {10.1145/3597926.3598037}
}