一句话：提出基于特征保留与消除扰动的统一评估框架，揭示深度学习漏洞检测器对伪相关特征的依赖。
现有基于深度学习的软件漏洞检测结果常不可复现，跨数据集或未见样本表现不稳定。本文分析既有工作与漏洞数据集，区分代码中真正贡献漏洞的句法/语义特征与伪相关特征，并提出统一表示来刻画这两类特征。基于此设计特征保留扰动（FPP）与特征消除扰动（FEP），以度量特征对检测器预测的影响。作者在五种先进DL漏洞检测器上开展大规模实证：对漏洞特征，平均仅约2%的FPP导致预测发生不良变化；但平均约84%的FEP仍保留漏洞预测。对伪特征，FPP可使基于图的检测器召回率最多下降29%。论文解释原因、提出改进策略，并公开评估框架。
Recent research has revealed that the reported results of an emerging body of deep learning-based techniques for detecting software vulnerabilities are not reproducible, either across different datasets or on unseen samples. This paper aims to provide the foundation for properly evaluating the research in this domain. We do so by analyzing prior work and existing vulnerability datasets for the syntactic and semantic features of code that contribute to vulnerability, as well as features that falsely correlate with vulnerability. We provide a novel, uniform representation to capture both sets of features, and use this representation to detect the presence of both vulnerability and spurious features in code. To this end, we design two types of code perturbations: feature preserving perturbations (FPP) ensure that the vulnerability feature remains in a given code sample, while feature eliminating perturbations (FEP) eliminate the feature from the code sample. These perturbations aim to measure the influence of spurious and vulnerability features on the predictions of a given vulnerability detection solution. To evaluate how the two classes of perturbations influence predictions, we conducted a large-scale empirical study on five state-of-the-art DL-based vulnerability detectors. Our study shows that, for vulnerability features, only$\sim 2 \%$of FPPs yield the undesirable effect of a prediction changing among the five detectors on average. However, on average,$\sim 84 \%$of FEPs yield the undesirable effect of retaining the vulnerability predictions. For spurious features, we observed that FPPs yielded a drop in recall up to 29 % for graph-based detectors. We present the reasons underlying these results and suggest strategies for improving DNN-based vulnerability detectors. We provide our perturbation-based evaluation framework as a public resource to enable independent future evaluation of vulnerability detectors.
DOI 原文 ·
@inproceedings{paperbot8887,
title = {Are We Learning the Right Features? A Framework for Evaluating DL-Based Software Vulnerability Detection Solutions},
author = {Satyaki Das and Syeda Tasnim Fabiha and Saad Shafiq and Nenad Medvidović},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00194}
}