一句话：评估六种 GNN 解释器在四个漏洞检测器上的稳定性、鲁棒性与有效性，并提出评估准则。
传统漏洞检测依赖大量人工；基于图神经网络（GNN）的漏洞检测因能利用代码结构而流行，但模型近似黑盒，解释对安全分析师缺乏清晰证据，且现有解释结果高度不一致、难以信服。为此，论文从漏洞检测关切出发，提出以稳定性、鲁棒性和有效性为核心的评估准则，并在四个漏洞检测器（DeepWukong、Devign、IVDetect、Reveal）上系统评测六种著名解释器（GNN-LRP、DeepLIFT、GradCAM、GNNExplainer、PGExplainer、SubGraphX）。实验表明这些解释器在有效性、稳定性和鲁棒性上表现较差；实例无关方法有效性更好，基于扰动的方法对模型参数微调更稳定，实例无关方法对相似漏洞给出一致解释。
Traditional vulnerability detection methods have limitations due to their need for extensive manual labor. Using automated means for vulnerability detection has attracted research interest, especially deep learning, which has achieved remarkable results. Since graphs can better convey the structural feature of code than text, graph neural network (GNN) based vulnerability detection is significantly better than text-based approaches. Therefore, GNN-based vulnerability detection approaches are becoming popular. However, GNN models are close to black boxes for security analysts, so the models cannot provide clear evidence to explain why a code sample is detected as vulnerable or secure. At this stage, many GNN interpreters have been proposed. However, the explanations provided by these interpretations for vulnerability detection models are highly inconsistent and unconvincing to security experts. To address the above issues, we propose principled guidelines to assess the quality of the interpretation approaches for GNN-based vulnerability detectors based on concerns in vulnerability detection, namely, stability, robustness, and effectiveness. We conduct extensive experiments to evaluate the interpretation performance of six famous interpreters (GNN-LRP, DeepLIFT, GradCAM, GNNExplainer, PGExplainer, and SubGraphX) on four vulnerability detectors (DeepWukong, Devign, IVDetect, and Reveal). The experimental results show that the target interpreters achieve poor performance in terms of effectiveness, stability, and robustness. For effectiveness, we find that the instance-independent methods outperform others due to their deep insight into the detection model. In terms of stability, the perturbation-based interpretation methods are more resilient to slight changes in model parameters as they are model-agnostic. For robustness, the instance-independent approaches provide more consistent interpretation results for similar vulnerabilities.
@inproceedings{paperbot8065,
title = {Interpreters for GNN-Based Vulnerability Detection: Are We There Yet?},
author = {Yutao Hu and Suyuan Wang and Wenke Li and Junru Peng and Yueming Wu and Deqing Zou and Hai Jin},
booktitle = {ISSTA},
year = {2023},
doi = {10.1145/3597926.3598145}
}