一句话：提出 Primevul 数据集与真实评测，揭示现有基准严重高估代码 LMs 的漏洞检测能力。
本文研究代码语言模型（code LMs）在漏洞检测中的有效性。分析发现，现有漏洞数据集存在数据质量差、标签准确率低和重复率高的问题，且常用评测方法不能代表真实漏洞检测场景。为此，作者提出 Primevul，一个用于训练和评测代码 LMs 漏洞检测的新数据集。它采用新的数据标注技术，在显著扩大数据规模的同时达到接近人工验证基准的标签准确率；还实施严格的去重和按时间划分策略以缓解数据泄漏，并引入更真实的评测指标与设置。在 Primevul 上评测表明，现有基准显著高估模型性能：某先进 7B 模型在 BigVul 上 F1 为 68.26%，在 Primevul 上仅 3.09%；GPT-3.5 和 GPT-4 在最严格设置下也接近随机猜测。
In the context of the rising interest in code language models (code LMs) and vulnerability detection, we study the effectiveness of code LMs for detecting vulnerabilities. Our analysis reveals significant shortcomings in existing vulnerability datasets, including poor data quality, low label accuracy, and high duplication rates, leading to unreliable model performance in realistic vulnerability detection scenarios. Additionally, the evaluation methods used with these datasets are not representative of real-world vulnerability detection. To address these challenges, we introduce Primevul, a new dataset for training and evaluating code LMs for vulnerability detection. Primevul incorporates a novel set of data labeling techniques that achieve comparable label accuracy to human-verified benchmarks while significantly expanding the dataset. It also implements a rigorous data de-duplication and chronological data splitting strategy to mitigate data leakage issues, alongside introducing more realistic evaluation metrics and settings. This comprehensive approach aims to provide a more accurate assessment of code LMs' performance in real-world conditions. Evaluating code LMs on Primevul reveals that existing benchmarks significantly overestimate the performance of these models. For instance, a state-of-the-art 7B model scored 68.26% Fl on BigVul but only 3.09% Fl on Primevul. Attempts to improve performance through advanced training techniques and larger models like GPT-3.5 and GPT-4 were unsuccessful, with results akin to random guessing in the most stringent settings. These findings underscore the considerable gap between current capabilities and the practical requirements for deploying code LMs in security roles, highlighting the need for more innovative research in this domain.
DOI 原文 ·
@inproceedings{paperbot8850,
title = {Vulnerability Detection with Code Language Models: How Far are We?},
author = {Yangruibo Ding and Yanjun Fu and Omniyyah Ibrahim and Chawin Sitawarin and Xinyun Chen and Basel Alomair and David Wagner and Baishakhi Ray and Yizheng Chen},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00038}
}