一句话：提出 BenchVul、TitanVul 与 RVG，揭示漏洞检测模型的分布外泛化差距并提升 Top 25 CWE 检测。
现有漏洞检测模型依赖含标签错误、重复和关键 CWE 覆盖不足的数据集，导致同分布测试虚高、真实泛化差。作者构建 BenchVul 手工平衡测试集，覆盖 MITRE Top 25 CWE；聚合七个公开来源并经去重与多智能体 LLM 验证，构建含 38,548 个函数的 TitanVul 训练集；提出 RVG 通过模拟开发流程为欠代表但关键的 CWE 合成上下文相关样本。评估表明 ID 性能不能可靠预测 BenchVul 上的 OOD 性能：BigVul 训练模型 ID 准确率 0.703 但 OOD 仅 0.493；TitanVul 训练模型 ID 仅 0.590，却在真实与合成部分分别达 0.881 和 0.785，较次优提升 5.3% 和 11.8%；加入 RVG 后真实数据准确率再升至 0.932（+5.8%）。
Automated vulnerability detection research has made substantial progress, yet its real-world impact remains limited. Prior work found that current vulnerability datasets suffer from issues including label inaccuracy rates of 20%-71%, extensive duplication, and poor coverage of critical Common Weakness Enumeration (CWE). These issues create a significant “generalization gap” where models achieve misleading In-Distribution (ID) accuracies (testing on splits from the same dataset) by exploiting spurious correlations rather than learning true vulnerability patterns. To address these limitations, we present a three-part solution. First, we introduce BenchVul, which is a manually curated and balanced test dataset covering the MITRE Top 25 Most Dangerous CWEs, to enable fair model evaluation. Second, we construct a high-quality training dataset, TitanVul, comprising 38,548 functions by aggregating seven public sources and applying deduplication and validation using a novel multi-agent LLM pipeline. Third, we propose a Realistic Vulnerability Generation (RVG) pipeline, which synthesizes context-aware vulnerability examples for underrepresented but critical CWE types through simulated development workflows. Our evaluation reveals that In-Distribution (ID) performance does not reliably predict Out-of-Distribution (OOD) performance on BenchVul. For example, a model trained on BigVul achieves the highest 0.703 ID accuracy but fails on BenchVul’s real-world samples (0.493 OOD accuracy). Conversely, a model trained on our TitanVul achieves the highest OOD performance on both the real-world (0.881) and synthesized (0.785) portions of BenchVul, improving upon the next-best performing dataset by 5.3% and 11.8% respectively, despite a modest ID score (0.590). Augmenting TitanVul with our RVG further boosts this leading OOD performance, improving accuracy on real-world data by 5.8% (to 0.932). Data is available at: https://github.com/yikun-li/TitanVul-BenchVul.
@inproceedings{abs-2507-21817,
title = {Out of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses?},
author = {Yikun Li and Ngoc Tan Bui and Ting Zhang and Chengran Yang and Xin Zhou and Martin Weyssow and Jinfeng Jiang and Junkai Chen and Huihui Huang and Huu Hung Nguyen and Chiok Yew Ho and Jie Tan and Ruiyin Li and Yide Yin and Han Wei Ang and Frank Liauw and Eng Lieh Ouh and Lwin Khin Shar and David Lo},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3787830}
}