一句话：发布 JavaVulBench：面向 Java 漏洞检测的真实划分、多后端统一评测与预训练泄漏感知基准。
针对 Java 漏洞检测评测中划分不真实、后端不统一和预训练污染影响可比性的问题，本文发布 JavaVulBench 数据集与评测框架。数据集包含约 30,600 个 Java 方法、覆盖 1,740 个 CVE 和 700 余个项目，提供方法级与行级标签、每个 CVE 的发布日期，以及随机、项目不相交、时间、去重和未见 CWE 家族五种真实划分。框架以统一的 LlmPrediction schema 支持编码器分类器、Ollama 本地生成模型和 OpenRouter API LLM，可从单条命令在相同条件下评测 12 个参考检测器。作者还提供预训练污染审计，以区分测试 CVE 是否在模型截止日期之后发布，并发布数据集、源码、微调检查点与演示视频。
We release JavaVulBench, a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset contains ∼30,600 Java methods spanning 1,740 CVEs and 700+ projects, labelled at both method and line granularity, with per-CVE publication dates and five realistic split strategies: random, project-disjoint, temporal, deduplicated, and unseen CWE-family. The harness provides a single LlmPrediction schema across three backend families (encoder classifiers, local generative models served by Ollama, and API-served LLMs routed through OpenRouter) so that twelve reference detectors CodeBERT, GraphCodeBERT, UniXcoder, DeepSeek-Coder-1.3B, and eight API/open-weight LLMs (GPT-4o, GPT-4.1-mini, Claude Sonnet 4, DeepSeek-v3, DeepSeek-Coder-v2, Qwen-2.5-Coder-14B/7B, CodeLlama-13B) are evaluated under identical conditions from a single command. A pre-training contamination audit is shipped alongside every model, enabling users to distinguish test CVEs published after the documented model cutoff from those that may have been seen during pre-training. To support reproducibility and reuse, the benchmark release includes the dataset, source code, fine-tuned model checkpoints, and a short demonstration video(https://www.youtube.com/watch?v=nMTX_hqkuoM).
@inproceedings{abs-2607-02825,
title = {JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-backend Harness, and a Leakage-Aware Evaluation Mode},
author = {Norbert Sándor Szolnoki and Gábor Antal},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834649}
}