一句话：提出首个 LLM 驱动的智能合约漏洞数据集自动构建框架，从真实审计报告抽取漏洞并按 CWE 分类，构建大规模数据集。
高质量智能合约漏洞数据集是评估安全工具与推进安全研究的基础，但人工构建存在两大局限：标注过程耗费人力且易出错，限制了数据集的规模、质量与持续演化；缺乏统一分类规则导致不同数据集的漏洞类别与标注结果不一致。为此，作者提出 FORGE——首个自动化构建智能合约漏洞数据集的方法。FORGE 采用 LLM 驱动的流水线，从真实世界审计报告中抽取高质量漏洞，并依据软件安全领域最广泛认可的 CWE 分类体系进行归类。它使用分治策略从非结构化审计报告中抽取结构化、自包含的漏洞信息，并借助思维树（tree-of-thoughts）技术将漏洞信息映射到层级化的 CWE 分类中。作者在 6,454 份真实审计报告上运行 FORGE，生成包含 81,390 个 Solidity 文件、27,497 条漏洞发现、覆盖 296 个 CWE 类别的数据集。人工评估显示抽取精度达 95.6%，与人类专家的分类一致性 k-α 为 0.87。作者还用该数据集对 13 个现有安全工具进行基准评测，揭示当前检测能力的显著不足，并从统一 CWE 视角分析严重度—频率分布，指出研究关注点与真实漏洞优先级之间的不一致。
High-quality smart contract vulnerability datasets are critical for evaluating security tools and advancing smart contract security research. Two major limitations of current manual dataset construction are (1) labor-intensive and error-prone annotation processes limit the scale, quality, and evolution of the dataset, and (2) absence of standardized classification rules results in inconsistent vulnerability categories and labeling results across different datasets. To address these limitations, we present forge, the first automated approach for constructing smart contract vulnerability datasets. forge leverages an LLM-driven pipeline to extract high-quality vulnerabilities from real-world audit reports and classify them according to the Common Weakness Enumeration (CWE), the most widely recognized classification in software security. forge employs a divide-and-conquer strategy to extract structured and self-contained vulnerability information from unstructured audit reports. Additionally, it uses a tree-of-thoughts technique to classify the vulnerability information into the hierarchical CWE classification. To evaluate forge’s effectiveness, we run forge on 6,454 real-world audit reports and generate a dataset comprising 81,390 solidity files and 27,497 vulnerability findings across 296 CWE categories. Manual assessment of the dataset demonstrates high extraction precision and classification consistency with human experts (precision of 95.6% and inter-rater agreement k-α of 0.87). We further validate the practicality of our dataset by benchmarking 13 existing security tools on our dataset. The results reveal the significant limitations in current detection capabilities. Furthermore, by analyzing the severity-frequency distribution patterns through a unified CWE perspective in our dataset, we highlight inconsistency between the current smart contract research focus and the priorities identified from real-world vulnerabilities. This analysis also reveals key differences from traditional software security concerns, providing practical implications for improving the smart contract security ecosystem.
@inproceedings{abs-2506-18795,
title = {FORGE: An LLM-driven Framework for Large-Scale Smart Contract Vulnerability Dataset Construction},
author = {Jiachi Chen and Yiming Shen and Jiashuo Zhang and Zihao Li and John Grundy and Zhenzhe Shao and Yanlin Wang and Jiashui Wang and Ting Chen and Zibin Zheng},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3764529}
}