一句话：首次系统比较 LLM 与深度学习自动程序修复模型修复 Java 安全漏洞的能力，并构建新基准 VJBench。
安全漏洞修复亟需自动化，大代码语言模型（LLM）与基于深度学习的自动程序修复（APR）技术都显示出潜力，但二者在漏洞修复上的能力尚未被系统比较。本文首次针对 Java 漏洞修复开展对比研究：在 Vul4J 与 VJBench 两个真实 Java 漏洞基准上评估了 5 个 LLM（Codex、CodeGen、CodeT5、PLBART、InCoder）、4 个微调 LLM 以及 4 种 DL-based APR 技术；为缓解 Codex 训练与测试数据重叠的威胁，设计了代码变换；并创建新基准 VJBench 及其变换版本 VJBench-trans。结果显示，现有 LLM 与 APR 模型只能修复极少数 Java 漏洞，Codex 修复最多（10.2 个，20.4%），且大量生成补丁无法编译；用通用 APR 数据微调可提升 LLM 的漏洞修复能力；VJBench 揭示模型难以修复 CWE-325、CWE-444 等类型；在变换漏洞上 Codex 仍修复 8.7 个，优于其他所有模型。作者据此呼吁构建更大规模的漏洞修复训练数据、用其微调 LLM，并引入代码简化变换以促进漏洞修复。
Security vulnerability repair is a difficult task that is in dire need of automation. Two groups of techniques have shown promise: (1) large code language models (LLMs) that have been pre-trained on source code for tasks such as code completion, and (2) automated program repair (APR) techniques that use deep learning (DL) models to automatically fix software bugs. This paper is the first to study and compare Java vulnerability repair capabilities of LLMs and DL-based APR models. The contributions include that we (1) apply and evaluate five LLMs (Codex, CodeGen, CodeT5, PLBART and InCoder), four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks (Vul4J and VJBench), (2) design code transformations to address the training and test data overlapping threat to Codex, (3) create a new Java vulnerability repair benchmark VJBench, and its transformed version VJBench-trans, to better evaluate LLMs and APR techniques, and (4) evaluate LLMs and APR techniques on the transformed vulnerabilities in VJBench-trans. Our findings include that (1) existing LLMs and APR models fix very few Java vulnerabilities. Codex fixes 10.2 (20.4%), the most number of vulnerabilities. Many of the generated patches are uncompilable patches. (2) Fine-tuning with general APR data improves LLMs’ vulnerability-fixing capabilities. (3) Our new VJBench reveals that LLMs and APR models fail to fix many Common Weakness Enumeration (CWE) types, such as CWE-325 Missing cryptographic step and CWE-444 HTTP request smuggling. (4) Codex still fixes 8.7 transformed vulnerabilities, outperforming all the other LLMs and APR models on transformed vulnerabilities. The results call for innovations to enhance automated Java vulnerability repair such as creating larger vulnerability repair training data, tuning LLMs with such data, and applying code simplification transformation to facilitate vulnerability repair.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{WuJPLD0BS23,
title = {How Effective Are Neural Networks for Fixing Security Vulnerabilities},
author = {Yi Wu and Nan Jiang and Hung Viet Pham and Thibaud Lutellier and Jordan Davis and Lin Tan and Petr Babkin and Sameena Shah},
booktitle = {ISSTA},
year = {2023},
doi = {10.1145/3597926.3598135}
}