一句话：实证研究神经代码编辑注入漏洞的可行性，揭示其潜力与差距，并证明可提升漏洞检测器F1达49.51%。
大规模真实漏洞数据集对软件安全基准测试与数据驱动方法至关重要，却严重匮乏。本文实证探索通过神经代码编辑向真实程序注入漏洞以生成此类数据集的可行性，在合成与真实数据集上评估三种最新神经代码编辑器。结果显示编辑器在真实数据上局限严重：最佳准确率仅10.03%，合成数据上为79.40%；基于图的编辑器优于序列模型，可在34.93%真实测试样本中注入漏洞，序列模型成功率为0，但仍受复杂代码结构与长编辑限制，源于预处理和深度学习模型设计不足。生成的漏洞样本可将基于深度学习的漏洞检测器F1提升至多49.51%。文末还总结编辑器擅长删除而不擅长替换等差距及改进建议。
The availability of large-scale, realistic vulnerability datasets is essential both for benchmarking existing techniques and for developing effective new data-driven approaches for software security. Yet such datasets are critically lacking. A promising solution is to generate such datasets by injecting vulnerabilities into real-world programs, which are richly available. Thus, in this paper, we explore the feasibility of vulnerability injection through neural code editing. With a synthetic dataset and a real-world one, we investigate the potential and gaps of three state-of-the-art neural code editors for vulnerability injection. We find that the studied editors have critical limitations on the real-world dataset, where the best accuracy is only 10.03%, versus 79.40% on the synthetic dataset. While the graph-based editors are more effective (successfully injecting vulnerabilities in up to 34.93% of real-world testing samples) than the sequence-based one (0 success), they still suffer from complex code structures and fall short for long edits due to their insufficient designs of the preprocessing and deep learning (DL) models. We reveal the promise of neural code editing for generating realistic vulnerable samples, as they help boost the effectiveness of DL-based vulnerability detectors by up to 49.51% in terms of F1 score. We also provide insights into the gaps in current editors (e.g., they are good at deleting but not at replacing code) and actionable suggestions for addressing them (e.g., designing effective editing primitives).
@inproceedings{paperbot6989,
title = {Generating realistic vulnerabilities via neural code editing: an empirical study},
author = {Yu Nong and Yuzhe Ou and Michael Pradel and Feng Chen and Haipeng Cai},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3549128}
}