一句话：实证表明不同代码预训练模型的嵌入对漏洞检测性能影响显著，并提出基于嵌入指标与随机森林推荐最优模型的 Coding-PTMs 框架。
漏洞检测中常直接复用 CodeBERT、CodeT5、CodeGen 等代码预训练模型生成代码嵌入，却缺乏选择依据，并默认不同模型嵌入对性能影响相似。本文系统研究十种代码预训练模型的嵌入对漏洞检测性能的影响，发现上述假设不成立：不同模型嵌入会显著影响性能，且依据参数规模与嵌入维度选择并不可靠。为量化差异，作者从统计、范数和分布三个维度定义十三种嵌入指标，构建代码预训练模型推荐数据集，并训练随机森林分类器为具体漏洞检测任务从候选模型库中推荐最优模型。结果支持使用 Coding-PTMs 评估候选模型嵌入特征并辅助选择。
Vulnerability detection is garnering increasing attention in software engineering, since code vulnerabilities possibly pose significant security. Recently, reusing various code pre-trained models (e.g., CodeBERT, CodeT5, and CodeGen) has become common for code embedding without providing reasonable justifications in vulnerability detection. The premise for casually utilizing pre-trained models (PTMs) is that the code embeddings generated by different PTMs would generate a similar impact on the performance. Is that TRUE? To answer this important question, we systematically investigate the effects of code embedding generated by ten different code PTMs on the performance of vulnerability detection, and get the answer, i.e., that is NOT true. We observe that code embedding generated by various code PTMs can indeed influence the performance and selecting an embedding technique based on parameter scales and embedding dimension is not reliable. Our findings highlight the necessity of quantifying and evaluating the characteristics of code embedding generated by various code PTMs to understand the effects. To achieve this goal, we analyze the numerical representation and data distribution of code embedding generated by different PTMs to evaluate differences and characteristics. Based on these insights, we propose Coding-PTMs, a recommendation framework to assist engineers in selecting optimal code PTMs for their specific vulnerability detection tasks. Specifically, we define thirteen code embedding metrics across three dimensions (i.e., statistics, norm, and distribution) for constructing a specialized code PTM recommendation dataset. We then employ a Random Forest classifier to train a recommendation model and identify the optimal code PTMs from the candidate model zoo. We encourage engineers to use our Coding-PTMs to evaluate the characteristics of code embeddings generated by candidate code PTMs on the performance and recommend optimal code PTMs for code embedding in their vulnerability detection tasks.CCS Concepts• Software and its engineering → Empirical software validation; Software defect analysis.
@inproceedings{ZhaoGHWW024,
title = {Coding-PTMs: How to Find Optimal Code Pre-trained Models for Code Embedding in Vulnerability Detection?},
author = {Yu Zhao and Lina Gong and Zhiqiu Huang and Yongwei Wang and Mingqiang Wei and Fei Wu},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695539}
}