一句话：首次系统实证比较19个源代码预训练模型在13项软件工程任务上的表现，并分析模型类别与任务性能的相关性。
近年来，大量源代码预训练模型被成功开发并应用于各类软件工程任务，但人们对这些模型的理解仍相当有限。为增进理解，本文首次对19个近期提出的源代码预训练模型在13项软件工程任务上进行了系统的实证比较。研究采用近期提出的四维预训练模型分类法，进一步探究不同类别的预训练模型与其在不同软件工程任务上的性能之间是否存在相关性。通过大规模实验，该工作揭示了模型类别与任务表现之间的关联，为后续模型选择与设计提供了实证依据。
While a large number of pre-trained models of source code have been successfully developed and applied to a variety of software engineering (SE) tasks in recent years, our understanding of these pre-trained models is arguably fairly limited. With the goal of advancing our understanding of these models, we perform the first systematic empirical comparison of 19 recently-developed pre-trained models of source code on 13 SE tasks. To gain additional insights into these models, we adopt a recently -developed 4-dimensional categorization of pre-trained models, and subsequently investigate whether there are correlations between different categories of pre-trained models and their performances on different SE tasks.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{NiuLNCGL23,
title = {An Empirical Comparison of Pre-Trained Models of Source Code},
author = {Changan Niu and Chuanyi Li and Vincent Ng and Dongxiao Chen and Jidong Ge and Bin Luo},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00180}
}