一句话：用语义保持变换扩充下游数据并配合课程学习微调，帮助预训练代码模型更好地适配新任务。
论文关注源代码理解中「先预训练、再微调」范式下的一个关键问题：相比从头训练大模型，如何有效地把预训练模型适配到新的下游任务仍未被充分探索。作者提出一种桥接预训练模型与代码相关任务的方法：一方面利用语义保持变换（semantic-preserving transformation）来丰富下游数据的多样性，促使预训练模型学习到对这些语义等价变换保持不变的语义特征；另一方面引入课程学习（curriculum learning），将变换后的数据按由易到难的方式组织，用于微调已有预训练模型。这样既缓解了下游任务数据有限、分布单一的问题，又让模型在微调阶段逐步适应更复杂的样本。摘要未报告具体数据集、基线或量化结果。
With the great success of pre-trained models, the pretrain-then-finetune paradigm has been widely adopted on downstream tasks for source code understanding. However, compared to costly training a large-scale model from scratch, how to effectively adapt pre-trained models to a new task has not been fully explored. In this paper, we propose an approach to bridge pre-trained models and code-related tasks. We exploit semantic-preserving transformation to enrich downstream data diversity, and help pre-trained models learn semantic features invariant to these semantically equivalent transformations. Further, we introduce curriculum learning to organize the transformed data in an easy-to-hard manner to fine-tune existing pre-trained models.
DOI 原文 ·
@inproceedings{paperbot6922,
title = {Bridging pre-trained models and downstream tasks for source code understanding},
author = {Deze Wang and Zhouyang Jia and Shanshan Li and Yue Yu and Yun Xiong and Wei Dong and Xiangke Liao},
booktitle = {ICSE},
year = {2022},
doi = {10.1145/3510003.3510062}
}