一句话：针对标注数据稀缺，探讨如何利用带噪伪标注的无标注数据来有效微调预训练代码模型。
预训练代码模型已在多项代码智能任务中取得显著进展：先在大规模无标注数据上以自监督方式任务无关地预训练，再在下游任务的有标注数据上微调。然而，下游标注数据规模通常有限，人工标注成本高，可能限制模型在特定任务上的表现。为缓解该问题，一种思路是在微调阶段利用大规模无标注数据，即通过伪标注为无标注样本生成伪标签，再用伪标注数据继续训练预训练模型。但摘要指出，直接使用伪标注数据会引入大量噪声（错误标签），从而导致性能次优。因此，如何有效利用含噪伪标注数据仍是一个具有挑战性且尚未充分探索的问题。摘要未给出具体方法细节与实验结论。
Pre-trained code models have recently achieved substantial improvements in many code intelligence tasks. These models are first pre-trained on large-scale unlabeled datasets in a task-agnostic manner using self-supervised learning, and then fine-tuned on labeled datasets in downstream tasks. However, the labeled datasets are usually limited in size (i.e., human intensive efforts), which may hinder the performance of pre-trained code models in specific tasks. To mitigate this, one possible solution is to leverage the large-scale unlabeled data in the tuning stage by pseudo-labeling, i.e., generating pseudo labels for unlabeled data and further training the pre-trained code models with the pseudo-labeled data. However, directly employing the pseudo-labeled data can bring a large amount of noise, i.e., incorrect labels, leading to suboptimal performance. How to effectively leverage the noisy pseudo-labeled data is a challenging yet under-explored problem.
DOI 原文 ·
@inproceedings{paperbot11457,
title = {Learning in the Wild: Towards Leveraging Unlabeled Data for Effectively Tuning Pre-trained Code Models},
author = {Shuzheng Gao and Wenxin Mao and Cuiyun Gao and Li Li and Xing Hu and Xin Xia and Michael R. Lyu},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639216}
}