一句话：提出ContraBERT，用九种PL/NL数据增强与对比预训练提升代码预训练模型的鲁棒性。
大规模代码预训练模型如CodeBERT、GraphCodeBERT在克隆检测、代码搜索和代码翻译等任务上表现优异，但易受对抗攻击：仅重命名变量等简单扰动就会导致性能显著下降，且这种弱点可能被下游模型继承并放大。为此，作者提出ContraBERT，在编程语言和自然语言数据上设计九种简单与复杂的数据增强算子来构造变体，并通过掩码语言建模（MLM）和对比预训练任务，在原始样本及其增强变体上继续训练已有预训练模型，以增强鲁棒性。大量实验表明，ContraBERT能有效提升现有代码预训练模型的鲁棒性；进一步研究也显示，增强鲁棒性后的模型在四个常见下游任务上优于原始模型。
Large-scale pre-trained models such as CodeBERT, GraphCodeBERT have earned widespread attention from both academia and industry. Attributed to the superior ability in code representation, they have been further applied in multiple downstream tasks such as clone detection, code search and code translation. However, it is also observed that these state-of-the-art pre-trained models are susceptible to adversarial attacks. The performance of these pre-trained models drops significantly with simple perturbations such as renaming variable names. This weakness may be inherited by their downstream models and thereby amplified at an unprecedented scale. To this end, we propose an approach namely ContraBERT that aims to improve the robustness of pre-trained models via contrastive learning. Specifically, we design nine kinds of simple and complex data augmentation operators on the programming language (PL) and natural language (NL) data to construct different variants. Furthermore, we continue to train the existing pre-trained models by masked language modeling (MLM) and contrastive pre-training task on the original samples with their augmented variants to enhance the robustness of the model. The extensive ex-periments demonstrate that ContraBERT can effectively improve the robustness of the existing pre-trained models. Further study also confirms that these robustness-enhanced models provide improvements as compared to original models over four popular downstream tasks.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2301-09072,
title = {ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning},
author = {Shangqing Liu and Bozhi Wu and Xiaofei Xie and Guozhu Meng and Yang Liu},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00207}
}