一句话：提出CoCoSoDa，通过软数据增强、动量负样本与多模态对比学习提升代码搜索性能。
代码搜索旨在根据自然语言查询检索语义相关的代码片段。现有对比学习方法虽已提升代码表示学习和搜索性能，但仍有改进空间。本文提出CoCoSoDa，从对比学习的两个关键因素入手：数据增强与负样本。具体地，采用软数据增强，对输入序列动态掩码或按类型替换部分token以生成正样本；利用队列和动量编码器，在小批量中生成大量且一致的负样本表示；并通过多模态对比学习拉近代码-查询配对表示、推开非配对表示。作者在六种编程语言的大规模数据集上开展实验，与18个基线比较，并进行消融、预训练模型迁移、超参数鲁棒性以及定性与定量分析。结果表明，CoCoSoDa平均MRR分别超过CodeBERT、GraphCodeBERT和UniXcoder 13.3%、10.5%和5.9%。
Code search aims to retrieve semantically relevant code snippets for a given natural language query. Recently, many approaches employing contrastive learning have shown promising results on code representation learning and greatly improved the performance of code search. However, there is still a lot of room for improvement in using contrastive learning for code search. In this paper, we propose CoCoSoDa to effectively utilize contrastive learning for code search via two key factors in contrastive learning: data augmentation and negative samples. Specifically, soft data augmentation is to dynamically masking or replacing some tokens with their types for input sequences to generate positive samples. Momentum mechanism is used to generate large and consistent representations of negative samples in a mini-batch through maintaining a queue and a momentum encoder. In addition, multimodal contrastive learning is used to pull together representations of code-query pairs and push apart the unpaired code snippets and queries. We conduct extensive experiments to evaluate the effectiveness of our approach on a large-scale dataset with six programming languages. Experimental results show that: (1) CoCoSoDa outperforms 18 baselines and especially exceeds CodeBERT, GraphCodeBERT, and UniXcoder by 13.3%, 10.5%, and 5.9% on average MRR scores, respectively. (2) The ablation studies show the effectiveness of each component of our approach. (3) We adapt our techniques to several different pre-trained models such as RoBERTa, CodeBERT, and GraphCodeBERT and observe a significant boost in their performance in code search. (4) Our model performs robustly under different hyper-parameters. Furthermore, we perform qualitative and quantitative analyses to explore reasons behind the good performance of our model.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{ShiWGDZHZS23,
title = {CoCoSoDa: Effective Contrastive Learning for Code Search},
author = {Ensheng Shi and Yanlin Wang and Wenchao Gu and Lun Du and Hongyu Zhang and Shi Han and Dongmei Zhang and Hongbin Sun},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00185}
}