一句话：提出“自然化”源代码的预训练目标，用六类语义保持变换生成非自然代码，让模型学会还原开发者书写的自然程序，在多个生成式任务上媲美 CodeT5。
该文关注代码预训练模型中预训练目标的设计问题：现有模型（如 PLBART、CodeT5、SPT-Code）虽在代码生成与翻译等任务上表现优异，但其预训练目标仍有改进空间。作者利用代码兼具形式通道与自然通道的“双模态”特性，提出名为“自然化”的新预训练目标：先定义六类语义保持变换，将原始代码改写为各种“非自然”形式，再强制模型从这些非自然版本还原出开发者书写的更自然的等价程序。该过程在大规模开源代码语料上自监督进行，无需人工标注，使模型同时学习代码的理解与生成。作者用有限的人工标注数据将模型微调到代码生成、代码翻译和代码精化三个生成式软件工程任务上，性能达到与 CodeT5 相当的最先进水平，并在零样本与少样本设置下尤其有竞争力，同时在语法、数据流等代码属性的学习上表现更好。
Pre-trained Generative Language models (e.g., PLBART, CodeT5, SPT-Code) for source code yielded strong results on several tasks in the past few years, including code generation and translation. These models have adopted varying pre-training objectives to learn statistics of code construction from very large-scale corpora in a self-supervised fashion; the success of pre-trained models largely hinges on these pre-training objectives. This paper proposes a new pre-training objective, “Naturalizing” of source code, exploiting code’s bimodal, dual-channel (formal & natural channels) nature. Unlike natural language, code’s bimodal, dual-channel nature allows us to generate semantically equivalent code at scale. We introduce six classes of semantic preserving transformations to introduce unnatural forms of code, and then force our model to produce more natural original programs written by developers. Learning to generate equivalent, but more natural code, at scale, over large corpora of open-source code, without explicit manual supervision, helps the model learn to both ingest & generate code. We fine-tune our model in three generative Software Engineering tasks: code generation, code translation, and code refinement with limited human-curated labeled data and achieve state-of-the-art performance rivaling CodeT5. We show that our pre-trained model is especially competitive at zero-shot and few-shot learning, and better at learning code properties (e.g., syntax, data flow)
DOI 原文 ·
@inproceedings{paperbot7116,
title = {NatGen: generative pre-training by “naturalizing” source code},
author = {Saikat Chakraborty and Toufique Ahmed and Yangruibo Ding and Premkumar T. Devanbu and Baishakhi Ray},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3549162}
}