一句话：在自然语言与源码上预训练 T5，单模型微调后在缺陷修复、变异注入、断言生成与注释生成四个任务上均超越各自基线。
本文研究 Text-To-Text Transfer Transformer（T5）在代码相关任务上的适用性。T5 先在大型通用数据上以自监督方式预训练，再针对具体任务微调。作者在一个由英文自然语言文本和源代码组成的数据集上预训练 T5，然后复用四项先前工作中基于深度学习的数据集分别微调，对应任务为：修复缺陷、注入代码变异体、生成 assert 语句以及生成代码注释。随后，作者将这一个统一模型的性能与四篇原始论文所报告的各自专用方案进行对比。结果显示，借助自监督预训练阶段引入的额外数据，该 T5 模型在四个任务上均能相对原始基线取得性能提升，说明统一的文本到文本架构与跨模态预训练可为多种代码相关任务提供有效支持。
Deep learning (DL) techniques are gaining more and more attention in the software engineering community. They have been used to support several code-related tasks, such as automatic bug fixing and code comments generation. Recent studies in the Natural Language Processing (NLP) field have shown that the Text-To-Text Transfer Transformer (T5) architecture can achieve state-of-the-art performance for a variety of NLP tasks. The basic idea behind T5 is to first pre-train a model on a large and generic dataset using a self-supervised task (e.g., filling masked words in sentences). Once the model is pre-trained, it is fine-tuned on smaller and specialized datasets, each one related to a specific task (e.g., language translation, sentence classification). In this paper, we empirically investigate how the T5 model performs when pre-trained and fine-tuned to support code-related tasks. We pre-train a T5 model on a dataset composed of natural language English text and source code. Then, we fine-tune such a model by reusing datasets used in four previous works that used DL techniques to: (i) fix bugs, (ii) inject code mutants, (iii) generate assert statements, and (iv) generate code comments. We compared the performance of this single model with the results reported in the four original papers proposing DL-based solutions for those four tasks. We show that our T5 model, exploiting additional data for the self-supervised pre-training phase, can achieve performance improvements over the four baselines.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{MastropaoloSCNP21,
title = {Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks},
author = {Antonio Mastropaolo and Simone Scalabrino and Nathan Cooper and David Nader Palacio and Denys Poshyvanyk and Rocco Oliveto and Gabriele Bavota},
booktitle = {ICSE},
year = {2021},
doi = {10.1109/icse43902.2021.00041}
}