一句话：通过实证研究揭示 GitHub Copilot 对语义等价自然语言描述的敏感性，约46%情况下生成代码不同。
本文研究基于深度学习的代码生成器在自然语言描述发生语义等价变化时是否保持鲁棒。作者以 GitHub Copilot 为对象，从 892 个 Java 方法的原始 Javadoc 出发，要求其生成函数，并分别人工和自动构造语义等价但措辞不同的描述，再比较 Copilot 输出代码是否改变。结果表明，修改描述后约 46% 的情况下生成代码推荐发生变化；这些语义等价描述差异还可能影响生成代码的正确性，幅度达 ±28%。该工作警示：开发者用不同措辞描述同一需求可能得到不同甚至不正确的代码，需关注代码生成模型的鲁棒性评估。
Software engineering research has always being concerned with the improvement of code completion approaches, which suggest the next tokens a developer will likely type while coding. The release of GitHub Copilot constitutes a big step forward, also because of its unprecedented ability to automatically generate even entire functions from their natural language description. While the usefulness of Copilot is evident, it is still unclear to what extent it is robust. Specifically, we do not know the extent to which semantic-preserving changes in the natural language description provided to the model have an effect on the generated code function. In this paper we present an empirical study in which we aim at understanding whether different but semantically equivalent natural language descriptions result in the same recommended function. A negative answer would pose questions on the robustness of deep learning (DL)-based code generators since it would imply that developers using different wordings to describe the same code would obtain different recommendations. We asked Copilot to automatically generate 892 Java methods starting from their original Javadoc description. Then, we generated different semantically equivalent descriptions for each method both manually and automatically, and we analyzed the extent to which predictions generated by Copilot changed. Our results show that modifying the description results in different code recommendations in ∼46% of cases. Also, differences in the semantically equivalent descriptions might impact the correctness of the generated code (±28%).
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2302-00438,
title = {On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot},
author = {Antonio Mastropaolo and Luca Pascarella and Emanuela Guglielmi and Matteo Ciniselli and Simone Scalabrino and Rocco Oliveto and Gabriele Bavota},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00181}
}