一句话：深入剖析 NMT 提交信息生成方法为何有效，并据此提出更快更准的最近邻方法 NNGen。
提交信息可视为软件变更的文档，但开发者常因时间不足而忽略撰写。Jiang 等人此前提出基于神经机器翻译（NMT）从代码 diff 自动生成简短提交信息的方法，效果看似可观，但其成功原因未被探究。本文首先对其实验结果做深入分析，发现：NMT 能生成高质量信息的测试 diff 大多与某个训练 diff 在 token 级别高度相似；Jiang 等人数据集中约 16% 的提交信息因自动生成或描述重复琐碎变更而含噪声；剔除这些噪声后 NMT 性能大幅下降。此外 NMT 结构复杂且耗时。受第一项发现启发，作者提出更简单快速的最近邻生成器 NNGen。实验表明，NNGen 比 NMT 快 2600 倍以上，BLEU 指标高出 21%。文章最后还讨论了自动提交信息生成未来研究方向的若干观察，以启发后续工作。
Commit messages can be regarded as the documentation of software changes. These messages describe the content and purposes of changes, hence are useful for program comprehension and software maintenance. However, due to the lack of time and direct motivation, commit messages sometimes are neglected by developers. To address this problem, Jiang et al. proposed an approach (we refer to it as NMT), which leverages a neural machine translation algorithm to automatically generate short commit messages from code. The reported performance of their approach is promising, however, they did not explore why their approach performs well. Thus, in this paper, we first perform an in-depth analysis of their experimental results. We find that (1) Most of the test diffs from which NMT can generate high-quality messages are similar to one or more training diffs at the token level. (2) About 16% of the commit messages in Jiang et al.’s dataset are noisy due to being automatically generated or due to them describing repetitive trivial changes. (3) The performance of NMT declines by a large amount after removing such noisy commit messages. In addition, NMT is complicated and time-consuming. Inspired by our first finding, we proposed a simpler and faster approach, named NNGen (Nearest Neighbor Generator), to generate concise commit messages using the nearest neighbor algorithm. Our experimental results show that NNGen is over 2,600 times faster than NMT, and outperforms NMT in terms of BLEU (an accuracy measure that is widely used to evaluate machine translation systems) by 21%. Finally, we also discuss some observations for the road ahead for automated commit message generation to inspire other researchers.
@inproceedings{paperbot4697,
title = {Neural-machine-translation-based commit message generation: how far are we?},
author = {Zhongxin Liu and Xin Xia and Ahmed E. Hassan and David Lo and Zhenchang Xing and Xinyu Wang},
booktitle = {ASE},
year = {2018},
doi = {10.1145/3238147.3238190}
}